Key takeaways
- Neural networks have diverse applications in machine learning and neuroscience, enhancing understanding of biological systems.
- Recent research has focused on sinusoidal activation functions, which are currently underutilised in machine learning.
- Challenges in training sinusoidal activation functions contribute to their limited use in neural networks.

What Is An Activation Function?
NN contains three layers in their structure:- Input
- Output
- Hidden layers
Sinusoidal Activation Functions: Inherent Problems
Activation functions in today’s deep NN framework generally incorporate non-periodic functions. Popular functions like sigmoid and rectifier linear unit (ReLU), are most popular. In a study conducted by academics at Tampere University of Technology, Finland, they analysed periodic activation functions such as sine waves and presented a basis for algorithms to learn faster using sinusoidal activation functions. They propounded that the non-monotonicity in periodic functions such as sine functions would make the activations fluctuate between strong and weak activations. They said: “Excluding the trivial case of constant functions, periodic functions are non-quasiconvex, and therefore non-monotonic. This means that for a periodic activation function, as the correlation with the input increases the activation will oscillate between stronger and weaker activations. This apparently undesirable behavior might suggest that periodic functions might be just as undesirable as activation functions in a typical learning task.” However, earlier developments in sinusoidal activation function involved introducing periodic functions in an NN with one hidden layer and then gradually moving with non-periodic functions such as sigmoids. In the paper, the authors also presented studies which reviewed the use of sinusoids in NN and found mixed opinions and results. In addition, the studies relied on representations such as Fourier series and transforms to evaluate activations based on periodic functions.Analysis With Sine Waves
The authors considered a deep neural network (DNN) which has a single hidden layer. The activation has linear properties at the output. The equation for activation and prediction is depicted below.h = F(WX+ bW)
ŷ = Ah +bA
Where X is the input vector, Y is the output target, h is the hidden activation and ŷ is the prediction. The terms W and A which indicate weight matrices,bW and bA are bias vectors, and F is the activation function. The F is actually a Fourier representation of sine function. After applying the minima of a functional for the network frequency, one of the three sincs created has a minimum of zero, which negates weight values around zero. This means, adding other terms such as bias and amplitude does not affect the minima and the “ripples” would still be there. (The exact mathematical computation can be found here). However, these won’t make any impact with the learning tasks as long as the input data does not have low frequencies. But, in reality this is not the case. Most data sets have data with very low frequencies. Hence, researchers find it difficult to use the sine function as a result. The authors also present a counterpoint that these can be mitigated or smoothened if the weights associated with the neurons are very small and exact in value, they can be initialised within a range of values.
Conclusion
This brings us to the fact that sinusoidal activation functions are not to be ignored or neglected in the creation of activation functions in NN. It all depends on the type of tasks present in the algorithms to be trained in the networks. Periodic functions do present an alternate take for activations in neural models.Spotted an error? Tell our editors. Read our editorial standards and corrections.




