In the past, it was common practice to use sigmoids as output activation functions and base final-layer loss functions on squared errors-sometimes even when classification labels were constrained to be 0 or 1. We distinguish the final-layer parameterization, from which the loss function is computed, from the intermediate-layer activation functions. Section 7.2 noted that different activation functions can be used for multilayer perceptrons. Pal, in Data Mining (Fourth Edition), 2017 Losses and Regularization For classification tasks and for the output layer, the softmax nonlinearity, combined with the cross-entropy cost function, is most commonly used. At the time this edition is compiled, it seems that the use of ReLU, in any one of its versions, is the more popular choice for the hidden layers. There is no definite answer to that and the choice depends on the specific application. The question that is now raised concerns the choice of the specific nonlinearity to be used in practice. The ReLU was first introduced in the context of dynamical networks in and it was motivated by biological arguments. A further modification, known as the max output unit, a number of, say, k different linear units are employed, whose parameters are learned during the training, and, each time, the one that results in the maximum value is selected to activate the corresponding neuron. In, О± is left as a parameter to be learned during training. If О± is assigned a fixed small value, e.g., 0.01, the resulting nonlinearity is known as the leaky ReLU. For О± = в€’ 1, the so-called absolute value rectification is obtained. Дepending on the value of О±, different variants result. Recall that in the backpropagation algorithm, when gradients with respect to the parameters of hidden layers are computed, the derivative of the activation function enters in a multiplicative fashion.ДЇ ( z ) = max вЃЎ.

Thus, selecting the ReLU as the activation function, one bypasses problems related to the slowing down when derivatives get small values. At z = 0 the derivative is not defined and if this happens during training, one can either choose 1 or 0 for those familiar with the notion of the subgradient (Chapter 8) such a choice is fully justified. For negative values, the derivative is zero.

Thus, it is desirable to set the biases of the neurons, during initialization, to some small positive value, e.g., Оё 0 = 0.1, in order to increase the probability that the input to the activation function is positive. Such an activation function does not suffer from saturation and its derivative is equal to one when the neuron operates in its active region ( z > 0).

It has been reported that in the context of deep networks, the use of the ReLU nonlinearity in the hidden layers can significantly speed up the training time. Note that for zвЂЇ>вЂЇ0 the derivative is equal to one and for zвЂЇ<вЂЇ0 it is equal to zero. This is the motivation behind the network representation of Figure 7, where each neuron has separate input ports due to separate deterministic-to-stochastic conversions.ДЇigure 18.13. For this reason, inputs must be independently converted for each neuron. (2.11) means that two neurons of the same layer do not necessarily have the same stream length, since their reset periodicities can differ. It can be easily seen that the shortest streams are equivalent to long streams of repeated bits: This translates into a (total) correlation between bits that again is detrimental in terms of stochastic noise on the output. Note that such an increase in length applies to all the streams in the neuron (i.e., the output stream and the inputs and weights streams). In other words, if we desire a highly nonlinear output function, we must increase the length of the streams in the neuron or accept a higher stochastic noise.

What is the need of activation function