Sigmoid ($\sigma(x) = \frac {1}{1 - e^{-z}}$) Can saturate when $z$ is large Has a nice derivative: $\sigma’(x) = \sigma(x)(1 - \sigma(x)) $ Transforms $(-\infty; \infty) \to (0, 1)$ Tanh ($\tanh(z) = \frac {e^z - e^{-z}}{e^z + e^{-z}}$) Is a rescaled version of the sigmoid: $\sigma(z)=\frac {1 + \tanh(\frac z 2)}{2}$ Transforms $(-\infty; \infty) \to (-1, 1)$, so is zero centered May require normalization of outputs (or even inputs) to a prob distribution $\tanh’(z) = 1 - \tanh^2(z)$ Is ...
