From sigmoid to modern activations: a brief history
Sigmoid and tanh dominated early deep learning. Sigmoid maps any input to 0-1; tanh maps to -1 to 1. Both are smooth and differentiable everywhere, which made them mathematically convenient for backpropagation. However, their gradients become tiny for extreme inputs, causing vanishing gradients in deep networks. ReLU changed the game: it is just max(0, x), piecewise linear, with large gradients outside the zero region. The simplicity and gradient properties made ReLU standard for decades. Yet ReLU dies at negative inputs (zero gradient), potentially losing information.
Modern variants and where they excel
GELU and SiLU are smoother alternatives that avoid ReLU's dead-neuron problem while retaining gradient flow. GELU (used in BERT and GPT models) uses the Gaussian cumulative distribution function, creating a soft version of ReLU. SiLU (used in modern CNNs like EfficientNet) uses a sigmoid-weighted linear term, balancing smoothness with efficiency. Each activation has nuances: SiLU is more expressive but slightly slower; GELU is more expensive but often trains faster. The choice depends on architecture, dataset, and computational constraints. Modern transformers favor GELU and SiLU, while older CNNs relied on ReLU variants.