Neural networks formulas

Multi-layer perceptrons and feed-forward neural networks. 15 formulas for GATE DA Machine Learning, each with when to use it. Each is explained in the topic notes.

All 15 Neural networks formulas

PERCEPTRON OUTPUT (STEP ACTIVATION)

z = w^{⊤} x + b, \overset{y}{^} = {10 z \geq 0 z < 0

Targets are 0 or 1. The tie z = 0 is assigned to class 1 here. A question may use the opposite tie rule or targets of -1 and +1, so read the convention given.

PERCEPTRON LEARNING RULE

w \leftarrow w + η (t - \overset{y}{^}) x, b \leftarrow b + η (t - \overset{y}{^})

Applied sample by sample, where t is the target. If the sample is classified correctly, t - y is 0 and nothing changes. A missed class-1 sample pulls w towards x, and a missed class-0 sample pushes w away from x.

LAYER-WISE FORWARD PROPAGATION

z^{(l)} = W^{(l)} a^{(l - 1)} + b^{(l)}, a^{(l)} = f^{(l)} (z^{(l)}), a^{(0)} = x

W(l) has one row per unit of layer l and one column per unit of layer l-1. f is applied element-wise, except softmax, which acts on the whole vector.

NUMBER OF PARAMETERS IN A DENSE LAYER

n_{l - 1} n_{l} + n_{l} = n_{l} (n_{l - 1} + 1)

n_l is the number of units in layer l. The network total is the sum over all layers after the input. The first term counts weights, the second counts biases.

SIGMOID AND ITS DERIVATIVE

σ (z) = \frac{1}{1 + e ^{- z}}, σ^{'} (z) = σ (z) (1 - σ (z))

Write the derivative in terms of the output y = sigma(z), so backpropagation needs no recomputation. Also 1 - sigma(z) = sigma(-z).

TANH AND ITS DERIVATIVE

tanh^{'} (z) = 1 - tanh^{2} (z)

Valid for all real z.

SOFTMAX

p_{i} = \frac{e ^{z_{i}}}{\sum _{j = 1}^{K} e ^{z_{j}}}

K output units. Outputs are positive and sum to 1. Adding the same constant to every z_i leaves p unchanged.

TWO-CLASS SOFTMAX IS A SIGMOID

K = 2 : p_{1} = \frac{e ^{z_{1}}}{e ^{z_{1}} + e ^{z_{2}}} = σ (z_{1} - z_{2})

Only the difference of the two logits matters, which is why one sigmoid unit is enough for binary classification.

SQUARED-ERROR LOSS (ONE SAMPLE)

L = \frac{1}{2} (\overset{y}{^} - t)^{2}, \frac{\partial L}{\partial y ^} = \overset{y}{^} - t

The factor 1/2 is a convention that cancels the 2 on differentiation. If a question defines L without it, the gradient is doubled.

BINARY CROSS-ENTROPY (ONE SAMPLE, SIGMOID OUTPUT)

L = - [t ln \overset{y}{^} + (1 - t) ln (1 - \overset{y}{^})], \frac{\partial L}{\partial z} = \overset{y}{^} - t

t is 0 or 1 and y = sigmoid(z). The gradient with respect to the output unit's net input z simplifies to y - t.

CATEGORICAL CROSS-ENTROPY WITH SOFTMAX

L = - i = 1 \sum K t_{i} ln p_{i}, \frac{\partial L}{\partial z _{i}} = p_{i} - t_{i}

t is a one-hot vector. With true class c, the loss is -ln p_c.

GRADIENT-DESCENT UPDATE

θ \leftarrow θ - η \frac{\partial L}{\partial θ}

theta is any weight or bias. All gradients are computed at the old parameter values, and then all parameters are updated together.

OUTPUT-LAYER ERROR SIGNAL FOR SQUARED ERROR

δ^{(L)} = (a^{(L)} - t) ⊙ f^{'} (z^{(L)})

Squared-error loss with factor 1/2, output activation f applied element-wise. For a single output unit this is (y - t) f'(z). For sigmoid output, f' = y(1 - y); for linear output, f' = 1. With cross-entropy and sigmoid or softmax, delta = y - t directly.

HIDDEN-LAYER ERROR SIGNAL

δ^{(l)} = ((W^{(l + 1)})^{⊤} δ^{(l + 1)}) ⊙ f^{'} (z^{(l)})

The symbol is the element-wise product. The error of a hidden unit is the weighted sum of the errors of the units it feeds, times its own activation slope. Use the weights as they were before this step's update.

WEIGHT AND BIAS GRADIENTS

\frac{\partial L}{\partial W ^{(l)}} = δ^{(l)} (a^{(l - 1)})^{⊤}, \frac{\partial L}{\partial b ^{(l)}} = δ^{(l)}

For a single weight: dL/dw = delta of the receiving unit times the output of the sending unit.

Where this was asked: 2024 (1), 2025 (3), 2026 (1). See the 5 previous-year questions on Neural networks. This shows what was asked, not what will be asked.

KEEP GOING

Neural networks: the notes and 5 solved previous-year questions