Logistic, k-NN, naive Bayes formulas

The output lies in (0, 1). $\sigma (0)=0.5$; $\sigma (z)\to 1$ as $z\to \infty$ and $\sigma (z)\to 0$ as $z\to -\infty$.

The derivative is at most 0.25, reached at $z=0$. This identity is what makes the gradient of the cross-entropy loss so simple.

Binary classification. The prediction is class 1 when this probability exceeds the chosen threshold, which is 0.5 unless the question states another.

With the other features fixed, each unit increase in $x_j$ adds $w_j$ to the log-odds, which multiplies the odds by $e^{w_j}$.

The boundary is a hyperplane, so logistic regression is a linear classifier. $\mathbf{w}$ is normal to the boundary and points towards the class-1 side. Changing $t$ moves the hyperplane parallel to itself.

Each sample contributes $\ln P(y_i\mid {\mathbf{x}}_i)$. Maximising $\ell$ and minimising $J$ give the same weights.

For one sample the loss is $-\ln p$ if $y=1$ and $-\ln (1-p)$ if $y=0$. Using ${\log }_2$ instead only multiplies the loss by a constant.

This is identical to the cross-entropy above, rewritten for $\pm 1$ labels using $\sigma (-z)=1-\sigma (z)$. Check which label convention a question uses before you substitute.

Labels in $\{0,1\}$. This has the same form as linear regression's gradient when that loss is written $\frac{1}{2n}\sum ({\hat{y}}_i-y_i{)}^2$, with $p_i$ in place of the linear prediction. For a single sample, drop the $\frac{1}{n}$ and the sum.

$\lambda \ge 0$ sets the strength of the penalty. The bias is usually not penalised, so $\partial J_{\lambda }/\partial b$ is unchanged. Books scale $\lambda$ differently (with or without $\frac{1}{n}$ or $\frac{1}{2}$), so use the form the question gives.

$\eta >0$ is the learning rate. With the squared-error loss $\frac{1}{2}(p-y{)}^2$ per sample instead, the gradient picks up an extra factor $p(1-p)$. Learning then slows when the sigmoid saturates, and the loss is no longer convex in $\mathbf{w}$.

Euclidean is $r=2$, Manhattan is $r=1$, and Chebyshev is the limit $r\to \infty$. $d_r$ is a metric for $r\ge 1$. The order is written $r$ here because $p$ means a predicted probability in these notes. The nearest neighbour can change from one metric to another, so always use the one the question names.

$N_k(\mathbf{x})$ is the set of the $k$ nearest training points. A distance-weighted version uses weights proportional to $1/d$. A query that coincides with a training point ($d=0$) is usually given that point's value.

$\bar{x}$ and $s$ are the mean and standard deviation of the feature. Compute $x_{\min }$, $x_{\max }$, $\bar{x}$ and $s$ from the training data, and apply the same values to the query.

This assumes the features are conditionally independent given the class. The denominator is the same for every class, so you need it only when the question asks for an actual probability.

Compare the unnormalised scores. In practice the logs are summed, $\ln P(c)+{\sum }_j\ln P(x_j\mid c)$, to avoid numerical underflow. With equal priors, the rule reduces to picking the class with the largest likelihood (maximum likelihood).

$N_c$ is the number of training samples in class $c$, and $N_{j,v,c}$ is how many of them have feature $j$ equal to $v$. These are maximum-likelihood estimates and can be zero. Priors are smoothed only if the question says so.

$V_j$ is the number of distinct values feature $j$ can take (2 for a binary feature). $\alpha =1$ is Laplace smoothing. It stops a single unseen value from forcing the whole product to zero.

Used for text classification. $\mathrm{count}(w,c)$ is the number of times word $w$ appears across the training documents of class $c$, $|V|$ is the vocabulary size, and $n_w$ is how often $w$ occurs in the document being classified.

For continuous features. ${\mu }_{jc}$ and ${\sigma }_{jc}^2$ are the mean and variance of feature $j$ within class $c$ (here ${\sigma }_{jc}$ is a standard deviation, not the sigmoid). The maximum-likelihood variance divides by $N_c$ and the unbiased one by $N_c-1$; use the one the question states. The result is a density, so it can exceed 1. Multiply the terms together as before.

The independence assumption makes the parameter count grow linearly in $d$ instead of exponentially. The $K-1$ term counts the priors, since they sum to 1.