Machine Learning

• Calculate the cross entropy of the prediction with respect to the values that are expected.

• Rectified Linear Unit

\[ \mathrm{ReLU}(x) := \max (0,x). \]

\[ \mathrm{LReLU}(x) := \max(x, \varepsilon x) \] where \( 0 < \varepsilon \ll 1. \)

• Exponential Linear Unit

\[ \mathrm{ELU}(x) := \begin{cases} x & x >0, \\ \alpha(e^x -1) & x\le 0 \] where \( \alpha \ge 0 \).

• Researched in 80s and 90s.
• But what is a neural network? | Deep learning chapter 1 - YouTube
• It is useful to have rough concrete meaning of each layer (beforehand), to assess the network.
• Layer \(n\) Neurons \[a_{j}^{(n)}=\sigma\left( b_{j}^{(n)} + \sum_{i}w_{ij}^{(n)}a_{i}^{(n-1)}\right)\]
  • where \(\sigma: \mathbb{R} \to [0,1]\) is usually the sigmoid function, aka logistic curve: \[ \sigma(x) = \frac{1}{1+e^{-x}} \]
    • It could be any other functions, such as ReLU.
  • with matrix notation: \[ \mathbf{a}^{(n)} = \sigma\left(\mathbf{b}^{(n)}+\mathbf{W}^{(n)}\mathbf{a}^{(n-1)}\right) \]
• The entire neural network can be thought of as a function, as well as each neuron. \[ \mathbf{a}^{(-1)} = \mathbf{M}\left(\mathbf{\mathbf{a}}^{(0)}\right) = \dots\mathbf{M}^{(2)}\mathbf{M}^{(1)}\mathbf{a}^{(0)} \]

• RNN
• Serial processing. Like for audio.

• Used for image processing
• Perform multiple 2D convolution at each layer, to detect features.

• Extract the core features and lift back to the original shape.

• Self Attention Layer: Dynamic layer that adapt the weights based on the context. Measures the distance in the concept space.
• It is so named because it transforms the meaning of a sentence in the concept space.
• It is shallower but wider.
• Transformer Neural Networks Derived from Scratch - YouTube
• Calculate the attention of every pair of input vectors with an attention head neural network, and use it as a weight to reduce it down.

• Use the output, as the input for the next inference.

When generating an image, the diffusion model performs Langevin sampling, specifically using the Euler-Maruyama method: \[ \mathbf{x}_{t+1} = \mathbf{x}_t + \frac{\epsilon}{2} F(\mathbf{x}_t) + \sqrt{2\epsilon} \mathbf{z}_t, \] in order to sample from the space of all images, where the score function \( F \) is approximated by the trained model \( M_{\theta} \).

The random noise \( \mathbf{z}_t \) serves two main purpose.

• Escaping the local maxima,
• Providing high frequency information.

The model only learns to remove high frequency data, since they have to remove noise. Therefore constant supply of high frequency data is crucial for the generation of realistic image. Put it in other words, the noise alleviates the high correlation of contiguous pixels, generating a more realistic image.

• Move in the direction of \(-\nabla C(\theta)\), repeatedly.
• In order to calculate the gradient, the activation must be continuous and differentiable.

• Method of simplifying calculation by calculating layer by layer, starting from the last layer.
• \[ \frac{\partial C_i}{\partial a_{k}^{(L-1)}} = \sum_j \frac{\partial C_i}{\partial a_j^{(L)}}\frac{\partial a_j^{(L)}}{\partial a_k^{(L-1)}} \]
• Note that the gradient is readily calculated at each layer, since you already have all the values that the gradients is evaluated at.
• This is an application of ((65747a84-db71-4b38-aaf9-a357720b2aa9)), which enables multiple partial derivative at the same time.
• e.g. \[ \frac{\partial C_i}{\partial a_{k}^{(L-1)}} = \sum_j \left[2(a_j^{(L)}-y_j)\right]\left[w_{jk}^{(L)}\sigma'\!\!\left( z^{(L)}_j\right)\right] \]
• where \(z^{(L)}_j = b_{j}^{(L)} + \sum_k w_{jk}^{(L)}a_{k}^{(L-1)}\).

• The coefficient of the gradient.

The goal is to minimize the error of the approximation function and the complexity of the it. It is achieved with the loss function being an linear combination of two function: \[ L = L_{\mathrm{MSE}} + \alpha L_{\mathrm{count}}. \]

The \( L_{\mathrm{count}} \) can be in various form, something similar to \[ L_{\mathrm{count}} = \sum_i|w_i|^p \] when \( p \) is close to zero, this is an approximation of the maximum function. But often \( p=1 \) is used while producing reasonable result, and with simpler calculation using convex optimization.

\( w_{i} \) is the coefficient of the predetermined feature function, that linearly combines to produce the approximation function.