Deep-Learning

Overview Model quantization and model pruning are both techniques for shrinking neural networks and speeding up inference, but they compress different things. Quantization keeps every weight but represents each one with fewer bits (e.g. FP32 → INT8), while pruning keeps full precision but removes weights, neurons, or channels judged unimportant. The two techniques are complementary and are frequently chained together in a single compression pipeline. Comparison Diagram Model QuantizationBefore: FP32 (32-bit)0.482-1.0370.917-0.203quantizeAfter: INT8 (8-bit)61-132117-26Same 4 values, fewer bits each≈4× smaller, faster mathModel PruningBefore: dense networkpruneAfter: sparse (pruned)Fewer neurons & connectionskeptremoved Comparison Table Aspect Model Quantization Model Pruning Core mechanism Reduces numeric precision of weights/activations (e.g. FP32 → INT8/INT4) Removes individual weights, neurons, or channels judged low-importance What changes Same parameter count, smaller representation per value Fewer parameters; model becomes sparse or physically smaller Granularity Per-tensor, per-channel, or per-group bit-width choices Unstructured (single weights) vs structured (filters/channels/layers) When applied Post-training quantization (PTQ) or quantization-aware training (QAT) Iterative pruning during training or magnitude-based pruning after training, usually with fine-tuning Hardware/runtime requirement Needs low-precision kernel support (INT8 cores, TensorRT, XNNPACK) Unstructured pruning needs sparse-matrix kernels for real speedup; structured pruning runs on standard dense hardware Compression achieved Typically 2-4x size reduction (FP32→INT8); INT4 pushes further at higher accuracy risk Can reach 50-90% sparsity, but unstructured sparsity often doesn’t translate to real speedup without special hardware Accuracy impact & recovery Small accuracy drop, usually recovered via calibration or QAT Larger accuracy drop at high sparsity, recovered via iterative fine-tuning/retraining Combinability Often applied last, to shrink an already-pruned model further Often applied first, before the pruned model is quantized Key Differences Quantization changes each value’s bit-width; pruning changes the model’s parameter count. Realizing pruning’s theoretical speedup often requires sparse kernels, while quantization’s speedup comes from standard INT8 hardware support. Quantization degrades accuracy gradually and predictably; aggressive pruning risks a sharp accuracy cliff without fine-tuning. The two are commonly chained into a single compression pipeline, pruning first and quantizing the result. Structured pruning changes the model’s architecture shape; quantization never touches the architecture. When to Use Each Model Quantization ...

Overview RNNs process sequences one token at a time, carrying context forward through a hidden state that updates at each step. Transformers instead process every token simultaneously, letting each position directly attend to every other via self-attention. The difference reshapes everything from training speed to how well long-range context survives. Comparison Diagram RNNTransformerx1x2x3x4h1h2h3h4state passed step by stepprocesses one token at a timex1x2x3x4z1z2z3z4every token attends to every tokenprocesses all tokens at once Comparison Table Aspect RNN Transformer Input processing order Tokens consumed one at a time, in sequence All tokens consumed simultaneously Context propagation Hidden state carried forward step to step Self-attention lets each position read all others directly Long-range dependencies Signal weakens over distance (vanishing/exploding gradients) Direct connection between any two positions regardless of distance Positional information Implicit, from the order tokens are fed in Explicit, via added positional encodings Training parallelization Limited — must unroll and step through time Fully parallel across the sequence dimension Computational cost O(n) sequential steps, O(1) state per step O(n^2) attention cost over sequence length Inference/generation Constant memory, naturally one step at a time Requires KV caching to avoid recomputing past attention Typical use cases LSTM/GRU for streaming or small-scale sequence tasks BERT/GPT-style models for large-scale language and vision tasks Key Differences Transformer processes all tokens in parallel via self-attention; RNN processes tokens sequentially through a hidden state RNN suffers from vanishing gradients over long sequences; Transformer links any two positions directly Transformer needs explicit positional encodings since attention has no inherent order; RNN gets order for free Transformer training scales with quadratic complexity in sequence length; RNN training is linear but hard to parallelize When to Use Each RNN ...

Overview Batch normalization and dropout are both inserted between layers of a neural network, but they solve different problems during training. Batch normalization rescales activations using batch statistics to stabilize and speed up training, while dropout randomly zeroes neurons to stop the network from over-relying on any single feature. Comparison Diagram Batch Normalization Dropout raw activations (batch) μ=0, σ=1 normalized, uniform scale recomputed from batch stats every pass pass 1 - random mask pass 2 - different mask random subset silenced each forward pass Comparison Table Aspect Batch Normalization Dropout Insertion point after a linear/conv layer, before the activation function after the activation function, on the layer’s output Core mechanism normalizes activations to zero mean/unit variance using batch statistics, then applies a learnable scale and shift randomly zeroes a fraction p of activations on each forward pass Primary goal stabilize and accelerate training by reducing internal covariate shift reduce overfitting by preventing neurons from co-adapting Learnable/hyperparameters learnable scale (gamma) and shift (beta) per channel; tracks running mean/variance no learnable parameters; single hyperparameter, the dropout rate p Train vs inference behavior uses batch statistics in training, switches to stored running averages at inference active during training, disabled entirely (identity function) at inference Sensitivity to batch size degrades with very small or inconsistent batches since statistics become noisy unaffected by batch size, operates independently per example Combined usage typically placed before dropout; can conflict with dropout’s variance shift often reduced or omitted alongside batch norm in modern CNNs due to that interaction Key Differences Batch normalization rescales using batch statistics; dropout relies on random masking. Batch normalization introduces learnable scale and shift parameters; dropout adds none. At inference, batch normalization switches to running averages while dropout is simply turned off. Batch normalization mainly targets training stability; dropout mainly targets overfitting. Combining them naively can cause a variance shift that hurts performance. When to Use Each Batch Normalization ...

Overview LSTM and GRU are both gated recurrent architectures built to capture long-range dependencies in sequences while avoiding the vanishing-gradient problem of vanilla RNNs. LSTM keeps a dedicated cell state alongside its hidden state, regulated by three gates, while GRU folds everything into a single hidden state updated by just two gates. That structural difference drives everything else: parameter count, training speed, and how precisely you can control what the network remembers. ...

Overview Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are neural architectures built for different data shapes: CNNs slide convolutional filters across a spatial grid to detect local patterns, while RNNs pass a recurrent hidden state across time steps to model sequential dependencies. Choosing between them (or their modern successors) depends on whether your data’s structure is spatial, temporal, or both. Comparison Diagram CNNConvolution over a spatial gridconvolveFilter weights shared across all positionsCaptures local spatial patternsBest for grid-structured data (images)RNNRecurrence over a sequenceh1h2h3x1x2x3y1y2y3Hidden state carries contextforward through the sequenceBest for sequential/time-ordered data Comparison Table Aspect CNN RNN Input data shape Fixed-size spatial grid (2D/3D tensors like images) Variable-length ordered sequence (text, time series, audio) Core operation Convolution: a filter slides over local receptive fields Recurrence: hidden state updated step-by-step from previous state plus current input Weight sharing Same filter weights reused across all spatial positions Same weight matrices reused across all time steps Context captured Local spatial neighborhoods, expanded via depth/pooling Temporal history accumulated in the hidden state over prior steps Order sensitivity Largely order-invariant beyond local structure; pooling discards exact position Strictly order-dependent; reordering the sequence changes the output Training parallelization Highly parallelizable across positions, channels, and layers Inherently sequential; each step waits on the previous hidden state Common failure mode Limited receptive field unless network is deep or uses dilation Vanishing/exploding gradients over long sequences Typical applications Image classification, object detection, segmentation Language modeling, time-series forecasting, speech recognition Key Differences CNNs assume spatially local structure and share filter weights across the whole input; RNNs share weights across time steps instead. CNN layers process all positions in parallel, while RNNs are sequential by construction since each step needs the prior hidden state. RNNs suffer from vanishing gradients over long sequences; CNNs sidestep this but need deeper stacks to grow their receptive field. Shuffling pixels barely changes what a CNN detects, but reordering a sequence fed to an RNN changes the output entirely, since RNNs are order-sensitive. CNNs expect fixed-size grid inputs, whereas RNNs natively handle variable-length sequences. When to Use Each CNN ...

Overview Deep Learning is technically a subset of Machine Learning, but in practice the two names are used to distinguish classical algorithms from neural-network-based approaches. Traditional Machine Learning relies on humans to hand-engineer features before a model like a decision tree or SVM can learn from them, while Deep Learning uses multi-layer neural networks that learn their own feature representations directly from raw data. The distinction matters because it drives very different requirements for data volume, compute, and interpretability. ...