Posts

Overview RNNs process sequences one token at a time, carrying context forward through a hidden state that updates at each step. Transformers instead process every token simultaneously, letting each position directly attend to every other via self-attention. The difference reshapes everything from training speed to how well long-range context survives. Comparison Diagram RNNTransformerx1x2x3x4h1h2h3h4state passed step by stepprocesses one token at a timex1x2x3x4z1z2z3z4every token attends to every tokenprocesses all tokens at once Comparison Table Aspect RNN Transformer Input processing order Tokens consumed one at a time, in sequence All tokens consumed simultaneously Context propagation Hidden state carried forward step to step Self-attention lets each position read all others directly Long-range dependencies Signal weakens over distance (vanishing/exploding gradients) Direct connection between any two positions regardless of distance Positional information Implicit, from the order tokens are fed in Explicit, via added positional encodings Training parallelization Limited — must unroll and step through time Fully parallel across the sequence dimension Computational cost O(n) sequential steps, O(1) state per step O(n^2) attention cost over sequence length Inference/generation Constant memory, naturally one step at a time Requires KV caching to avoid recomputing past attention Typical use cases LSTM/GRU for streaming or small-scale sequence tasks BERT/GPT-style models for large-scale language and vision tasks Key Differences Transformer processes all tokens in parallel via self-attention; RNN processes tokens sequentially through a hidden state RNN suffers from vanishing gradients over long sequences; Transformer links any two positions directly Transformer needs explicit positional encodings since attention has no inherent order; RNN gets order for free Transformer training scales with quadratic complexity in sequence length; RNN training is linear but hard to parallelize When to Use Each RNN ...

Overview Batch normalization and dropout are both inserted between layers of a neural network, but they solve different problems during training. Batch normalization rescales activations using batch statistics to stabilize and speed up training, while dropout randomly zeroes neurons to stop the network from over-relying on any single feature. Comparison Diagram Batch Normalization Dropout raw activations (batch) μ=0, σ=1 normalized, uniform scale recomputed from batch stats every pass pass 1 - random mask pass 2 - different mask random subset silenced each forward pass Comparison Table Aspect Batch Normalization Dropout Insertion point after a linear/conv layer, before the activation function after the activation function, on the layer’s output Core mechanism normalizes activations to zero mean/unit variance using batch statistics, then applies a learnable scale and shift randomly zeroes a fraction p of activations on each forward pass Primary goal stabilize and accelerate training by reducing internal covariate shift reduce overfitting by preventing neurons from co-adapting Learnable/hyperparameters learnable scale (gamma) and shift (beta) per channel; tracks running mean/variance no learnable parameters; single hyperparameter, the dropout rate p Train vs inference behavior uses batch statistics in training, switches to stored running averages at inference active during training, disabled entirely (identity function) at inference Sensitivity to batch size degrades with very small or inconsistent batches since statistics become noisy unaffected by batch size, operates independently per example Combined usage typically placed before dropout; can conflict with dropout’s variance shift often reduced or omitted alongside batch norm in modern CNNs due to that interaction Key Differences Batch normalization rescales using batch statistics; dropout relies on random masking. Batch normalization introduces learnable scale and shift parameters; dropout adds none. At inference, batch normalization switches to running averages while dropout is simply turned off. Batch normalization mainly targets training stability; dropout mainly targets overfitting. Combining them naively can cause a variance shift that hurts performance. When to Use Each Batch Normalization ...

Overview Overfitting and underfitting describe the two ways a model can fail to generalize: one learns the training data too well, the other not well enough. Understanding which failure mode you’re in determines whether you should simplify or add regularization, or instead increase capacity and train longer. Overfitting traps a model in the noise of its training set, while underfitting leaves it unable to capture the underlying pattern at all. Comparison Diagram OverfittingUnderfittingFits every point exactlycaptures noise, not signalMisses the curved trendtoo simple for the pattern Comparison Table Aspect Overfitting Underfitting Underlying cause Model too complex relative to the data, so it learns noise and idiosyncrasies Model too simple to represent the true relationship in the data Training error Very low, often near zero High, the model struggles even on data it was trained on Validation/test error High, much worse than training error High, similar in magnitude to training error Bias-variance profile Low bias, high variance High bias, low variance Generalization to new data Poor, predictions swing wildly on unseen inputs Poor, predictions are consistently and systematically off Learning curve signature Training and validation loss diverge as training continues Training and validation loss both plateau high and close together Typical remedies Regularization, more training data, dropout, early stopping, simpler model Increase model capacity, add features, train longer, reduce regularization Key Differences Overfitting memorizes noise in the training set, while underfitting never learns the underlying pattern at all. Overfitting shows near-zero training error but a wide train/validation gap, a sign of high variance; underfitting shows poor performance on both, a sign of high bias. Overfitting is treated with regularization or more data; underfitting is treated by increasing model capacity. Overfitting gets worse the longer an overly flexible model keeps training; underfitting persists regardless of duration, since it’s a structural limit. When to Use Each Overfitting ...

Overview Precision and recall are two classification metrics computed from the same confusion matrix but answering different questions about a model’s positive predictions. Precision asks how many predicted positives were correct, while recall asks how many actual positives were found. Optimizing one in isolation almost always trades off against the other. Comparison Diagram Predicted PositiveActual PositiveFPTPFNwrong alarmsmissed casesPrecision = TP / (TP + FP)Recall = TP / (TP + FN) Comparison Table Aspect Precision Recall Question answered Of items predicted positive, how many actually are positive? Of items that are actually positive, how many did the model find? Formula TP / (TP + FP) TP / (TP + FN) Denominator basis Total predicted positive (TP + FP) Total actual positive (TP + FN) Error type penalized False positives (false alarms) False negatives (missed detections) Increases when Model makes fewer incorrect positive calls Model catches more of the true positive cases Threshold trade-off Raising the decision threshold typically raises precision Lowering the decision threshold typically raises recall Failure mode at extreme High precision, low recall: model is overly conservative and misses real cases High recall, low precision: model is overly liberal and floods results with false alarms Key Differences Precision’s denominator is predicted positives; recall’s denominator is actual positives, so they measure against different totals Precision is hurt by false positives; recall is hurt by false negatives Adjusting the classification threshold pushes precision and recall in opposite directions Neither metric alone summarizes model quality, which is why the F1 score combines them A model with 100% recall can trivially predict everyone positive, and a model with 100% precision can trivially predict almost no one positive When to Use Each Precision ...

Overview Bagging and Boosting are both ensemble techniques that combine many weak learners into one stronger model, but they build that ensemble in fundamentally different ways. Bagging trains learners independently in parallel on bootstrap samples and averages their outputs to cut variance, while Boosting trains learners one after another, each one correcting the last model’s mistakes through sequential reweighting to cut bias. The choice affects training time, overfitting risk, and how robust the model is to noisy data. ...

Overview LSTM and GRU are both gated recurrent architectures built to capture long-range dependencies in sequences while avoiding the vanishing-gradient problem of vanilla RNNs. LSTM keeps a dedicated cell state alongside its hidden state, regulated by three gates, while GRU folds everything into a single hidden state updated by just two gates. That structural difference drives everything else: parameter count, training speed, and how precisely you can control what the network remembers. ...

Overview Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are neural architectures built for different data shapes: CNNs slide convolutional filters across a spatial grid to detect local patterns, while RNNs pass a recurrent hidden state across time steps to model sequential dependencies. Choosing between them (or their modern successors) depends on whether your data’s structure is spatial, temporal, or both. Comparison Diagram CNNConvolution over a spatial gridconvolveFilter weights shared across all positionsCaptures local spatial patternsBest for grid-structured data (images)RNNRecurrence over a sequenceh1h2h3x1x2x3y1y2y3Hidden state carries contextforward through the sequenceBest for sequential/time-ordered data Comparison Table Aspect CNN RNN Input data shape Fixed-size spatial grid (2D/3D tensors like images) Variable-length ordered sequence (text, time series, audio) Core operation Convolution: a filter slides over local receptive fields Recurrence: hidden state updated step-by-step from previous state plus current input Weight sharing Same filter weights reused across all spatial positions Same weight matrices reused across all time steps Context captured Local spatial neighborhoods, expanded via depth/pooling Temporal history accumulated in the hidden state over prior steps Order sensitivity Largely order-invariant beyond local structure; pooling discards exact position Strictly order-dependent; reordering the sequence changes the output Training parallelization Highly parallelizable across positions, channels, and layers Inherently sequential; each step waits on the previous hidden state Common failure mode Limited receptive field unless network is deep or uses dilation Vanishing/exploding gradients over long sequences Typical applications Image classification, object detection, segmentation Language modeling, time-series forecasting, speech recognition Key Differences CNNs assume spatially local structure and share filter weights across the whole input; RNNs share weights across time steps instead. CNN layers process all positions in parallel, while RNNs are sequential by construction since each step needs the prior hidden state. RNNs suffer from vanishing gradients over long sequences; CNNs sidestep this but need deeper stacks to grow their receptive field. Shuffling pixels barely changes what a CNN detects, but reordering a sequence fed to an RNN changes the output entirely, since RNNs are order-sensitive. CNNs expect fixed-size grid inputs, whereas RNNs natively handle variable-length sequences. When to Use Each CNN ...

Overview Classification and regression are the two core types of supervised learning, distinguished by what kind of output they predict. Classification assigns inputs to a discrete class, while regression estimates a continuous value. Picking the wrong one for your target variable leads to mismatched loss functions, evaluation metrics, and model outputs. Comparison Diagram ClassificationRegressionClass AClass Boutput: discrete categoryoutput: continuous number Comparison Table Aspect Classification Regression Target variable type Discrete, categorical labels from a finite set of classes Continuous, ordered numeric values Learning objective Learn a decision boundary that separates classes Learn a function mapping inputs to a continuous output Typical loss function Cross-entropy, log loss, or hinge loss Mean squared error or mean absolute error Model output format Class label or probability distribution over classes Single scalar value (or vector of scalars) Common algorithms Logistic regression, SVM, decision trees, kNN, softmax networks Linear regression, ridge/lasso, decision trees, kNN, regression networks Evaluation metrics Accuracy, precision/recall, F1, ROC-AUC, confusion matrix RMSE, MAE, R-squared, MAPE Error interpretation Prediction is simply right, wrong, or confused with another class Prediction error has magnitude and direction, showing how far off it was Key Differences Classification predicts a discrete label from a fixed set of classes, while regression predicts a continuous value on a numeric scale. Classification models typically optimize cross-entropy loss to separate classes, while regression models optimize squared error to minimize distance from the true value. Classification is evaluated with metrics like accuracy/F1, while regression is evaluated with metrics like RMSE/R-squared. A classification error is simply right, wrong, or a class confusion, while a regression error carries a magnitude showing how far off the prediction was. When to Use Each Classification ...

Overview Supervised learning trains a model on labeled data, teaching it to map inputs to known outputs it can later predict. Unsupervised learning works on raw, unlabeled data and instead performs pattern discovery, uncovering structure like clusters or reduced representations with no target to match against. The distinction matters because it determines what data you need, how you measure success, and which problems each approach can actually solve. Comparison Diagram Supervised LearningUnsupervised Learningyyyfeatures + known labelsModelPredicted labelchecked against true yfeatures only, no labelsModelDiscovered clustersno ground truth to check Comparison Table Aspect Supervised Learning Unsupervised Learning Input data Labeled examples: features paired with a known target value Unlabeled examples: features only, no target provided Learning objective Minimize the error between predicted and true labels Discover inherent structure, grouping, or compressed representation in the data Training signal Explicit feedback from a loss function computed against ground truth No explicit feedback; relies on similarity, density, or variance within the data itself Model output A predicted class label or continuous value Cluster assignments, reduced dimensions, or anomaly scores Evaluation Direct measurement on a held-out labeled test set (accuracy, F1, RMSE) Indirect measurement (silhouette score, reconstruction error) or human interpretation Common tasks Classification and regression Clustering, dimensionality reduction, and anomaly detection Data/labeling cost Requires a labeled dataset, often costly and time-consuming to build Uses raw data as-is, cheaper and faster to collect at scale Typical algorithms Logistic regression, random forests, gradient boosting, supervised neural nets k-means, PCA, DBSCAN, autoencoders Key Differences Supervised learning requires labeled data; unsupervised learning works directly on raw data. Supervised models are scored against ground truth; unsupervised models are judged by internal structure metrics instead. Supervised learning targets prediction of a known outcome; unsupervised learning targets discovery of unknown structure. Labeling is usually the bottleneck cost for supervised systems, while unsupervised systems scale with raw data volume. Supervised errors are measurable per-example; unsupervised quality is often assessed via proxy metrics or manual review. When to Use Each Supervised Learning ...

Overview Deep Learning is technically a subset of Machine Learning, but in practice the two names are used to distinguish classical algorithms from neural-network-based approaches. Traditional Machine Learning relies on humans to hand-engineer features before a model like a decision tree or SVM can learn from them, while Deep Learning uses multi-layer neural networks that learn their own feature representations directly from raw data. The distinction matters because it drives very different requirements for data volume, compute, and interpretability. ...