Machine-Learning

Overview Model quantization and model pruning are both techniques for shrinking neural networks and speeding up inference, but they compress different things. Quantization keeps every weight but represents each one with fewer bits (e.g. FP32 → INT8), while pruning keeps full precision but removes weights, neurons, or channels judged unimportant. The two techniques are complementary and are frequently chained together in a single compression pipeline. Comparison Diagram Model QuantizationBefore: FP32 (32-bit)0.482-1.0370.917-0.203quantizeAfter: INT8 (8-bit)61-132117-26Same 4 values, fewer bits each≈4× smaller, faster mathModel PruningBefore: dense networkpruneAfter: sparse (pruned)Fewer neurons & connectionskeptremoved Comparison Table Aspect Model Quantization Model Pruning Core mechanism Reduces numeric precision of weights/activations (e.g. FP32 → INT8/INT4) Removes individual weights, neurons, or channels judged low-importance What changes Same parameter count, smaller representation per value Fewer parameters; model becomes sparse or physically smaller Granularity Per-tensor, per-channel, or per-group bit-width choices Unstructured (single weights) vs structured (filters/channels/layers) When applied Post-training quantization (PTQ) or quantization-aware training (QAT) Iterative pruning during training or magnitude-based pruning after training, usually with fine-tuning Hardware/runtime requirement Needs low-precision kernel support (INT8 cores, TensorRT, XNNPACK) Unstructured pruning needs sparse-matrix kernels for real speedup; structured pruning runs on standard dense hardware Compression achieved Typically 2-4x size reduction (FP32→INT8); INT4 pushes further at higher accuracy risk Can reach 50-90% sparsity, but unstructured sparsity often doesn’t translate to real speedup without special hardware Accuracy impact & recovery Small accuracy drop, usually recovered via calibration or QAT Larger accuracy drop at high sparsity, recovered via iterative fine-tuning/retraining Combinability Often applied last, to shrink an already-pruned model further Often applied first, before the pruned model is quantized Key Differences Quantization changes each value’s bit-width; pruning changes the model’s parameter count. Realizing pruning’s theoretical speedup often requires sparse kernels, while quantization’s speedup comes from standard INT8 hardware support. Quantization degrades accuracy gradually and predictably; aggressive pruning risks a sharp accuracy cliff without fine-tuning. The two are commonly chained into a single compression pipeline, pruning first and quantizing the result. Structured pruning changes the model’s architecture shape; quantization never touches the architecture. When to Use Each Model Quantization ...

Overview Zero-shot and few-shot learning describe how much task-specific example data a model is given before it has to perform a task. Zero-shot relies solely on a task description, while few-shot conditions its predictions on a small set of labeled examples, usually trading a little setup cost for higher accuracy. Comparison Diagram Zero-ShotFew-ShotTask instructiononly(0 examples)Task instruction+ K examples123PretrainedModelPretrainedModelPredictionPredictionno task-specific datalearns from few examples Comparison Table Aspect Zero-Shot Learning Few-Shot Learning Core definition Model performs a task it was never explicitly shown examples for, guided only by natural-language instructions or class descriptions Model performs a task after being shown a small number (typically 1-100) of labeled examples at inference or fine-tuning time Examples provided at inference None — only a task description or prompt A handful of input-output pairs included in the prompt or used for fine-tuning Underlying mechanism Relies entirely on knowledge encoded during pretraining plus semantic alignment between labels and text Uses in-context learning or lightweight fine-tuning to infer the task pattern directly from the provided examples Labeling/data cost Effectively zero — no labeled data needed for the target task Low but nonzero — requires curating a small, representative set of examples Prompt/context length Short — just the instruction or class names Longer — instruction plus example pairs, consuming more context tokens Typical accuracy Lower and more variable, especially on niche or ambiguous tasks Generally higher and more stable since examples disambiguate intent Sensitivity to example choice Not applicable — there are no examples to choose High — accuracy can swing significantly with example selection, order, and count Common techniques Prompt engineering, CLIP-style embedding matching, instruction-tuned LLMs Few-shot prompting, meta-learning (e.g. MAML), lightweight fine-tuning or LoRA Key Differences Zero-shot uses no task examples at all, relying purely on pretrained knowledge and instructions Few-shot conditions the model on a small support set of labeled examples at inference time Few-shot generally achieves higher accuracy because examples disambiguate an otherwise vague instruction Zero-shot has zero labeling cost, while few-shot requires curating representative examples Few-shot performance is sensitive to example selection, a variable that zero-shot simply doesn’t have When to Use Each Zero-Shot Learning ...

Overview Fine-tuning and RAG (Retrieval-Augmented Generation) are two ways to make a large language model produce better, more relevant answers, but they intervene at different points in the pipeline. Fine-tuning permanently adjusts the model’s weights through additional training, while RAG leaves the model untouched and instead injects context by retrieving documents at query time. The choice matters because it determines how you update knowledge, control latency and cost, and trace where an answer came from. ...

Overview Generative and discriminative models represent two different answers to “what should a model actually learn from labeled data?” A generative model learns the full joint distribution of inputs and labels — effectively how each class produces its data — while a discriminative model learns only the boundary needed to tell classes apart, without modeling how the data itself was produced. That difference drives everything from data efficiency to whether the model can create new examples. ...

Overview BERT and GPT are both transformer-based language models, but they’re built from opposite halves of the transformer and trained for opposite jobs. BERT uses an encoder trained to fill in masked words using context from both directions, making it suited to understanding text, while GPT uses a decoder trained to predict the next word from only what came before, making it suited to generating text. Comparison Diagram BERT GPT Bidirectional Encoder Autoregressive Decoder the cat [MASK] on mat sees full sentence context (left + right) to fill the mask -> classification, embeddings, NER the cat sat on mat ? each token sees only itself + prior tokens (causal mask) -> generation, chat, completion Comparison Table Aspect BERT GPT Architecture Encoder-only transformer stack Decoder-only transformer stack Pretraining objective Masked language modeling: predict randomly hidden tokens, plus next-sentence prediction Causal language modeling: predict the next token given all prior tokens Attention pattern Bidirectional self-attention; every token attends to the full sequence Causal (masked) self-attention; each token attends only to itself and earlier tokens Output generation One contextual embedding per input token, produced in a single forward pass Text generated autoregressively, one token at a time, each output fed back as input Typical adaptation Fine-tuned with a task-specific head on top of the pretrained encoder Adapted via prompting, instruction tuning, or fine-tuning to continue text Primary use cases Classification, named entity recognition, semantic search, sentence embeddings Open-ended generation, chat, code completion, summarization Inference cost per query Fixed: one pass regardless of desired output Scales with number of generated tokens, each requiring a forward pass Key Differences BERT’s encoder attends to both left and right context; GPT’s decoder attends only to prior tokens. BERT trains on masked language modeling; GPT trains on next-token prediction. BERT produces embeddings in a single pass; GPT produces text through autoregressive decoding. BERT is optimized for understanding tasks; GPT is optimized for generation tasks. When to Use Each BERT ...

Overview Overfitting and underfitting describe the two ways a model can fail to generalize: one learns the training data too well, the other not well enough. Understanding which failure mode you’re in determines whether you should simplify or add regularization, or instead increase capacity and train longer. Overfitting traps a model in the noise of its training set, while underfitting leaves it unable to capture the underlying pattern at all. Comparison Diagram OverfittingUnderfittingFits every point exactlycaptures noise, not signalMisses the curved trendtoo simple for the pattern Comparison Table Aspect Overfitting Underfitting Underlying cause Model too complex relative to the data, so it learns noise and idiosyncrasies Model too simple to represent the true relationship in the data Training error Very low, often near zero High, the model struggles even on data it was trained on Validation/test error High, much worse than training error High, similar in magnitude to training error Bias-variance profile Low bias, high variance High bias, low variance Generalization to new data Poor, predictions swing wildly on unseen inputs Poor, predictions are consistently and systematically off Learning curve signature Training and validation loss diverge as training continues Training and validation loss both plateau high and close together Typical remedies Regularization, more training data, dropout, early stopping, simpler model Increase model capacity, add features, train longer, reduce regularization Key Differences Overfitting memorizes noise in the training set, while underfitting never learns the underlying pattern at all. Overfitting shows near-zero training error but a wide train/validation gap, a sign of high variance; underfitting shows poor performance on both, a sign of high bias. Overfitting is treated with regularization or more data; underfitting is treated by increasing model capacity. Overfitting gets worse the longer an overly flexible model keeps training; underfitting persists regardless of duration, since it’s a structural limit. When to Use Each Overfitting ...

Overview Precision and recall are two classification metrics computed from the same confusion matrix but answering different questions about a model’s positive predictions. Precision asks how many predicted positives were correct, while recall asks how many actual positives were found. Optimizing one in isolation almost always trades off against the other. Comparison Diagram Predicted PositiveActual PositiveFPTPFNwrong alarmsmissed casesPrecision = TP / (TP + FP)Recall = TP / (TP + FN) Comparison Table Aspect Precision Recall Question answered Of items predicted positive, how many actually are positive? Of items that are actually positive, how many did the model find? Formula TP / (TP + FP) TP / (TP + FN) Denominator basis Total predicted positive (TP + FP) Total actual positive (TP + FN) Error type penalized False positives (false alarms) False negatives (missed detections) Increases when Model makes fewer incorrect positive calls Model catches more of the true positive cases Threshold trade-off Raising the decision threshold typically raises precision Lowering the decision threshold typically raises recall Failure mode at extreme High precision, low recall: model is overly conservative and misses real cases High recall, low precision: model is overly liberal and floods results with false alarms Key Differences Precision’s denominator is predicted positives; recall’s denominator is actual positives, so they measure against different totals Precision is hurt by false positives; recall is hurt by false negatives Adjusting the classification threshold pushes precision and recall in opposite directions Neither metric alone summarizes model quality, which is why the F1 score combines them A model with 100% recall can trivially predict everyone positive, and a model with 100% precision can trivially predict almost no one positive When to Use Each Precision ...

Overview Bagging and Boosting are both ensemble techniques that combine many weak learners into one stronger model, but they build that ensemble in fundamentally different ways. Bagging trains learners independently in parallel on bootstrap samples and averages their outputs to cut variance, while Boosting trains learners one after another, each one correcting the last model’s mistakes through sequential reweighting to cut bias. The choice affects training time, overfitting risk, and how robust the model is to noisy data. ...

Overview Classification and regression are the two core types of supervised learning, distinguished by what kind of output they predict. Classification assigns inputs to a discrete class, while regression estimates a continuous value. Picking the wrong one for your target variable leads to mismatched loss functions, evaluation metrics, and model outputs. Comparison Diagram ClassificationRegressionClass AClass Boutput: discrete categoryoutput: continuous number Comparison Table Aspect Classification Regression Target variable type Discrete, categorical labels from a finite set of classes Continuous, ordered numeric values Learning objective Learn a decision boundary that separates classes Learn a function mapping inputs to a continuous output Typical loss function Cross-entropy, log loss, or hinge loss Mean squared error or mean absolute error Model output format Class label or probability distribution over classes Single scalar value (or vector of scalars) Common algorithms Logistic regression, SVM, decision trees, kNN, softmax networks Linear regression, ridge/lasso, decision trees, kNN, regression networks Evaluation metrics Accuracy, precision/recall, F1, ROC-AUC, confusion matrix RMSE, MAE, R-squared, MAPE Error interpretation Prediction is simply right, wrong, or confused with another class Prediction error has magnitude and direction, showing how far off it was Key Differences Classification predicts a discrete label from a fixed set of classes, while regression predicts a continuous value on a numeric scale. Classification models typically optimize cross-entropy loss to separate classes, while regression models optimize squared error to minimize distance from the true value. Classification is evaluated with metrics like accuracy/F1, while regression is evaluated with metrics like RMSE/R-squared. A classification error is simply right, wrong, or a class confusion, while a regression error carries a magnitude showing how far off the prediction was. When to Use Each Classification ...

Overview Supervised learning trains a model on labeled data, teaching it to map inputs to known outputs it can later predict. Unsupervised learning works on raw, unlabeled data and instead performs pattern discovery, uncovering structure like clusters or reduced representations with no target to match against. The distinction matters because it determines what data you need, how you measure success, and which problems each approach can actually solve. Comparison Diagram Supervised LearningUnsupervised Learningyyyfeatures + known labelsModelPredicted labelchecked against true yfeatures only, no labelsModelDiscovered clustersno ground truth to check Comparison Table Aspect Supervised Learning Unsupervised Learning Input data Labeled examples: features paired with a known target value Unlabeled examples: features only, no target provided Learning objective Minimize the error between predicted and true labels Discover inherent structure, grouping, or compressed representation in the data Training signal Explicit feedback from a loss function computed against ground truth No explicit feedback; relies on similarity, density, or variance within the data itself Model output A predicted class label or continuous value Cluster assignments, reduced dimensions, or anomaly scores Evaluation Direct measurement on a held-out labeled test set (accuracy, F1, RMSE) Indirect measurement (silhouette score, reconstruction error) or human interpretation Common tasks Classification and regression Clustering, dimensionality reduction, and anomaly detection Data/labeling cost Requires a labeled dataset, often costly and time-consuming to build Uses raw data as-is, cheaper and faster to collect at scale Typical algorithms Logistic regression, random forests, gradient boosting, supervised neural nets k-means, PCA, DBSCAN, autoencoders Key Differences Supervised learning requires labeled data; unsupervised learning works directly on raw data. Supervised models are scored against ground truth; unsupervised models are judged by internal structure metrics instead. Supervised learning targets prediction of a known outcome; unsupervised learning targets discovery of unknown structure. Labeling is usually the bottleneck cost for supervised systems, while unsupervised systems scale with raw data volume. Supervised errors are measurable per-example; unsupervised quality is often assessed via proxy metrics or manual review. When to Use Each Supervised Learning ...