Nlp on IT Comparison

Zero-Shot Learning vs Few-Shot Learning: No Examples vs a Handful of Examples

Mon, 03 Aug 2026 03:45:54 +0900

Overview

Zero-shot and few-shot learning describe how much task-specific example data a model is given before it has to perform a task. Zero-shot relies solely on a task description, while few-shot conditions its predictions on a small set of labeled examples, usually trading a little setup cost for higher accuracy.

Comparison Diagram

Comparison Table

Aspect	Zero-Shot Learning	Few-Shot Learning
Core definition	Model performs a task it was never explicitly shown examples for, guided only by natural-language instructions or class descriptions	Model performs a task after being shown a small number (typically 1-100) of labeled examples at inference or fine-tuning time
Examples provided at inference	None — only a task description or prompt	A handful of input-output pairs included in the prompt or used for fine-tuning
Underlying mechanism	Relies entirely on knowledge encoded during pretraining plus semantic alignment between labels and text	Uses in-context learning or lightweight fine-tuning to infer the task pattern directly from the provided examples
Labeling/data cost	Effectively zero — no labeled data needed for the target task	Low but nonzero — requires curating a small, representative set of examples
Prompt/context length	Short — just the instruction or class names	Longer — instruction plus example pairs, consuming more context tokens
Typical accuracy	Lower and more variable, especially on niche or ambiguous tasks	Generally higher and more stable since examples disambiguate intent
Sensitivity to example choice	Not applicable — there are no examples to choose	High — accuracy can swing significantly with example selection, order, and count
Common techniques	Prompt engineering, CLIP-style embedding matching, instruction-tuned LLMs	Few-shot prompting, meta-learning (e.g. MAML), lightweight fine-tuning or LoRA

Key Differences

Zero-shot uses no task examples at all, relying purely on pretrained knowledge and instructions
Few-shot conditions the model on a small support set of labeled examples at inference time
Few-shot generally achieves higher accuracy because examples disambiguate an otherwise vague instruction
Zero-shot has zero labeling cost, while few-shot requires curating representative examples
Few-shot performance is sensitive to example selection, a variable that zero-shot simply doesn’t have

When to Use Each

Zero-Shot Learning

BERT vs GPT: Bidirectional Understanding vs Autoregressive Generation

Mon, 03 Aug 2026 03:42:01 +0900

Overview

BERT and GPT are both transformer-based language models, but they’re built from opposite halves of the transformer and trained for opposite jobs. BERT uses an encoder trained to fill in masked words using context from both directions, making it suited to understanding text, while GPT uses a decoder trained to predict the next word from only what came before, making it suited to generating text.

Comparison Diagram

Comparison Table

Aspect	BERT	GPT
Architecture	Encoder-only transformer stack	Decoder-only transformer stack
Pretraining objective	Masked language modeling: predict randomly hidden tokens, plus next-sentence prediction	Causal language modeling: predict the next token given all prior tokens
Attention pattern	Bidirectional self-attention; every token attends to the full sequence	Causal (masked) self-attention; each token attends only to itself and earlier tokens
Output generation	One contextual embedding per input token, produced in a single forward pass	Text generated autoregressively, one token at a time, each output fed back as input
Typical adaptation	Fine-tuned with a task-specific head on top of the pretrained encoder	Adapted via prompting, instruction tuning, or fine-tuning to continue text
Primary use cases	Classification, named entity recognition, semantic search, sentence embeddings	Open-ended generation, chat, code completion, summarization
Inference cost per query	Fixed: one pass regardless of desired output	Scales with number of generated tokens, each requiring a forward pass

Key Differences

BERT’s encoder attends to both left and right context; GPT’s decoder attends only to prior tokens.
BERT trains on masked language modeling; GPT trains on next-token prediction.
BERT produces embeddings in a single pass; GPT produces text through autoregressive decoding.
BERT is optimized for understanding tasks; GPT is optimized for generation tasks.

When to Use Each

BERT

Transformer vs RNN: Parallel Attention vs Sequential Recurrence

Mon, 03 Aug 2026 03:40:07 +0900

Overview

RNNs process sequences one token at a time, carrying context forward through a hidden state that updates at each step. Transformers instead process every token simultaneously, letting each position directly attend to every other via self-attention. The difference reshapes everything from training speed to how well long-range context survives.

Comparison Diagram

Comparison Table

Aspect	RNN	Transformer
Input processing order	Tokens consumed one at a time, in sequence	All tokens consumed simultaneously
Context propagation	Hidden state carried forward step to step	Self-attention lets each position read all others directly
Long-range dependencies	Signal weakens over distance (vanishing/exploding gradients)	Direct connection between any two positions regardless of distance
Positional information	Implicit, from the order tokens are fed in	Explicit, via added positional encodings
Training parallelization	Limited — must unroll and step through time	Fully parallel across the sequence dimension
Computational cost	O(n) sequential steps, O(1) state per step	O(n^2) attention cost over sequence length
Inference/generation	Constant memory, naturally one step at a time	Requires KV caching to avoid recomputing past attention
Typical use cases	LSTM/GRU for streaming or small-scale sequence tasks	BERT/GPT-style models for large-scale language and vision tasks

Key Differences

Transformer processes all tokens in parallel via self-attention; RNN processes tokens sequentially through a hidden state
RNN suffers from vanishing gradients over long sequences; Transformer links any two positions directly
Transformer needs explicit positional encodings since attention has no inherent order; RNN gets order for free
Transformer training scales with quadratic complexity in sequence length; RNN training is linear but hard to parallelize

When to Use Each

RNN