Appendix · Module 9

Overview of Training

How a raw next-token predictor becomes a helpful, aligned assistant — and how its weights get squeezed for cheap inference. We walk the pre-train → fine-tune → align pipeline, open up the RLHF loop, show what quantization does to a weight, and meet LoRA.

From base model to aligned assistant

step-through

The training pipeline: pre-training learns a base model, then post-training (SFT then preference alignment) shapes it into something useful.

data / tokens weights / updates preferred response dispreferred response

Pre-training is self-supervised: the only label is the next token in the corpus, so it scales to huge text dumps. The resulting base model can continue text but isn't trying to be helpful. Post-training — first SFT, then preference alignment — is what turns it into a chatbot that produces the kinds of answers humans actually want.

RLHF: learning from preferences

step-through

Humans compare candidate responses → those comparisons train a reward model → the policy is optimized to score high while a KL leash keeps it near the reference.

chosen (preferred) rejected policy / reward weights prompts / responses

RLAIF When the scoring model is trained from AI feedback instead of human labels — e.g. a stronger model judging responses against a written constitution — the same loop becomes Reinforcement Learning from AI Feedback. The mechanism is identical; only the source of the preference labels changes.

Quantization: fewer bits per weight

auto-play

Continuous high-precision weights (FP16/BF16) get snapped onto a small grid of integer levels (INT8 → INT4). The model shrinks and runs faster; accuracy slips a little.

true FP16 weight quant level / precision memory + throughput win rounding error

Quantization trades a little accuracy for a smaller footprint, so it's mostly an inference optimization. Calibrated methods like GPTQ and AWQ use sample data — AWQ keeps the most important weights at higher precision — while HQQ and k-quants reduce weight error without any calibration data.

LoRA: parameter-efficient fine-tuning

step-through

Freeze the giant pre-trained weight matrix; train two small low-rank matrices beside it. Fewer than ~1% of the parameters move — and they can be merged back at deploy time.

frozen pre-trained W trainable LoRA A·B quantized W (QLoRA)

QLoRA You can combine both tricks: QLoRA fine-tunes a quantized copy of the frozen base model while keeping the small LoRA matrices in full precision — cutting fine-tuning memory enough to adapt very large models on modest hardware.