P

Preface

Why inference engineering has become a discipline of its own.

4 min read

Printed pages: 9–14

Star on GitHub

The book's argument in one minute

Inference engineering is the work of turning a capable generative model into a fast, affordable, and dependable product service. It crosses boundaries that are often owned by separate teams: model architecture, GPU programming, serving runtimes, distributed systems, and production operations. The preface argues that this combination is no longer a narrow specialty reserved for a few frontier laboratories.

Why the field expanded

  • In the early generative AI market, production inference expertise was concentrated in model laboratories and large technology companies because they trained and served most of the useful models.
  • A rapidly growing supply of downloadable model weights gave ordinary product teams the option to deploy and adapt models themselves instead of consuming every capability through a closed application programming interface.
  • The practical distinction is access to weights: a closed model withholds them, while an open model makes them available under a license whose commercial and redistribution conditions still require careful review.
  • Closed and open models can both improve quickly. The engineering opportunity appears when an available model crosses the quality threshold for a particular workflow and the team can now optimize the rest of the system around it.

From benchmark progress to product capability

A product rarely needs the universally highest-scoring model. It needs a model that clears the task-specific quality bar while satisfying the system's other constraints. As models improve, they cross thresholds that make new experiences possible; once an open model crosses the same threshold, a builder gains far more control over how the capability is delivered.

  1. Define the behavior that makes the product useful and the evaluation that proves the model is good enough for that behavior.
  2. Choose a model that clears the quality threshold rather than treating a general leaderboard position as the product requirement.
  3. When useful, adapt an open model to the domain so a smaller or more controllable system can meet the same task bar.
  4. Optimize the serving path only after the quality bar and workload are explicit, because the best latency and cost choices depend on both.

What inference engineering changes

Latency

Interaction speed

A dedicated deployment can be shaped around real-time behavior instead of a shared service's throughput policy.

Availability

Failure control

Owning deployment topology and failover creates room to target stronger service continuity.

Cost

Unit economics

At sufficient scale, model and serving choices can materially reduce the cost of each useful result.

Control

Differentiation

Weights, runtime, hardware, and operations become product design variables rather than fixed properties of an external endpoint.

How to use this distilled edition

  1. Start with the product constraint. Chapters 0 and 1 define the inference stack, model-selection frame, workload, and user-facing metrics.
  2. Build the mechanism model. Chapters 2 through 4 connect model architecture to hardware behavior, kernels, runtimes, benchmarks, and profiles.
  3. Choose levers by bottleneck. Chapters 5 and 6 organize optimization techniques and show why text, vision, audio, and video require different pipelines.
  4. Close the loop in production. Chapter 7 turns optimized execution into a capacity-managed, observable, secure, and evolvable service.

Read each chapter as a decision guide, not a list of fashionable tools. Keep a workload trace, a quality evaluation, and a performance baseline nearby. For every proposed technique, state the suspected bottleneck, the metric that should move, the quality or reliability risk, and the rollback condition.

Source map

  • Open-model shift

    Printed pages 9 to 11 describe the expanding supply of models and distinguish open from closed weights.

  • Capability thresholds

    Printed pages 11 to 12 connect model progress and customization to newly viable products.

  • Product dimensions

    Printed pages 12 to 13 motivate latency, availability, cost, and differentiation as serving concerns.

  • Learning invitation

    Printed pages 13 to 14 frame inference engineering as an early, cross-stack discipline grounded in operating experience.

The rise of open-weight models and the widening audience for inference engineering
Printed pages: 9–11; PDF pages: 11–13
Capability thresholds, product categories, and model customization
Printed pages: 11–12; PDF pages: 13–14
Latency, availability, cost, and product differentiation
Printed pages: 12–13; PDF pages: 14–15
The emerging discipline, intended audience, and source of the book's practical perspective
Printed pages: 13–14; PDF pages: 15–16