Modalities

A modality changes more than the payload. A vision language model (VLM) converts images or video into visual tokens before language decode. An embedding model emits one vector instead of a sequence. Automatic speech recognition (ASR) maps audio to text, while text to speech (TTS) adds an audio-token decoder and waveform stage. Image and video generators usually refine noise through repeated denoising rather than autoregressive token generation. The right optimization follows the pipeline: reduce visual context, batch embeddings, orchestrate speech chunks, tune diffusion kernels, or distribute video attention. The right metric follows the user-visible unit, not a universal token counter.

Trace every request as conversions between representations. Record the raw input, preprocessing, model-native representation, dominant execution loop, postprocessing, transport, and the first useful output. Then attach a resource and metric to each boundary. This reveals whether to optimize a model kernel, remove an unnecessary conversion, parallelize independent chunks, or change the product protocol.

For visual understanding, the encoder patches and embeds images or sampled frames, then the language model processes those visual tokens with the prompt. Resolution and frame rate buy detail but expand prefill and the key-value cache. Process a coherent short clip in one call when motion matters; downsample enough to fit it, transcribe audio separately when required, and reuse repeated visual prefixes across follow-up questions.

For embeddings, separate the bulk lane from the lookup lane. Backfills favor large request batches, broad queues, and horizontal replicas. User-facing retrieval favors bounded queues and low tail latency. BERT-style encoder models remain useful for simple latency-sensitive work, while LLM-derived embedding models generally offer stronger capability. Output dimensionality mostly changes downstream vector storage, transfer, and similarity cost rather than model runtime, while vectors from different model spaces are not interchangeable. Migration therefore requires re-embedding the corpus and evaluating retrieval, not merely matching dimensions.

For live transcription, voice activity detection (VAD) segments one continuous audio stream into ordinary ASR calls and text returns over the same connection. Sequential placement can pass prior text as context. Long files invert the priority: remove silence, make meaningful chunks, distribute them across replicas, then stitch by timestamp. Track real-time factor (RTF), inspect suspicious repetition or implausible speaking rates, and retry affected chunks with a changed temperature or segmentation. Diarization is a separate segmentation-embedding-clustering pipeline rather than Whisper-like decoding; profile it independently because the edition reports even optimized diarization taking at least twice as long as transcription.

For speech synthesis, autoregressive generation produces audio tokens and a separate decoder converts them into a waveform. Optimize and batch both stages, but do not assume the decoder supports the engine's token-level scheduler. Once generation is faster than real-time playback, additional speed does not help one listener; spend the gain on more simultaneous streams. A cascaded voice product also budgets VAD, ASR, language reasoning, TTS, and network turns as one conversation loop. For speech-to-speech, this edition finds no commercially viable open model and describes closed alternatives as less capable and more expensive than cascaded systems; revalidate that rapidly changing trade-off.

For image diffusion, completion time grows with denoising passes. Memory-efficient attention and fusion must first expose the compute-bound path; lower-precision matrix work can then use faster tensor hardware. Quality-speed policy can reduce steps or stop prompt guidance after early composition is established. Video extends the latent through time, making attention the dominant cost. It commonly uses batch one across a full node, cached intermediate work, selective attention precision, and context parallelism instead of ordinary throughput batching.

Keep a modality-specific quality gate beside performance. Compare embedding vectors and retrieval outcomes after precision changes; inspect transcription errors and retry heuristics by audio condition; evaluate speech naturalness and intelligibility; use human preference for image and video changes because automated visual grading is directional. An optimization is valid only when its quality loss stays inside the product's declared tolerance. Run the gate through production preprocessing and postprocessing, and retain baseline outputs plus failure cases so later engine, precision, or hardware changes can replay the same judgment.

A useful benchmark matrix follows the variables the product actually exposes. For a visual assistant, cross image count, resolution, sampled frames, prompt length, cache state, and response length. For speech, cross language, noise, speaker pace, chunk size, silence, and simultaneous streams. For diffusion, cross resolution, clip duration, step schedule, guidance, precision, and GPU topology. Warm each path, repeat enough runs to see normal variation, and preserve raw distributions, resource traces, and representative outputs. This prevents a configuration tuned for one clean sample from becoming a claim about the entire modality, and lets a later rerun distinguish traffic drift from runtime or quality-policy regression.

Capacity planning for a composed system starts at the slowest stage, then follows variability upstream. If a voice activity detector can segment hundreds of streams but transcription handles tens, unrestricted admission only builds a transcription queue and amplifies overload. If speech synthesis produces audio faster than clients consume it, buffers, network egress, and connection slots become capacity. Assign every stage a measured service rate, finite buffer, timeout, retry budget, and backpressure action. Scale stages independently, but keep request identity, configuration, cancellation, and deadline visible across them so nested retries do not multiply expensive work or return a late result that the user has already abandoned.

Roll out modality changes by population, not only by aggregate averages. A lower visual resolution may preserve ordinary scenes while losing small text; a transcription chunk policy may work for quiet English and fail on noisy multilingual audio; a diffusion precision change may preserve composition but damage faces or motion. Route a controlled slice to the candidate, retain the baseline output when practical, and tag results with input traits, model revision, runtime, precision, and configuration. Promote only when tail latency, sustainable throughput, failure rate, unit cost, and the relevant human or task quality gate pass for each important slice; otherwise roll back or narrow eligibility instead of hiding the regression in an average.

Choose service metrics after identifying the model's generation archetype and useful output unit. Autoregressive models extend a tokenized sequence, while diffusion models iteratively denoise an output. The first family includes vision-language, embedding, automatic speech recognition, and text-to-speech models; image and video generation generally follow the second. Many LLM engines and techniques transfer to related autoregressive modalities, but diffusion optimization follows a different path. Latency, throughput, and quality must then be expressed in terms the modality's user can perceive. For TTS, one audio token is not meaningful on its own, so measure time to first word or first sentence rather than time to first token.