Architecture
How a token is made.
DynaLM is organised as four layers with narrow interfaces between them. This page describes what each layer owns and how a request moves through them. It is an overview, not an API reference, and some paths described here are previews.
System
Layered by responsibility.
Select a layer to highlight it and its connections. Each links to its section below.
- StableReady for everyday use.
- PreviewWorks and passes the test suite. Feedback welcome.
- ExperimentalResearch code. Interfaces and results may change.
- PlannedIntended, not started or not yet usable.
-
The engine people run. Loads models, schedules requests, batches decode steps and serves the OpenAI-compatible HTTP API and CLI.
- CLI: pull · run · serve
- Continuous batching
- Paged KV cache
- HTTP API & metrics
-
The device abstraction beneath the runtime. Owns tensor storage, memory mapping and the SIMD CPU kernels, and exposes GPU devices through one interface. CPU stable · GPU backends in preview.
- Device abstraction
- Tensor & memory layout
- AVX2 · AVX-512 · NEON kernels
- Metal · Vulkan · CUDA devices
-
An intermediate representation of the model forward pass. Used to plan execution, fuse operations and drive the compiled kernel mode.
- Graph representation
- Execution planning
- Operation fusion
- Compiled mode
-
DynaKernel, Low-level kernel generation
ExperimentalExperimental kernel code generation, including an LLVM-based JIT path. Not required to run models and not part of any stability promise.
- Kernel codegen
- LLVM JIT backend
- Compact weight layouts
- Kernel benchmarks
DynaLMRuntime · serving · scheduling
The engine people run. Loads models, schedules requests, batches decode steps and serves the OpenAI-compatible HTTP API and CLI.
Responsibilities
- Model registry, download and loading
- Request admission and scheduling
- Continuous batching and chunked prefill
- Paged KV cache and prefix sharing
- Sampling and streaming output
- OpenAI-compatible HTTP API, CLI and Prometheus metrics
The runtime is the stable surface most people use. Its CLI and HTTP API are documented in the repository.
CPU stable · GPU backends in preview
DynaCoreDevices · tensors · memory · CPU kernels
The device abstraction beneath the runtime. Owns tensor storage, memory mapping and the SIMD CPU kernels, and exposes GPU devices through one interface.
Responsibilities
- Device abstraction for CPU and GPU
- Tensor storage and memory mapping
- SIMD CPU kernels with runtime feature detection
- GPU device layer shared by Metal, Vulkan and CUDA
CPU execution is stable on the listed platforms. GPU backends are previews: models a GPU cannot run fall back to the CPU under --backend auto.
DynaIRInference graph · execution planning
An intermediate representation of the model forward pass. Used to plan execution, fuse operations and drive the compiled kernel mode.
Responsibilities
- Graph representation of the forward pass
- Execution planning and scheduling of operations
- Operation fusion
- Input to the compiled kernel mode
Placeholder description. The interfaces of this layer are still evolving; consult the source before depending on them.
DynaKernelLow-level kernel generation
Experimental kernel code generation, including an LLVM-based JIT path. Not required to run models and not part of any stability promise.
Responsibilities
- Kernel program representation and lowering
- Code generation, including an LLVM-based JIT
- Compact weight layouts
- Kernel micro-benchmarks
Experimental. Not required to run models, not enabled by default, and outside any stability promise.
Execution
CPU and GPU execution
With --backend auto, DynaLM tries the GPU first and uses the CPU when the GPU backend cannot run the model. --cpu and --gpu force a choice; --gpu reports why instead of falling back.
GPU Preview
- Metal
- Vulkan
- CUDA
CPU Stable
- AVX2
- AVX-512 / AMX
- NEON
- Portable fallback
Execution modes --mode
auto- Default. Picks the fastest validated path for the device.
exact- Keeps DynaLM’s own accuracy contract on every device; slower on the CPU.
safe- Plain fp32 CPU kernels, for debugging.
Loading
Model loading path
Placeholder overview of how a model file becomes a runnable graph. Exact behaviour depends on the format and is documented in the repository.
- ResolveA catalog name, Hugging Face path or local file.
- Read metadataArchitecture, tensor table and tokenizer from GGUF or SafeTensors.
- ValidateUnsupported tensor types are refused at load, with the tensor named.
- MapWeights are memory-mapped; GPTQ and AWQ weights are repacked.
- PrepareTokenizer and chat template are recognised by model family.
- ReadyThe model is placed on the selected device and can accept requests.
Scheduling
Scheduler and batching
Requests are admitted between decode steps, so a new request does not wait for running ones to finish. Long prompts are prefilled in chunks interleaved with decoding to limit latency spikes for active streams.
- Continuous batching
- Chunked prefill
- Speculative decoding with a draft model
Memory
KV cache and prefix cache
Attention keys and values are stored in fixed-size pages from a shared, reference-counted pool. Sequences with a common prefix — a system prompt, a chat history — point at the same pages; a page is copied only when a sequence writes to it.
- Paged allocation from a block pool
- Prefix (radix) sharing
- Copy-on-write pages
- Optional Q8_0 pages to halve KV memory
Lifecycle
From request to token
The path of one request through the engine. Simplified; error handling and cancellation are omitted.
- ReceiveHTTP request or CLI prompt arrives.
- TokenizeChat template applied; text becomes tokens.
- ReuseLongest cached prefix is matched to existing pages.
- AdmitScheduler adds the request to the running batch.
- PrefillRemaining prompt tokens are processed, in chunks.
- DecodeOne token per step for every sequence in the batch.
- StreamEach sampled token is streamed to the client.
- ReleaseOn completion, unshared pages return to the pool.
This overview is maintained alongside the code. For exact behaviour, read the source.