Open-source · Local AI inference
An inference engine for developers who want to understand, optimize, and control how language models run.
The background illustration is decorative and does not show live engine activity.
01 — Premise
Between a prompt and a token sits a chain of decisions: which device executes each operation, how tensors are laid out in memory, when a waiting request joins a running batch, and which cached state can be reused.
DynaLM keeps those decisions visible. It is built in layers you can read, measure and change — from the serving scheduler down to the kernels.
02 — Architecture
Each layer has a single responsibility and a stated maturity. Select a layer to see what it owns.
Maturity reflects the repository on the snapshot date. DynaKernel is experimental research code.
The engine people run. Loads models, schedules requests, batches decode steps and serves the OpenAI-compatible HTTP API and CLI.
The device abstraction beneath the runtime. Owns tensor storage, memory mapping and the SIMD CPU kernels, and exposes GPU devices through one interface. CPU stable · GPU backends in preview.
An intermediate representation of the model forward pass. Used to plan execution, fuse operations and drive the compiled kernel mode.
Experimental kernel code generation, including an LLVM-based JIT path. Not required to run models and not part of any stability promise.
03 — Capabilities
3.1
Choose the device per run or let the engine pick: GPU first, CPU when the GPU cannot run the model. Execution modes trade speed against a stated accuracy contract.
3.2
Requests join a running batch between decode steps instead of waiting for it to drain. Chunked prefill keeps long prompts from stalling active streams.
3.3
Attention state lives in fixed-size pages from a shared, reference-counted pool. Requests with a common prefix share the same pages.
3.4
One native binary with an OpenAI-compatible API, so existing SDKs and tools connect by changing the base URL.
/v1/chat/completions/v1/completions/v1/models/health/metricshttp://127.0.0.1:8000/v104 — Quick start
Commands as documented in the project README. Check the installation guide for Windows, Docker and source builds.
Install & run
# Install (Linux and macOS)
curl -fsSL https://raw.githubusercontent.com/dynalm-project/dynalm/main/scripts/install.sh | sh
# Download a model and chat with it
dynalm pull qwen2.5:1.5b
dynalm run qwen2.5:1.5bServe
# Start a local OpenAI-compatible server
dynalm serve qwen2.5:1.5b # http://127.0.0.1:8000/v1
# Check what your machine supports
dynalm doctorPython client
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")
reply = client.chat.completions.create(
model="qwen2.5:1.5b",
messages=[{"role": "user", "content": "Give me three tips for clean code."}],
)
print(reply.choices[0].message.content)05 — Benchmarks
Performance numbers will appear here only with the hardware, model, quantization, backend and method that produced them.
Reproducible performance results are being prepared.
Every number on this page will come with its hardware, model, quantization, backend, method and reproduction commands. Until then, nothing is shown.
Decode throughput
—tok/s: not yet measured
Output tokens generated per second for one stream, after the first token.
↑ higher is betterAwaiting run
Prefill throughput
—tok/s: not yet measured
Prompt tokens processed per second before generation starts.
↑ higher is betterAwaiting run
Time to first token
—ms: not yet measured
Median delay from request arrival to the first streamed token.
↓ lower is betterAwaiting run
Multi-user throughput
—tok/s: not yet measured
Total output tokens per second across all concurrent requests.
↑ higher is betterAwaiting run
Inter-token latency P95 / P99
—ms: not yet measured
99th-percentile gap between consecutive tokens — the stutter users notice.
↓ lower is betterAwaiting run
Memory usage
—MB: not yet measured
Peak resident memory of the server process, model loaded.
↓ lower is betterAwaiting run
Every result will state
Early measurements are documented in the repository: docs/benchmarks.md (opens in a new tab)
07 — Roadmap
Status copied from the repository. No dates, no percentages — only what is known.
Explore the engine. Follow the work. Help shape what comes next.