bankML

bankML in one paragraph

bankML runs language models on the computer you already have: a laptop, a small server, no graphics card needed. It specialises in 1-bit and ternary models, which store each weight in one or two bits instead of sixteen. An 8-billion-parameter model then fits in 1.2 GB (1-bit) or 2.3 GB (ternary). bankML gives exactly the same answers as llama.cpp, the reference engine, and checks that it does, bit for bit, before it is allowed to be faster. Every answer carries a receipt that says which model file produced it.

the full story: README · the technical report: TECHNICAL.md

The advantages

1. It is fast where it matters most

llama.cpp has no fast x86 code for ternary weights; it falls back to plain C. bankML wrote the missing kernel. Each matrix is 9.4–10× faster, and whole answers on the ternary model come out at about 8× llama-server's speed (2.3–2.4 tokens/s against 0.30 on the same laptop). The ternary model gives the better answers of the two, so this is the speed that counts.

PERFORMANCE.md · the kernel: modules/q2_0.md, bankML/q2_0.rs

2. It is exact, and proves it

Speed only counts if the numbers are the same. Every kernel, the tokenizer, the chat templates, the samplers and whole conversations are checked against llama.cpp b11192's own compiled code. These checks are called oracles, and a release ships only when every oracle passes in its release gate.

oracles.md · the gate: testing/release_gate.sh and its records in testing/results/

3. It refuses to answer from a model it cannot verify

Before a model answers, three gates run. A guard reads the file's header and refuses known-bad files. A pin checks the file's sha256 against the record of where it came from. The oracle, run ahead of time, has proven the arithmetic. Each answer then carries a receipt with the model's sha256 and the hashes of the request and the answer.

TECHNICAL.md §III.1 · receipts (usage.md §9) · the guard: bankML/gguf.rs · the pin: bankML/sha256.rs

4. It is small and has nothing to install around it

bankML is one Rust crate with zero external dependencies. The GGUF reader, sha256, f16 maths, the thread pool, the HTTP server and every kernel are written in the crate. You get one binary and nothing else to trust.

one page per source file · Cargo.toml

5. It speaks the languages your tools already use

• the OpenAI chat API (/v1/chat/completions), streamed or not — serve.md
• the Ollama API (/api/chat, /api/generate, a registry of pinned models), so programs written for Ollama work unchanged — OLLAMA.md
• a C library (libbankml) for C, C++, Python, Go or Swift programs — CAPI.md
• JSON mode and JSON schemas: answers forced into a shape you choose — grammar.md, schema.md

6. It measures itself

bankML reports its own time to first token, tokens per second and, where the machine allows, the energy each token costs. It also limits how much of a graphics card it may use.

metrics.md and console.md (with 0.3.7) · the GPU: gpu.md

What people use it for

new here? start with ./install.sh: usage.md §1

How Rust helps bankML go fast and stay exact

Rust lets bankML write code as close to the hardware as C, while the compiler checks much of what C leaves to the programmer.

• Direct access to the CPU's vector instructions. The kernels use AVX2 through std::arch, compiled with #[target_feature(enable = "avx2,fma,f16c")]. bankML checks at run time which instructions the CPU has, so one binary runs everywhere and uses the fast path where it can. — q1_0.md, q2_0.md
• No hidden costs. No garbage collector and no runtime pauses. Iterators and fixed-size chunks (as_chunks) compile down to plain loops, which lets the compiler drop bounds checks inside the hot kernels.
• Safe threads, the same bits on any thread count. The thread pool hands out rows from an atomic counter. Ownership rules guarantee each output chunk has exactly one writer, so threads can never race, and the answer does not depend on how many threads ran. — par.md, bankML/par.rs
• unsafe kept in small, named places. The memory map, the vector intrinsics and the GPU loader carry documented safety contracts behind safe functions; the compiler checks the rest. — gguf.md
• Zero-copy model loading. The model file is memory-mapped, and the kernels read the weights straight from the mapping. — bankML/gguf.rs
• Hostile files handled safely. All header arithmetic is checked (checked_add, checked_mul), so a malformed file is refused with a reason instead of crashing.
• Whole-program optimisation. Release builds use link-time optimisation, one code-generation unit and abort-on-panic: one small, fast binary. — Cargo.toml
• A pinned toolchain. Rust 1.99 is pinned, so a build today gives the same machine code as a build next month, which an exactness project needs. — rust-toolchain.toml