ashhart/TensorFold

▲ 37 stars today★ 1,811⑂ 385

LLM Inference Engine for Metal, CUDA and Vulkan.

About ashhart/TensorFold

ashhart/TensorFold is an open-source project on GitHub, mainly written in Zig. LLM Inference Engine for Metal, CUDA and Vulkan. It currently holds 1,811 stars and 385 forks with 465 open issues, and was last pushed on 2026-11-12 (repository created 2026-06-19).

Project Overview

Git Homed tracks it on the Today's Trending board, currently at rank #73 with 37 new stars today.

GitHub Repository Details

Repository ashhart/TensorFold · default branch main · size 71520 KB · watchers 10 · source: GitHub REST API and repository README

README

TensorFold

TensorFold serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs. The engine reads checkpoints, tokenizes requests and runs Metal or CUDA kernels directly. Serving needs no Python or MLX installation.

Building with a coding agent? Have it load the contributor skill before it writes any code.

Contributing with agents

Please use Agent Traffic Control before editing TensorFold so contributors and agents can avoid duplicate work and conflicting changes. Check the live agent radar before choosing an issue and follow CONTRIBUTING.md and the ATC contributor instructions.

Install or upgrade ATC to 0.9.4 or later, then run from an updated TensorFold clone:

npm install -g github:ashhart/ATC
atc check ISSUE --json
atc claim ISSUE --json

Replace ISSUE with the issue number and claim only after a complete check says the work is free. TensorFold enables automatic public check-ins: a successful new claim starts an anonymous session on the radar and returns its session number. Keep that number instead of starting a duplicate session. For work already yours, start a session with atc start ISSUE --json. Add --github-name to claim or start only if you want your verified GitHub name shown; --local keeps a new session on your machine. The board shows the issue, activity state and last check-in; GitHub still associates the verification comment with its author. Check that publication is public; unavailable means the public update failed even if the local claim or session succeeded.

Pulse only while the agent is working, before its five-minute check-in expires:

atc pulse ISSUE SESSION working --json
atc who path/to/file --json
atc overlap --json
atc land ISSUE SESSION --json
atc release ISSUE --json

Use reviewing or blocked when appropriate, land the session when it stops, and resolve overlaps before opening a pull request. Before opening a pull request, have the agent run the pull request skill; agents that support skills can install it as tensorfold-pull-request.

Quick start

Start with Qwen3.5-2B, the small 4-bit checkpoint checked on Metal and a GB10. The commands below download its pinned revision, then serve it as local-model. Model weights have their own licenses. Reviewing or benchmarking TensorFold? Evaluating TensorFold covers pinning, signature checks, a container and the tests that need no GPU.

Mac with Apple Silicon

Install with Homebrew, then start the server:

brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Linux with an NVIDIA GPU

Use the release archive for your CPU: linux-aarch64 on a GB10, linux-x86_64 on an x86_64 host. It needs glibc 2.28 or newer and NVIDIA driver R580 or newer (CUDA 13.0), providing libcuda.so.1. Keep the archive's directories together. Its kernels are built with CUDA 13, so TensorFold refuses an older driver at startup; nvidia-smi shows the installed one. See the CUDA runbook for runtime libraries and supported GPUs.

V=1.2.7
case "$(uname -m)" in
  aarch64|arm64) PLATFORM=linux-aarch64 ;;
  x86_64) PLATFORM=linux-x86_64 ;;
  *) echo 'No release archive for this CPU' >&2; exit 1 ;;
esac
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz"
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz.sha256"
sha256sum -c "tensorfold-$V-$PLATFORM.tar.gz.sha256"
tar -xzf "tensorfold-$V-$PLATFORM.tar.gz"
export PATH="$PWD/tensorfold-$V-$PLATFORM/bin:$PATH"
tensorfold() { command tensorfold-native "$@"; }
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Send a request

Leave the server running. In another terminal, on either platform:

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'

The API also serves completions, Responses and Anthropic Messages. /v1/models lists the served name. On Windows with an NVIDIA GPU, follow the native Windows build guide. Use tensorfold --help, tensorfold capabilities --json and tensorfold serve REPO --help to inspect your binary. You can also serve a complete local checkpoint directory or an existing Hugging Face cache.

tensorfold web [ARGS...] opens TensorFold Web, a small local web app for picking, running and chatting with models. If it needs installing, an interactive release binary offers to install its matching version with uv after you say yes. See the web launcher for manual installation and unattended use.

--parallel N caps active requests at N; extra requests wait for a free slot. Omitting the flag uses --parallel auto, which asks for up to eight streams as memory and the engine allow. An explicit number is refused if the engine cannot fit that many streams. Read the engine's startup capacity line, such as 8 streams at once on CUDA, for the count it accepted. With API authentication off, /health reports live.connections as active plus waiting requests and live.waiting as the queued count. For example, 8 connections with 4 waiting means 4 active requests. Completing or cancelling an active request frees its slot; cancelling a waiting request removes it from the queue.

Mac app

The TensorFold desktop workspace bundles the local engine, model downloads, hardware recommendations, live generation metrics, coding workers and document editing. Download the Apple-silicon beta 9 DMG, or build from app/. The desktop app launches the bundled tensorfold-native executable directly. Mobile clients in app/mobile/ remain preview software.

TensorFold Starlight (enterprise)

TensorFold Starlight gives an organisation one governed OpenAI- and Anthropic-compatible endpoint for a fleet of TensorFold engines on Macs, Linux GPU servers and Windows machines. It signs people in through the organisation's identity provider (OIDC or SAML, with SCIM provisioning), decides who may use which model and where it runs, places and supervises models on nodes, and accounts for every token. It runs fully offline, or brokers approved frontier providers under the same identity, policy and budgets; the install guide covers Docker, Kubernetes and Linux services. Starlight lives in its own repository, ashhart/tf-starlight: it pins the TensorFold release it is tested with and checks every flag its nodes pass to tensorfold serve against that release in CI. Starlight is new: the gates that still need real GPU clusters or identity-provider tenants are listed in its qualification notes.

Living Weights (experimental)

Living Weights teaches a running model new facts in its own weights. Start a Nemotron 3.5 Lightning server with --slide and tell it something: each fact that passes the lesson's checks is written into the checkpoint folder's files. It is experimental: related facts can disturb each other, recall drops past a few dozen facts and with thinking on, and learned facts cost some decode speed on a Mac. Known issues has the details.

tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
hub="${HF_HUB_CACHE:-${HF_HOME:-$HOME/.cache/huggingface}/hub}/models--TensorFold--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit"
src="$hub/snapshots/$(cat "$hub/refs/main")"
mkdir -p ~/models && cp -RL "$src" ~/models/nemotron-living   # always teach a copy
(umask 077; openssl rand -hex 24 > ~/models/slide-key)        # --slide starts only with a key
tensorfold serve ~/models/nemotron-living --slide --api-key-file ~/models/slide-key

Leave that server running and send a lesson from another terminal:

curl -N http://127.0.0.1:8080/v1/slide/learn -H "Authorization: Bearer $(cat ~/models/slide-key)" \
  -H 'Content-Type: application/json' -d '{"text": "I like blue."}'

Send it a document instead and it finds the facts in it and learns them one by one. Open http://127.0.0.1:8080/slide to watch in 3D: the facts, the neurons each one took in the model's layers, the shard files that hold them, and the lesson running now.

Learning runs on Apple-silicon Macs and on an NVIDIA GB10 (DGX Spark), and a folder that learned on one serves on the other. On an APFS volume, cp -cRL makes a clone that shares the original's data until lessons change it. It rewrites the folder's files: before the first lesson the server keeps the originals in a hidden snapshot, and tensorfold slide rollback ~/models/nemotron-living puts them back, undoing every lesson. The Living Weights guide covers the key, teaching files and web pages, omp's /learn command, the 3D view, undoing lessons, and what to expect.

Pick a model

These are the native release's serving paths. A recipe may also describe the older Python 0.6.6 engine; use its native section and the current changes together. Hardware below records the documented scope, not an unmeasured minimum. Context, parallel requests, prompt caches and drafters need memory beyond the weights. tensorfold info REPO reports the checkpoint's memory floor. Where no RAM capacity is published, inspect that floor before serving a large checkpoint. CUDA release images include sm_80, sm_86, sm_120 and sm_121. The A100 (sm_80) has kernel-check evidence (#846), but no served checkpoint on record, so it is not model-qualified. Nemotron and dense Qwen have NVFP4/FP8 fallback code for sm_80 to sm_89 (#901, #1470); no such model run on sm_80 to sm_89 is recorded.

Run a table command after installation. On a Linux archive installation, use the tensorfold shell function from Quick start. REPO@revision pins a pull; serve the repository name without the revision after pulling it. Conversions are explicitly marked: their upstream weights need the linked conversion before native serving.

| Model / format | Download | Machine and memory scope | Native recipe / limits | | --- | --- | --- | --- | | dots.mocr, BF16 OCR | Use the local dots-studio/dots.mocr checkpoint. | Native HIP; AMD R9700 (gfx1201) checked, one GPU. | Native OCR recipe; full model-card demo matches HF tokens. No video, MTP head, CUDA, or Metal qualification. | | Qwen3.5-2B, MLX 4-bit | tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3 | One Mac (M5 Max checked), or one GB10; no RAM minimum published. | 2B; CUDA release checks. Text only. | | Qwen3.8-27B, MLX 4-bit | tensorfold pull TensorFold/Qwen3.8-27B-MLX-4bit
tensorfold pull z-lab/Qwen3.8-27B-DFlash2 | One Mac (M5 Max/M3 Ultra), or one GB10; a 32 GB Mac needs iogpu.wired_limit_mb raised (the startup refusal names the value). | 27B. Add --drafter z-lab/Qwen3.8-27B-DFlash2; without it decoding is plain. | | Nemotron 3.5 Lightning 30B-A3B, MLX 4-bit | tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit | One Mac (M1-M5), or one GB10; also checked on RTX 3090 / RTX PRO 6000. | Lightning. Includes MTP. | | Lightning, mixed oQ4 / oQ6 / oQ8 | tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ4
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ6
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-oQ8 | One M3 Ultra checked. | Width and MTP instructions. Metal only; copy the 4-bit checkpoint's MTP head to draft. | | Lightning, NVIDIA NVFP4 | tensorfold pull nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 | One GB10 checked. Non-cluster fallback code exists for sm_80 to sm_89; no NVFP4/FP8 serving run is recorded on sm_80. | Lightning; CUDA changes. | | Nemotron 3.5 Super VL 120B-A12B, NVFP4 | tensorfold pull nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4@82d93699acd3be9e4617c53360fe5b3f5dbeb1b3 | One or two GB10s; weights 70.8 GiB, 73.2 GiB with vision on one. | One/two-GB10 commands. Add --vision for images. | | Nemotron 3.5 Bolt 74B-A9B | NVIDIA's checkpoint | Native CUDA; 24 GB checkpoint, checked on RTX PRO 6000. | Bolt/Omni video; release qualification. | | Qwen3.8 Flash Next, MLX 6-bit | tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP | One M5 Ultra checked; RAM minimum not published. | Metal, runbook. | | Qwen3.8 Flash Next, MLX 4-bit | tensorfold pull TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP | One GB10; 73.3 GiB uploaded, plus mapped tables and caches. Metal has a native 4-bit path too (M3 Ultra checked). | Native CUDA. NVFP4/two-local-rank and vision paths are experimental. Metal's shared-session limits are in known issues. | | GLM-5.3-Flash, MLX 4-bit | tensorfold pull TensorFold/GLM-5.3-Flash-MLX-4bit-MTP | Two M5 Ultras; single-Mac native serving is not qualified. | Paired Metal runbook. Do not use the Python recipe's single-Mac command. | | GLM-5.3-Flash, EXL3/TR3 | tensorfold pull Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw | Two CUDA ranks; the release checks the two-GB10 setup. | Release scope, checkpoint. Experimental checkpoint; use native rank options, not the Python installation. | | GLM-5.3-Flash, NVFP4 | tensorfold pull nvidia/GLM-5.3-Flash-NVFP4@423acf37583782c51c142d145aef733d72943d93 | Two GB10s at TP=2: @Bizuayeu loaded the whole checkpoint on 1.1.0 in 302.1 s and served it. | Native CUDA; contributed run: one token hash per prompt across three runs and a passphrase found in 199,307 tokens; scored prompts and more than two ranks were not covered. | | GLM-5.3-Flash, FP4-Local | tensorfold pull nvidia/GLM-5.3-Flash-FP4-Local@76348ddb2be2e35e39ecfc00e75dc07703f4f3d5 | Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu loaded it as stored and served it at both sizes. 165.3 GiB of weights, 13% less than NVFP4's. | Native CUDA, in 1.1.5 (#1050). NVIDIA's mixed-precision checkpoint: NVFP4 routed experts, FP8 for most of the rest. As close to BF16 as NVFP4 at its smaller size (receipt). | | GLM-5.3-Flash, XSFP4 | tensorfold pull Bizuayeu/GLM-5.3-Flash-XSFP4@88736f916f8939607fde8a4b881dcae749f7551d | Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu served it at both sizes. 165.7 GiB of weights. | Native CUDA, in 1.1.5 through FP4-Local's path. FP4-Local with its routed experts refitted from the BF16 weights and a smoothing folded in, on the same kernels. Closer to BF16 than FP4-Local and NVFP4 by all three measures in its card. | | GLM-5.3-Flash, Jammed XSFP4 | tensorfold pull Bizuayeu/GLM-5.3-Flash-Jammed-XSFP4@0a22bb46ecef3d0f5f96e41dc6011be12974fbf1 | Two GB10s at TP=2, or three at TP=3 over a ring of direct links: @Bizuayeu served it at both sizes. 160.2 GiB of weights; at TP=2 each GB10 holds 6.6 GiB less than at block 16. | Native CUDA, in 1.2.3 (#2041). Refitted from BF16 at block 32 with XSFP4's smoothing, each row's scales packed (csf0_v1, read only by this path), at XSFP4's speed. Its KL and top-1 are nearer BF16 than NVIDIA's two checkpoints', short of XSFP4's (its card). | | GLM-5.3, EXLR8 home | tensorfold pull kindlingai/glm-5.3-exlr8-k2-k3-k4@f459f5d0272a529403dd9b3fd32f771f4250a0d4 --exlr8 home --node 0/2 --dir ~/models/glm-5.3-exlr8-home | Two M5 Ultra Macs, 256 GB each; about 160 GB resident per Mac at 8k context / four streams. | Two-Mac recipe. Use --node 1/2 on the other Mac; Thunderbolt 5 and MCDMA required. No CUDA backend. | | LFM2.5-24B-A2B, MLX 4 / 6 / 8-bit | tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-4bit@667408d494bde191c2cb9dd4f4549827816b9275
tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-6bit@1deec53eb49571e21cdcbf82fd5839248f07ed88
tensorfold pull LiquidAI/LFM2.5-24B-A2B-MLX-8bit@67ab417dd02d0c9545802a726df668620eae049d | One Mac (M4 Pro checked), or one NVIDIA GPU with the 4-bit export (GB10 checked). | Native recipe (CUDA), release history. Weights: LFM Open License v1.0. | | Qwen3.6-35B-A3B, MLX with MTP | tensorfold pull TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP@81169a9bc511a27c1b4eedb77a2cd98ced431847 | One GB10; 20.9 GB checkpoint. | Qwen3.6. Native CUDA receipt here. Metal also registers this model type; the recipe's Metal commands describe Python 0.6.6. | | Qwen3.6-35B-A3B, NVIDIA NVFP4 with MTP | tensorfold pull nvidia/Qwen3.6-35B-A3B-NVFP4@1355db6a052410cfd62085d94b58866fd0f2c3c5 | One GB10; 20.3 GiB on the device, 20.9 GiB with the drafter. | Native CUDA. Serve with --drafter naming the same checkpoint for MTP drafts. | | Gemma 4 31B, MLX 4-bit | tensorfold pull mlx-community/gemma-4-31B-it-4bit@696d436c404745a59f30e4939a658162b0a9e57f
tensorfold pull z-lab/gemma-4-31B-it-DFlash@eabd648301ce28583cc14757912e5e0f84e152e1 | One Mac (M5 Ultra checked), or one GB10 on CUDA. Pre-M5 Metal has a row path; its assistant and DFlash drafters require M5 tensor units. | Native 31B on Metal, CUDA qualification. Name the drafter with --drafter; images with --vision. | | Gemma 4 12B, converted MLX 4-bit | hf download google/gemma-4-12B-it --revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7 --local-dir ~/models/gemma-4-12b-hf
hf download google/gemma-4-12B-it-assistant --revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51 --local-dir ~/models/gemma-4-12b-assistant | One Mac (M5 Ultra checked); converted text weights 6.3 GB, image weights 100 MB, plus assistant/caches. MLX 4-bit also served on one GB10. Pre-M5 Metal has a row path; the MTP assistant requires M5 tensor units. | Convert and serve. Upstream download needs hf; conversion needs Python, mlx and mlx-lm. Serving is native; CUDA qualification includes the MTP assistant. | | Step-5-Preview, converted affine | hf download stepfun-ai/Step-5-Preview --local-dir ~/models/step5-preview-fp8 | One 256 GB Mac; text 166.9 GiB, MTP adds 2.5 GiB and vision 3.7 GiB. | Convert and serve. The documented 256 GB setup serves converted affine weights; converting the upstream FP8 download needs Python and mlx. | | MiniMax-M3.1-Flash, community affine 3/4-bit | hf download BobClawblaw/MiniMax-M3.1-Flash-MLX-mix34 --revision 9524999057e0d277d26438606e033e2191fb465c --local-dir ~/models/MiniMax-M3.1-Flash-MLX-mix34
hf download BobClawblaw/MiniMax-M3.1-Flash-MLX-mix256k --revision 916e7ca1044f1c3e7566163096243c940327224d --local-dir ~/models/MiniMax-M3.1-Flash-MLX-mix256k | One 256 GB M5 Ultra; 203.7 GiB (32,768-token window) or 189.2 GiB (147,456), plus caches. | MiniMax-M3.1-Flash. Set MINIMAX_LOAD_LIMIT_GB; images with --vision. Community checkpoints: lossy conversions of MiniMax's NVFP4 release (KL 0.244 / 0.295). MiniMax Community License. | | Kolibri-1, block FP8 (research) | tensorfold pull Aleph-Alpha/Kolibri-1@e52eb4627d11516b0c01de49210ab5a4e4061444 | One NVIDIA GPU; GB10 measured; 74 GB checkpoint. | Kolibri. Text only; copy drafts, no EAGLE-3/neural drafter. | | Liquid AI d1-3B, decisions | Not documented yet. | Native CUDA; RAM minimum not published. | API section below. /v1/systemone and /v1/decisions. | | Fish Audio Drama 3, speech | Not documented yet. | Native Metal or CUDA; GB10 measured. | Release audio instructions, reference voices and dialogue speakers. /v1/audio/speech with sampling controls; checkpoint and codec.pth required. | | Drama 3 pruned NVFP4, speech | hf download plotarmordev/drama-3-NVFP4-Pruned --revision 494db08481db952f834b2241bce63fb3dfa72554 --local-dir drama-3-NVFP4-Pruned | Native Metal; roughly 21 GiB checkpoint plus runtime state. Not a minimum-RAM or cross-backend quality claim. | Packed experts, BF16 dense/tower weights and FP32 accumulation; no conversion. Use serve drama-3-NVFP4-Pruned --cfg-gamma 1.5 --cfg-batched with an attested catalog voice. |

GLM-5.3: one machine or two?

The native release does not qualify GLM-5.3 on one machine. GLM-5.3 EXLR8 home needs two 256 GB M5 Ultra Macs; its recipe gives both pull and serve commands. GLM-5.3-Flash is a different model: its native Metal setup needs two M5 Ultras. Native CUDA has recorded TP=2 and TP=3 setups, including FP4-Local and XSFP4 on two and three GB10s. The single-256-GB-Mac GLM-5.3-Flash command in the older recipe is for Python 0.6.6, not this release. For the NVFP4 path, see GLM-5.3-Flash on CUDA and what the contributed run on two GB10s covered.

DeepSeek checkpoints are not served yet by any release binary: whole-model gate #827 is still open. MiMo's native backend is not in a release yet. Bonsai's ternary pack and Gemma 4 26B-A4B remain Python-only; they are not native download recommendations. Known issues lists current limits, including Flash Next shared-session cache/performance behavior and the 27B's drafter limits.

Exact decoding

Every accepted draft must equal the token the same native engine would produce with "draft": false. A resumed request must equal fresh execution, and each concurrent stream must equal its solo run. The comparison fixes the checkpoint, backend, settings and runtime. Different quantizations and different backends can produce different outputs. Flash Next on Metal uses shared target rounds with an explicit --parallel N above one, when that many sessions fit. A lone greedy request there uses GPU-side rounds and hands

GitHub Stars & Activity

1,811Stars
385Forks
465Open issues
ZigLanguage

GitHub Popularity

GitHub stars1,811
Forks385
Open issues465
Primary languageZig
LicenseApache-2.0
Stars gained today37
Created2026-06-19
Last pushed2026-11-12

Trending History

Daily boardrank #73 · ▲ 37 stars

Related GitHub Projects

1

mattpocock / skills

Shell★ 324,722⑂ 27,265▲ 706 stars
→
2

obra / superpowers

Shell★ 317,811⑂ 28,304▲ 428 stars
→
3

msitarzewski / agency-agents

Shell★ 168,432⑂ 27,323▲ 232 stars
→
4

ggml-org / llama.cpp

C++★ 135,142⑂ 25,639▲ 240 stars
→
5

JuliusBrussee / caveman

Go★ 120,274⑂ 7,011▲ 227 stars
→
6

addyosmani / agent-skills

JavaScript★ 117,718⑂ 12,322▲ 411 stars
→
7

rasbt / LLMs-from-scratch

Jupyter Notebook★ 108,432⑂ 16,778▲ 57 stars
→
8

localsend / localsend

Dart★ 99,080⑂ 5,677▲ 144 stars
→

More Trending Repositories