FlashML-org/FreeToken

▲ 140 stars today★ 15,073⑂ 1,532

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

About FlashML-org/FreeToken

FlashML-org/FreeToken is an open-source project on GitHub, mainly written in Python. FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently. It currently holds 15,073 stars and 1,532 forks with 469 open issues, and was last pushed on 2026-10-30 (repository created 2026-07-20).

Project Overview

Git Homed tracks it on the Today's Trending board, currently at rank #35 with 140 new stars today.

GitHub Repository Details

Repository FlashML-org/FreeToken · default branch main · size 4465 KB · watchers 97 · source: GitHub REST API and repository README

README

https://github.com/FlashML-org/FreeToken/blob/HEAD/FreeToken

| Download | Paper | Developer Slack | Community Discord | Community WeChat |

Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.

About

FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include:

Getting Started

Desktop app

Download FreeToken for Windows or Linux at flashml.ai. It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine.

https://github.com/FlashML-org/FreeToken/blob/HEAD/FreeToken Desktop

CLI

Install FreeToken with uv (recommended) or pip:

uv pip install "freetoken[accel]"

Or build from source:

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

For More details:

Citation

If you use FreeToken for your research, please cite our paper:

@article{yang2026freetoken,
  title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution},
  author={Yang, Shuo and Fan, Xiaoze and Pan, Melissa and Xi, Haocheng and Wang, Zhe and Sun, Shanlin and Keutzer, Kurt and Han, Song and Zaharia, Matei and Xu, Chenfeng and Stoica, Ion},
  journal={arXiv preprint arXiv:2608.16157},
  year={2026}
}

Acknowledgment

FreeToken was deeply inspired by mini-sglang, and learned the design and reused code from the following projects: SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.

License

Apache License 2.0.

GitHub Stars & Activity

15,073Stars
1,532Forks
469Open issues
PythonLanguage

GitHub Popularity

GitHub stars15,073
Forks1,532
Open issues469
Primary languagePython
LicenseApache-2.0
Stars gained today140
Created2026-07-20
Last pushed2026-10-30

Trending History

Daily boardrank #35 · ▲ 140 stars

Related GitHub Projects

1

github / spec-kit

Python★ 147,618⑂ 13,243▲ 808 stars
→
2

harry0703 / MoneyPrinterTurbo

Python★ 139,519⑂ 22,242▲ 617 stars
→
3

D4Vinci / Scrapling

Python★ 97,245⑂ 10,188▲ 2,005 stars
→
4

calesthio / OpenMontage

Python★ 76,847⑂ 10,191▲ 577 stars
→
5

shiyu-coder / Kronos

Python★ 46,901⑂ 7,742▲ 503 stars
→
6

davila7 / claude-code-templates

Python★ 33,138⑂ 3,871▲ 30 stars
→
7

Usagi-org / ai-goofish-monitor

Python★ 17,269⑂ 2,923▲ 984 stars
→
8

The-PR-Agent / pr-agent

Python★ 14,988⑂ 2,103▲ 280 stars
→

More Trending Repositories