HeartMuLa/heartlib

★ 3,804⑂ 0

HeartMuLa Official Repo: The Most Powerful Open-Source Music Generation Model of 2026

About HeartMuLa/heartlib

HeartMuLa/heartlib is an open-source project on GitHub, mainly written in Python. HeartMuLa Official Repo: The Most Powerful Open-Source Music Generation Model of 2026 It currently holds 3,804 stars and 0 forks with 0 open issues, and was last pushed on an unknown date (repository created unknown).

Project Overview

Git Homed tracks it on the Audio Trending board and on the AI Audio Trending list.

GitHub Repository Details

Repository HeartMuLa/heartlib · default branch - · size 0 KB · watchers 0 · source: GitHub REST API and repository README

README

https://github.com/HeartMuLa/heartlib/blob/HEAD/HeartMuLa

🎵 Try HeartMuLa online → heartmula.com

Demo 🎶  |  📑 Paper
HeartMuLa-oss-3B-happy-new-year 🤗  |  HeartMuLa-oss-3B-happy-new-year
HeartCodec-oss-encoder 🤗  |  MuLaCover https://github.com/HeartMuLa/heartlib/blob/HEAD/GitHub  |  MuLaCover 🤗

---

HeartMuLa: A Family of Open Sourced Music Foundation Models

HeartMuLa is a family of open sourced music foundation models including: 1. HeartMuLa: a music language model that generates music conditioned on lyrics and tags with multilingual support covering almost all languages. 2. HeartCodec: a 12.5 hz music codec with high reconstruction fidelity; 3. HeartTranscriptor: a whisper-based model specifically tuned for lyrics transcription; Check this page for its usage. 4. HeartCLAP: an audio–text alignment model that establishes a unified embedding space for music descriptions and cross-modal retrieval. ---

Below shows the experiment result of our oss-3B version compared with other baselines.

---

🔥 Highlight

Our latest internal version of HeartMuLa-7B achieves comparable performance with Suno in terms of musicality, fidelity and controllability.

🎵 Want to make songs without setting anything up? Try heartmula.com in your browser.

📰 News

Join on Discord! https://github.com/HeartMuLa/heartlib/blob/HEAD/join discord Introducing MuLaCover, our controllable cover-song and music-remix model! Start from reference audio or melody/chord MIDI, keep the original lyrics or write new ones, and reshape the genre, instrumentation, and mood through text. MuLaCover uses symbolic melody and harmony to preserve the source's musical identity while allowing flexible reinterpretation. Code, model weights, and generation instructions are now available in the dedicated MuLaCover repository. Code | Model Weights | Generation Guide. We release HeartCodec-oss-encoder, adding audio tokenization and reconstruction to HeartCodec. Paired with HeartCodec-oss-20260123, it converts mono or stereo audio into discrete tokens and reconstructs 48 kHz stereo audio. Encoder and decoder checkpoints can be downloaded and loaded separately.

We hope this contribution will be useful to the music research community.

Encoder Weights | Reconstruction Guide.

We launched online demo spaces on Hugging Face and ModelScope. We released our HeartMuLa-oss-3B-happy-new-year version. This version is currently the best open-sourced model in terms of lyrics controllability and music quality. We recommend using HeartMuLa-oss-3B-happy-new-year and HeartCodec-oss-20260123 for music generation. We have released our HeartMuLa-Benchmark (referred to as HeartBeats Benchmark in our paper) as introduced in our paper. This benchmark comprises heterogeneous AI-generated lyrics and tags across diverse languages and genres, providing a rigorous and fair evaluation framework. By leveraging Reinforcement Learning, we have continuously refined our model and are proud to officially release HeartMuLa-RL-oss-3B-20260123. This version is designed to achieve more precise control over styles and tags. Simultaneously, we are launching HeartCodec-oss-20260123, which optimizes audio decoding quality. Benji has created a wonderful ComfyUI custom node for HeartMuLa. Thanks Benji! License update: We update the license of this repo and all related model weights to Apache 2.0. The official release of HeartTranscriptor-oss and the first HeartMuLa-oss-3B version along with our HeartCodec-oss.

---

🧭 TODOs

HeartCodec-oss, HeartMuLa-oss-3B, and HeartTranscriptor-oss.

---

🛠️ Local Deployment

⚙️ Environment Setup

We recommend using python=3.10 for local deployment.

Clone this repo and install locally.

git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
pip install -e .

Download our pretrained checkpoints from huggingface or modelscope using the following command:

# if you are using huggingface
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' HeartMuLa/HeartCodec-oss-20260123

if you are using modelscope

modelscope download --model 'HeartMuLa/HeartMuLaGen' --local_dir './ckpt' modelscope download --model 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year' --local_dir './ckpt/HeartMuLa-oss-3B' modelscope download --model 'HeartMuLa/HeartCodec-oss-20260123' --local_dir './ckpt/HeartCodec-oss'

After downloading, the ./ckpt subfolder should structure like this:

./ckpt/
├── HeartCodec-oss/
├── HeartMuLa-oss-3B/
├── config.json
├── gen_config.json
└── tokenizer.json

For audio reconstruction, download the separate encoder and decoder checkpoints:

hf download HeartMuLa/HeartCodec-oss-encoder --local-dir ./ckpt/HeartCodec-oss-encoder
hf download HeartMuLa/HeartCodec-oss-20260123 --local-dir ./ckpt/HeartCodec-oss-20260123

See the music reconstruction example for usage.

▶️ Example Usage

To generate music, run:

python ./examples/run_music_generation.py --model_path=./ckpt --version="3B"

By default this command will generate a piece of music conditioned on lyrics and tags provided in ./assets folder. The output music will be saved at ./assets/output.mp3.

FAQs

1. How to specify lyrics and tags?

The model will load lyrics from the txt file --lyrics link to (by default ./assets/lyrics.txt). If you would like to use your own lyrics, just modify the content in ./assets/lyrics.txt. If you would like to save your lyrics to another path, e.g. my_awesome_lyrics.txt, remember to input arguments --lyrics my_awesome_lyrics.txt.

For tags it's basically the same.

2. CUDA out of memory?

If you have multi-GPUs (e.g. 2 4090s), we recommend placing the params of HeartMuLa and HeartCodec separately on different devices. You can do it by typing --mula_device cuda:0 --codec_device cuda:1

If you are running on a single GPU, use --lazy_load true so that modules will be loaded on demand and deleted once inference completed to save GPU memory.

All parameters:

Recommended format of lyrics and tags:
[Intro]

[Verse] The sun creeps in across the floor I hear the traffic outside the door The coffee pot begins to hiss It is another morning just like this

[Prechorus] The world keeps spinning round and round Feet are planted on the ground I find my rhythm in the sound

[Chorus] Every day the light returns Every day the fire burns We keep on walking down this street Moving to the same steady beat It is the ordinary magic that we meet

[Verse] The hours tick deeply into noon Chasing shadows,chasing the moon Work is done and the lights go low Watching the city start to glow

[Bridge] It is not always easy,not always bright Sometimes we wrestle with the night But we make it to the morning light

[Chorus] Every day the light returns Every day the fire burns We keep on walking down this street Moving to the same steady beat

[Outro] Just another day Every single day

Regarding tags, check this issue for reference. Our different tags are comma-separated without spaces as illustrated below:

piano,happy,wedding,synthesizer,romantic

---

⚖️ License

This repository is licensed under the Apache 2.0 License.

---

📚 Citation

@misc{yang2026heartmulafamilyopensourced,
      title={HeartMuLa: A Family of Open Sourced Music Foundation Models}, 
      author={Dongchao Yang and Yuxin Xie and Yuguo Yin and Zheyu Wang and Xiaoyu Yi and Gongxi Zhu and Xiaolong Weng and Zihan Xiong and Yingzhe Ma and Dading Cong and Jingliang Liu and Zihang Huang and Jinghan Ru and Rongjie Huang and Haoran Wan and Peixu Wang and Kuoxi Yu and Helin Wang and Liming Liang and Xianwei Zhuang and Yuanyuan Wang and Haohan Guo and Junjie Cao and Zeqian Ju and Songxiang Liu and Yuewen Cao and Heming Weng and Yuexian Zou},
      year={2026},
      eprint={2601.10547},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2601.10547}, 
}

📬 Contact

If you are interested in HeartMuLa, feel free to reach us at contact@mulalabs.ai. Website: heartmula.com.

Welcome to join us through Discord or our WeChat group.

Our WeChat group now has more than 200 members. To join, scan our team member's QR code below and include HeartMuLa Group Invite in your friend request. We will invite you into the group manually.

Data Collaboration

For data collaboration, please contact us at contact@mulalabs.ai.

Join MuLa Labs, Vera Praxis Lab

We are always excited to meet people with a strong interest in audio and music. MuLa Labs has internship openings for candidates who want to build the next generation of music and audio technology. To apply, email your resume or CV, together with a short introduction, to contact@mulalabs.ai.

GitHub Stars & Activity

3,804Stars
0Forks
0Open issues
PythonLanguage

GitHub Popularity

GitHub stars3,804
Forks0
Open issues0
Primary languagePython
License-
Stars gained today0
Created-
Last pushed-

Trending History

Trending statusnot on today's boards

Related GitHub Projects

1

huggingface / transformers

Python★ 169,134⑂ 0
→
2

debpalash / VoiceStudio

Python★ 112,289⑂ 0
→
3

RVC-Boss / GPT-SoVITS

Python★ 65,245⑂ 0
→
4

coqui-ai / TTS

Python★ 46,240⑂ 0
→
5

OpenBMB / VoxCPM

Python★ 42,576⑂ 0
→
6

2noise / ChatTTS

Python★ 40,013⑂ 0
→
7

myshell-ai / OpenVoice

Python★ 38,265⑂ 0
→
8

babysor / MockingBird

Python★ 36,862⑂ 0
→

More Trending Repositories