espnet/espnet

• ESPnet 202610 —

• ESPnet 202609 —

• ESPnet 202604 —

• ESPnet 202511 —

• ESPnet 202509 —

• ESPnet 202506 —

• ESPnet 202503 —

Full history: Releases.

pip install "espnet[all]"                       # training plus every task extra (except sds)
pip install git+https://github.com/espnet/espnet  # latest master

• Full setup (recipes, DNN training, Kaldi-style tooling): see the

• Docker: see docker/ and the Docker docs.
• Task-specific tools live in tools/installers.
• ESPnet1 is no longer supported — use ESPnet2 (egs2/) or ESPnet3 (egs3/). See the ESPnet1 notice.

|system/pytorch ver.|2.11.0|2.13.0|2.14.0| | :---- | :---: | :---: | :---: | |ubuntu/python3.12/pip|||| |ubuntu/python3.13/pip|||| |debian12/python3.12/conda|||| |windows/python3.12/pip|||| |macos/python3.12/pip|||| |macos/python3.12/conda||||

Each badge is its workflow's aggregate status on master, where the full grid runs. Coverage is not uniform — some suites run on one pytorch only, and a pull request runs less than master does. What each column covers.

cd egs3/librispeech_100/asr
train=conf/tuning/training_e_branchformer.yaml   # the config the model was trained with
python run.py --stages pack_model  --training_config $train --publication_config conf/publication.yaml  # -> exp/.../model_pack
python run.py --stages pack_demo   --training_config $train --demo_config conf/demo.yaml                # -> demo/
python run.py --stages upload_demo --training_config $train --demo_config conf/demo.yaml                # needs hf auth login

Run the packed app locally with python demo/app.py.

Kaldi-style complete recipe

• Support numbers of ASR recipes (WSJ, Switchboard, CHiME-4/5, Librispeech, TED, CSJ, AMI, HKUST, Voxforge, REVERB, Gigaspeech, etc.)
• Support numbers of TTS recipes in a similar manner to the ASR recipe (LJSpeech, LibriTTS, M-AILABS, etc.)
• Support numbers of ST recipes (Fisher-CallHome Spanish, Libri-trans, IWSLT'18, How2, Must-C, Mboshi-French, etc.)
• Support numbers of MT recipes (IWSLT'14, IWSLT'16, the above ST recipes etc.)
• Support numbers of SLU recipes (CATSLU-MAPS, FSC, Grabo, IEMOCAP, JDCINAL, SNIPS, SLURP, SWBD-DA, etc.)
• Support numbers of SE/SS recipes (DNS-IS2020, LibriMix, SMS-WSJ, VCTK-noisyreverb, WHAM!, WHAMR!, WSJ-2mix, etc.)
• Support voice conversion recipe (VCC2020 baseline)
• Support speaker diarization recipe (mini_librispeech, librimix)
• Support singing voice synthesis recipe (ofuton_p_utagoe_db, opencpop, m4singer, etc.)

ASR: Automatic Speech Recognition

• State-of-the-art performance in several ASR benchmarks (comparable/superior to hybrid DNN/HMM and CTC)
• Hybrid CTC/attention based end-to-end ASR
• Fast/accurate training with CTC/attention multitask training
• CTC/attention joint decoding to boost monotonic alignment decoding
• Encoder: VGG-like CNN + BiRNN (LSTM/GRU), sub-sampling BiRNN (LSTM/GRU), Transformer, Conformer, Branchformer, or E-Branchformer
• Decoder: RNN (LSTM/GRU), Transformer, or S4
• Attention: Flash Attention, Dot product, location-aware attention, variants of multi-head
• Incorporate RNNLM/LSTMLM/TransformerLM/N-gram trained only with text data
• Batch GPU decoding
• Data augmentation
• Transducer based end-to-end ASR
• Architecture:
• Custom encoder supporting RNNs, Conformer, Branchformer (w/ variants), 1D Conv / TDNN.
• Decoder w/ parameters shared across blocks supporting RNN, stateless w/ 1D Conv, MEGA, and RWKV.
• Pre-encoder: VGG2L or Conv2D available.
• Search algorithms:
• Greedy search constrained to one emission by timestep.
• Default beam search algorithm [[Graves, 2012]](https://arxiv.org/abs/1211.3711) without prefix search.
• Alignment-Length Synchronous decoding [[Saon et al., 2020]](https://ieeexplore.ieee.org/abstract/document/9053040).
• Time Synchronous Decoding [[Saon et al., 2020]](https://ieeexplore.ieee.org/abstract/document/9053040).
• N-step Constrained beam search modified from [[Kim et al., 2020]](https://arxiv.org/abs/2002.03577).
• modified Adaptive Expansion Search based on [[Kim et al., 2021]](https://ieeexplore.ieee.org/abstract/document/9250505) and NSC.
• Features:
• Unified interface for offline and streaming speech recognition.
• Multi-task learning with various auxiliary losses:
• Encoder: CTC, auxiliary Transducer and symmetric KL divergence.
• Decoder: cross-entropy w/ label smoothing.
• Transfer learning with an acoustic model and/or language model.
• Training with FastEmit regularization method [[Yu et al., 2021]](https://arxiv.org/abs/2010.11148).

• CTC segmentation
• Non-autoregressive model based on Mask-CTC
• ASR examples for supporting endangered language documentation (see egs2/puebla_nahuatl and egs2/yoloxochitl_mixtec)
• Wav2Vec2.0 pre-trained model as Encoder, imported from FairSeq.
• Self-supervised learning representations as features, using upstream models in S3PRL in frontend.
• Set frontend to s3prl
• Select any upstream model by setting the frontend_conf to the corresponding name.
• Transfer Learning :
• easy usage and transfers from models previously trained by your group or models from ESPnet Hugging Face repository.
• Documentation and toy example runnable on colab.
• Streaming Transformer/Conformer ASR with blockwise synchronous beam search.
• Restricted Self-Attention based on Longformer as an encoder for long sequences
• OpenAI Whisper model, robust ASR based on large-scale, weakly-supervised multitask learning

• Real-time ASR demo with ESPnet2
• Hosted demo: OWSM-CTC v4, maintained from egs2/owsm_ctc_v4/s2t1/demo — recognises and identifies 151 languages, translates into 25 of them, and decodes long-form audio
• Hosted demo: OWSM v4, maintained from egs2/owsm_v4/s2t1/demo — the same four tasks with the encoder-decoder model, which also takes a text prompt
• Streaming Transformer ASR Local Demo with ESPnet2.

TTS: Text-to-speech

• Architecture
• Tacotron2
• Transformer-TTS
• FastSpeech
• FastSpeech2
• Conformer FastSpeech & FastSpeech2
• VITS
• JETS
• Multi-speaker & multi-language extension
• Pre-trained speaker embedding (e.g., X-vector)
• Speaker ID embedding
• Language ID embedding
• Global style token (GST) embedding
• Mix of the above embeddings
• End-to-end training
• End-to-end text-to-wav model (e.g., VITS, JETS, etc.)
• Joint training of text2mel and vocoder
• Various language support
• En / Jp / Zn / De / Ru / And more...
• Integration with neural vocoders
• Parallel WaveGAN
• MelGAN
• Multi-band MelGAN
• HiFiGAN
• StyleMelGAN
• Mix of the above models

• Real-time TTS demo with ESPnet2
• Integrated to Hugging Face Spaces with Gradio. See demo:

• [kan-bayashi/ParallelWaveGAN