Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

Shaolei Zhang, Shoutao Guo, Qingkai Fang, Yan Zhou, Yang Feng*

Stream-Omni is a GPT-4o-like language-vision-speech chatbot that simultaneously supports interaction across various modality combinations, with the following features💡:

• Omni Interaction: Support multimodal inputs including text, vision, and speech, and generate both text and speech responses.
• Seamless "see-while-hear" Experience: Simultaneously output intermediate textual results (e.g., ASR transcriptions and model responses) during speech interactions, like the advanced voice service of GPT-4o.
• Efficient Training: Require only a small amount of omni-modal data for training.

🖥 Demo

🎧 Vision-grounded Speech Interaction (simultaneously produce intermediate text) 🎧

https://github.com/user-attachments/assets/25807982-aa95-4633-9e92-10d995900258

https://github.com/user-attachments/assets/df8d79ba-63db-487c-a4a9-f183372168a1

[!NOTE]

Stream-Omni can produce intermediate textual results (ASR transcription and text response) during speech interaction, offering users a seamless "see-while-hear" experience.

• Downlaod Stream-Omni model from here, put in ${STREAMOMNI_CKPT}.
• Downlaod CosyVoice (Tokenizer & Flow Model) from here, put in COSYVOICE_CKPT=./CosyVoice-300M-25Hz:

  from modelscope import snapshot_download
  snapshot_download('iic/CosyVoice-300M-25Hz', local_dir='./CosyVoice-300M-25Hz')
  

• Run these scripts to launch the API and interface, and then interact through the browser (http://localhost:7860):

  # controller
  python stream_omni/serve/controller.py --host 0.0.0.0 --port 10000
  
  # CosyVoice worker
  COSYVOICE_CKPT=path_to_CosyVoice-300M-25Hz # e.g., ./CosyVoice-300M-25Hz
  WAV_DIR=path_to_save_generated_audio
  CUDA_VISIBLE_DEVICES=0 PYTHONPATH=CosyVoice/third_party/Matcha-TTS python ./CosyVoice/cosyvoice_worker.py --port 21003 --model ${COSYVOICE_CKPT} --wav_dir ./gen_wavs/
  
  # Stream-Omni worker, add --load-8bit for VRAM lower than 32GB 
  STREAMOMNI_CKPT=path_to_stream-omni-8b # e.g., ./stream-omni-8b
  CUDA_VISIBLE_DEVICES=1  python ./stream_omni/serve/model_worker.py --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ${STREAMOMNI_CKPT} --model-name stream-omni
  
  # Interface
  python stream_omni/serve/gradio_web.py --controller http://localhost:10000 --model-list-mode reload  --port 7860
  

• You can also refer to api.py for the usage of API.

🔥 Quick Start

[!Tip]

Stream-Omni achieves modality alignments through sequence-dimension concatenation for vision-text alignment and layer-dimension mapping for speech-text alignment.

Requirements

• Install packages:

  conda create -n streamomni python=3.10 -y
  conda activate streamomni
  pip install -e .
  pip install flash-attn --no-build-isolation
  pip install -r requirements.txt
  pip install -r CosyVoice/requirements.txt
  

Command Interaction

• Run these scripts for vision-grounded speech interaction:

  export CUDA_VISIBLE_DEVICES=0
  export PYTHONPATH=CosyVoice/third_party/Matcha-TTS
  
  STREAMOMNI_CKPT=path_to_stream-omni-8b
  
  # Replace the path of cosyvoice model in run_stream_omni.py (e.g., cosyvoice = CosyVoiceModel('./CosyVoice-300M-25Hz')) 
  # add --load-8bit for VRAM lower than 32GB 
  python ./stream_omni/eval/run_stream_omni.py \
      --model-path ${STREAMOMNI_CKPT} \
      --image-file ./stream_omni/serve/examples/cat.jpg --conv-mode stream_omni_llama_3_1 --model-name stream-omni  \
      --query ./stream_omni/serve/examples/cat_color.wav
  

  You should get the following outputs:

  ASR Outputs:
  What is the color of the cat
  LLM Outputs:
  The cat is gray and black.
  Speech Tokens:
  <Audio_2164><Audio_2247><Audio_671><Audio_246><Audio_2172><Audio_1406><Audio_119><Audio_203><Audio_2858><Audio_2099><Audio_1716><Audio_22><Audio_1736><Audio_1038><Audio_4082><Audio_1655><Audio_2409><Audio_2104><Audio_571><Audio_2255><Audio_73><Audio_760><Audio_822><Audio_701><Audio_2583><Audio_1038><Audio_2203><Audio_1185><Audio_2103><Audio_1718><Audio_2610><Audio_1883><Audio_16><Audio_792><Audio_8><Audio_8><Audio_535><Audio_67>
  Speech Outputs:
  Audio saved at ./output_893af1597afe2551d76c37a75c813b16.wav
  

• Interaction across various modality combinations:

  Inputs	Outputs	Intermediate Outputs	Scripts
  Text + Vision (or None)	Text	/	run_stream_omni_t2t.py
  Text + Vision (or None)	Speech	Text result of model outputs	run_stream_omni_t2s.py
  Speech + Vision (or None)	Text	ASR transciption of user inputs	run_stream_omni_s2t.py
  Speech + Vision (or None)	Speech	Text result of model outputs, ASR transciption of user inputs	run_stream_omni_s2s.py

  Control the interaction mode via inference_type in model.generate() (select from text_to_text, text_to_speech, speech_to_text, speech_to_speech)

Evaluation

• Refer to ./scripts/stream_omni/ for evaluation scripts.

🤝 Acknowledgement

• LLaVA/LLaVA-NeXT/LLaVA-OneVision: Stream-Omni is built upon the LLaVA and LLaVA-NeXT codebases and incorporates image instruction data from LLaVA-OneVision.
• CosyVoice: Stream-Omni uses the tokenizer and flow model of CosyVoice.
• UltraEval-Audio: Some normalization processing during evaluation refer to UltraEval-Audio.
• VisIT-Bench: Stream-Omni constructs SpokenVisIT benchmark based on VisIT-Bench for the evaluation of vision-grounded speech interaction.

🖋Citation

If this repository is useful for you, please cite as:

@misc{streamomni,
      title={Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model}, 
      author={Shaolei Zhang and Shoutao Guo and Qingkai Fang and Yan Zhou and Yang Feng},
      year={2025},
      eprint={2506.13642},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.13642}, 
}

If you have any questions, please feel free to submit an issue or contact [email protected].