EximiusLabs/fusion-embedding-1-2b-preview · Hugging Face

Fusion Embedding 1 extends Qwen3-VL-Embedding-2B with an audio modality. A trained connector (~16M parameters) maps frozen Qwen2.5-Omni audio-tower features into the base model's embedding space; the base model itself is unmodified. The result is a single embedding space covering text, images, video, and audio, with retrieval supported in any direction between modalities.

Highlights

• Leads every unified embedding model we measured on audio↔text. On a single cross-modal protocol, this model exceeds ImageBind, LanguageBind, and Gemini Embedding 2 on audio↔text in both directions, and both language-bound baselines on emergent audio↔image (full tables below).
• Unmodified base. Only the connector is trained; the base model's parameters are byte-identical to the original release, so its text/image/video retrieval performance (MMEB-V2) carries over unchanged.
• Emergent cross-modal alignment. The connector is trained exclusively on audio–text pairs. Audio→image retrieval nonetheless reaches R@10 0.407 over 696 VGGSound candidates (chance: 0.014) with no audio-visual pairs in training — alignment to text places audio in the space the base already shares across modalities.
• Matryoshka representation. Embeddings truncate to {2048, 1536, 1024, 512, 256, 128, 64} dimensions with renormalization.
• Compact distribution. This repository ships the connector and normalization statistics (~60 MB); the frozen towers are downloaded from their original repositories. The parameter count shown for this repository (16.4M) is the trained connector — model.safetensors and the .pt checkpoint contain the same weights; inference.py loads the .pt.

This is a research preview, currently at v0.3: the v0.2 contrastive stage (484K pairs) followed by a connector-only in-domain fine-tune on the AudioCaps train split. Earlier versions remain downloadable via the v0.1-preview and v0.2-preview tags; v0.3-preview pins the current version. All are compared below; pin a tag if you build on this model.

Architecture

A perceiver-resampler (width 384, 64 latent queries) translates frozen audio-tower frames into the base model's input embedding space; its outputs occupy placeholder positions in the input stream, mirroring the base model's image-token mechanism. Training is contrastive (InfoNCE over the Matryoshka ladder, symmetric, with a full-corpus frozen-text negative bank — 484K captions at v0.2) against the base model's text embeddings in its native input format. v0.3 adds a second, connector-only fine-tuning stage on the AudioCaps train split (400 steps at a reduced learning rate), warm-started from the v0.2 checkpoint.

Input formatting. All inputs use the base model's chat-template format (instruction in the system turn, content in the user turn, last-token pooling). Embedding quality is sensitive to this formatting; use the templates in inference.py. For cross-modal ranking, per-modality mean-centering of the gallery is recommended (FusionEmbedder.center).

Evaluation

Cross-modal retrieval — versus unified embedding models

VGGSound-AV, 696 audio/video-frame pairs (chance R@10 = 0.014). R@10 shown as audio-side → other / other → audio-side:

ImageBind trains directly on audio–image pairs, so that pair is its supervised direction; its audio–text alignment is emergent. LanguageBind trains audio against language (its audio↔text is supervised; the value shown is its best readout, using the audio branch's own text tower); its audio↔image is emergent. This model trains on audio–text only; its audio–image alignment is emergent. All models evaluated with identical clips, frames, and scoring, using the released imagebind_huge checkpoint and revision-pinned LanguageBind checkpoints (LanguageBind_Audio_FT + LanguageBind_Image). Note on LanguageBind: its branches fine-tune separate copies of the text tower, which diverge (mean caption cosine 0.55 between the audio and image branches' text embeddings) — the cross-branch binding weakens, which is consistent with its emergent audio↔image score. This model's shared space cannot drift by construction (the base is frozen; every training run asserts parameter-level identity). Gemini Embedding 2 is Google's natively multimodal embedding API (text/image/video/audio in one space), evaluated at its documented default invocation (model id gemini-embedding-2, 3072-d native output, inline audio+image+text, google-genai 2.10.0) on the evaluation date shown; API models may change after that date. One shared caveat: the evaluation captions are model-generated, which could favor models whose text tower shares that caption style — all models received identical inputs.

Full audio→image metrics (per-modality mean-centered gallery — the readout implemented by FusionEmbedder.center; chance R@10 = 0.014):

The v0.3 in-domain fine-tune costs ~1 point of emergent audio→image alignment while improving audio↔text (see the cross-modal table); v0.2 remains available if audio→image is the primary use case.

What audio→image retrieval looks like. These numbers are not only aggregates — the retrievals are organized by sound. Real examples (v0.2 checkpoint) on VGGSound-696 (query clip's frame left, top-5 retrieved images right; green = the clip's exact frame):

Example frames from the VGGSound dataset (CC-BY-4.0), shown for evaluation illustration.

Direct hits — the clip's own frame is returned in the top 5, among the same kind of scene:

Right neighbourhood — the exact frame ranks lower (often a poor still), but the top results are the correct sound category:

MAEB (beta)

On 10 tasks of the MTEB team's Massive Audio Embedding Benchmark (mteb 2.18.0, v0.2 checkpoint; ranks vs the live leaderboard as of 2026-07-09, 21–65 models per task): UrbanSound8K T2A retrieval #3, Ravdess zero-shot #4, FSD2019Kaggle #6 (disclosed only — 13.6% of its test clips appear in the FSD50K dev split used in training, verified by Freesound id, so it is withheld from the official submission), BeijingOpera #6, with mid-field placements on speech/music tasks the model was never trained for. Official leaderboard submission in progress.

Audio–text retrieval — versus specialist CLAP models

AudioCaps test — 883 clips, five reference captions per clip, recall computed as min-rank over references:

CLAP-family models fine-tune both encoders end-to-end and include AudioCaps and Clotho training data; this model keeps both towers frozen and trains only the connector.

Clotho v2.1 evaluation — 1,045 clips × 5 references, zero-shot (Clotho is excluded from training data):

v0.3's in-domain AudioCaps stage trades 1.5 points of zero-shot Clotho A→T for the AudioCaps gains above; T→A improves in both settings.

Text, image, and video benchmarks are the base model's published MMEB-V2 results, which are unaffected by this extension.

Usage

# pip install git+https://github.com/Eximius-Labs/fusion-embedding-1  (+ transformers, torchvision, pillow)
from inference import FusionEmbedder

fe = FusionEmbedder.from_pretrained("EximiusLabs/fusion-embedding-1-2b-preview",
                                    device="cuda")
# or pin a version: revision="v0.3-preview" (current) / "v0.2-preview" / "v0.1-preview"

a = fe.embed_audio("dog_barking.wav")                        # [2048]
t = fe.embed_text("a dog barks while rain falls")            # [2048]
i = fe.embed_image("dog_photo.jpg")                          # [2048]

print((a @ t), (a @ i), (t @ i))                             # cosine similarities

a256 = fe.embed_audio("dog_barking.wav", dim=256)            # Matryoshka truncation

Training data and license

v0.2 was trained on ~484K audio–caption pairs: the full AudioCaps train split (45K), FSD50K, WavCaps/AudioSet_SL, and a 318K-clip subset of LAION-FreeSound, using 10-second training windows (random crop for longer clips). v0.3 continues the v0.2 checkpoint with a 400-step fine-tune on the AudioCaps train split only. v0.1 used a 131K-pair subset of the same sources. As this mix includes YouTube-sourced and research-licensed corpora, the preview is released under CC-BY-NC-4.0. Evaluation sets (AudioCaps test, Clotho, VGGSound, ESC-50) are excluded from training by clip id.

Limitations

• Trained on sound-event data; speech content, speaker attributes, and music description are supported by the instruction taxonomy but not yet trained to comparable quality.
• English captions; 16 kHz mono input; 30 s per window (longer audio is chunked).
• Audio–text retrieval is below fully fine-tuned CLAP-family models at this checkpoint (see Evaluation).

Roadmap

Further corpus scaling, speech and music coverage, a commercially licensed release tier, and the 8B model.

Citation

@software{fusion_embedding_2026,
  title  = {Fusion Embedding 1: A Unified Embedding Space for Text,
            Image, Video, and Audio},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  url    = {https://github.com/Eximius-Labs/fusion-embedding-1}
}

Built on Qwen3-VL-Embedding and Qwen2.5-Omni, with training data from AudioCaps, WavCaps, and FSD50K.