Skip to content

Does BitNet-embedding-0.6B / BitNet-embedding-270M support GPU inference? #599

Description

@yazdanifar

The README's "Model Releases" section gives BitNet-b1.58-2B-4T an explicit
"GPU Support: Official GPU inference kernel available for accelerated
deployment" bullet, but the BitNet-embedding-0.6B and BitNet-embedding-270M
sections have no equivalent bullet — only CPU/prefill-speedup claims.

The gpu/README.md also states the W2A8 CUDA kernel "is tailored for use
with the BitNet-b1.58-2B-4T model," and the HF model cards for the two
embedding models only document bitnet.cpp/llama-embedding (CPU) usage.

Trying to run the embedding models' I2_S GGUF through a CUDA-enabled
llama-embedding build (even with -ngl 0) currently crashes with:

GGML_ASSERT(nb10 % sizeof(src1_t) == 0) failed

in ggml-cuda/binbcast.cu, because the CUDA backend still gets registered
and the scheduler routes some ops (e.g. certain flash-attention configs)
to CUDA0 regardless of -ngl.

Could you confirm:

  1. Is GPU inference for the embedding models officially unsupported today?
  2. Is there a plan to extend the gpu/ W2A8 kernel (or fix the llama.cpp
    CUDA binbcast bug) to support these embedding architectures
    (Qwen3-0.6B / Gemma3 backbones)?

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions