The README's "Model Releases" section gives BitNet-b1.58-2B-4T an explicit
"GPU Support: Official GPU inference kernel available for accelerated
deployment" bullet, but the BitNet-embedding-0.6B and BitNet-embedding-270M
sections have no equivalent bullet — only CPU/prefill-speedup claims.
The gpu/README.md also states the W2A8 CUDA kernel "is tailored for use
with the BitNet-b1.58-2B-4T model," and the HF model cards for the two
embedding models only document bitnet.cpp/llama-embedding (CPU) usage.
Trying to run the embedding models' I2_S GGUF through a CUDA-enabled
llama-embedding build (even with -ngl 0) currently crashes with:
GGML_ASSERT(nb10 % sizeof(src1_t) == 0) failed
in ggml-cuda/binbcast.cu, because the CUDA backend still gets registered
and the scheduler routes some ops (e.g. certain flash-attention configs)
to CUDA0 regardless of -ngl.
Could you confirm:
- Is GPU inference for the embedding models officially unsupported today?
- Is there a plan to extend the
gpu/ W2A8 kernel (or fix the llama.cpp
CUDA binbcast bug) to support these embedding architectures
(Qwen3-0.6B / Gemma3 backbones)?
Thanks!
The README's "Model Releases" section gives
BitNet-b1.58-2B-4Tan explicit"GPU Support: Official GPU inference kernel available for accelerated
deployment" bullet, but the
BitNet-embedding-0.6BandBitNet-embedding-270Msections have no equivalent bullet — only CPU/prefill-speedup claims.
The
gpu/README.mdalso states the W2A8 CUDA kernel "is tailored for usewith the BitNet-b1.58-2B-4T model," and the HF model cards for the two
embedding models only document
bitnet.cpp/llama-embedding(CPU) usage.Trying to run the embedding models' I2_S GGUF through a CUDA-enabled
llama-embeddingbuild (even with-ngl 0) currently crashes with:in ggml-cuda/binbcast.cu, because the CUDA backend still gets registered
and the scheduler routes some ops (e.g. certain flash-attention configs)
to CUDA0 regardless of -ngl.
Could you confirm:
gpu/W2A8 kernel (or fix the llama.cppCUDA binbcast bug) to support these embedding architectures
(Qwen3-0.6B / Gemma3 backbones)?
Thanks!