inclusionAI/Ling-3.0-flash

Overview

inclusionAI/Ling-3.0-flash uses the BailingMoeV3ForCausalLM architecture with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token), one shared expert, and a native multi-token prediction head. The 42-layer base model has 124.4B total and 5.5B active parameters. The checkpoint also contains a 3.1B MTP layer, bringing the complete checkpoint to 127.5B parameters. A serialized block-FP8 checkpoint is available as inclusionAI/Ling-3.0-flash-fp8.

Prerequisites

• vLLM: a build containing native Bailing V3 support;
• Validated hardware: NVIDIA H20, H20-3e, and H200
• Precision: BF16 or serialized block FP8 weights with BF16 compute
• Context length: 262,144 tokens

Launching the Server

NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For the serialized FP8 variant on 2x H200, use:

vllm serve inclusionAI/Ling-3.0-flash-fp8 \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

For the validated TP4+EP4 alternative on 4x H200, use:

vllm serve inclusionAI/Ling-3.0-flash-fp8 \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

The native MTP head is available with either FP8 topology by adding:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Single-GPU int4

inclusionAI/Ling-3.0-flash-int4 is a compressed-tensors / pack-quantized checkpoint: symmetric W4, group_size: 32, applied to the routed experts only — attention, lm_head, the shared expert and the dense projections are all in the config's ignore list. It is not covered by the model-specific quantization plumbing in bailing_moe_v3.py (which handles block FP8 and mxfp4); it loads through vLLM's generic compressed-tensors path, and that is enough to serve it on a single H200 at TP=1:

vllm serve inclusionAI/Ling-3.0-flash-int4 \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching

The engine names the mechanism at startup:

quantization=compressed-tensors ... trust_remote_code=True, dtype=torch.bfloat16
INFO [int_wna16.py:297] Using 'MARLIN' WNA16 MoE backend.
INFO [cuda.py:492]      Using FLASH_ATTN_MLA attention backend
INFO [int_wna16.py:409] Using MarlinExperts

The vendor model card documents SGLang only, and there is no merged vLLM PR specific to Ling int4 — this variant is recorded because the generic path works, not because there is model-specific support behind it. Tool-call and reasoning parsers, chunked prefill and CUDA graph capture behave as they do on the other variants.

Validation

Both the default FP8 TP2 path and the TP4+EP4 path were validated on NVIDIA H200. TP2 remains the recommended default because it uses fewer GPUs; TP4+EP4 is an alternative for deployments that prefer expert parallelism.

The int4 variant was verified on 1x H200 (SM90), TP=1, on a main build reporting 0.26.1rc1.dev1133+gf94666b60, serving at --max-model-len 32768: 77.0 GB of weights on disk, 70.27 GiB resident after load (165 s), 5 min 56 s from container start to ready, FULL_AND_PIECEWISE capture (68 PIECEWISE + 68 FULL, 2.15 GiB), and a 4,072,594-token KV pool. Single-stream decode measured 159.5 tok/s at TTFT p50 0.17 s; 973 output tok/s aggregate at concurrency 16, with 0 failed requests and 0 preemptions over a 1..24 concurrency sweep.

Thinking Mode

Thinking is selected per request through the chat template rather than by a server flag:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=0.0,
    max_tokens=200000,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)

When serving the FP8 variant, set model="inclusionAI/Ling-3.0-flash-fp8" in the client request.

References

• Ling-3.0-flash on Hugging Face
• Ling-3.0-flash-fp8 on Hugging Face
• Ling-3.0-flash-int4 on Hugging Face