Qwen3.6-35B-A3B

Qwen3.6-35B-A3B

35 billion parameters, three active, and a layer layout Qwen publish as a formula.

The Architecture, Exactly as Specified

Qwen give the hidden layout in one line, and it is worth reading literally:

10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))

A four-layer block — three linear-attention layers and one full-attention layer — repeated ten times. Forty layers total, of which only ten use full attention.

Why that split. Full attention costs O(n²), which is prohibitive across forty layers at a 256K context. Gated DeltaNet handles local-to-medium-range context through a recurrent state update — constant cost regardless of input length. The ten full-attention layers provide unrestricted global token mixing where exact retrieval matters.

Three quarters of the depth costs nothing extra as the input grows. That is the entire reason a 262,144-token window is practical on a model with three billion active parameters.

Full Specification

Gated DeltaNet — the linear attention layers

Gated Attention — the full attention layers

Sixteen query heads against two key-value heads is an 8:1 ratio, cutting KV cache memory by a factor of eight. Applied only to the ten full-attention layers, where the cache exists at all.

Mixture of Experts

Nine experts of 256 — roughly three and a half percent of the pool per token.

Two Details Worth Knowing

Both come from the architecture analysis, and both explain behaviour you would otherwise attribute to luck.

The shared expert is gated by a learned scalar. Its contribution is sigmoid(Linear(x, 1)) × shared(x) — a single learned value deciding how much the always-on expert contributes for this token. It handles common patterns so the routed experts can specialise, and the gate lets the model decide per token how much of that generic handling it needs.

Load balancing runs on an auxiliary loss with a coefficient of 0.001. It penalises uneven expert utilisation, ensuring all 256 experts get used across a batch.

That coefficient is the interesting part. Load-balancing losses cost quality — you are optimising for two objectives and only one of them is the task. A coefficient of 0.001 is a deliberate choice about how much quality to trade for balance, and it is small.

Multi-Token Prediction

An optional head that predicts a second token from the same hidden state, enabling self-speculative decoding — the model acts as its own draft model, producing two tokens per forward pass.

Trained with multiple steps, per the model card, rather than as a single-step add-on.

No separate draft model is needed. Conventional speculative decoding requires a small companion model that drafts and a large one that verifies — two models to load, two to keep in sync. Here the capability is inside the same weights.

Context: 262K Native, 1M Extended

Note the extended figure is not a round million. 1,010,000 is a measured ceiling rather than a marketing number, which is a small sign the extension was tested rather than asserted.

Behaviour differs between the two. Native context is what the model was trained at. Extension widens the window; it does not guarantee the far end behaves like the near end. Measure on your own data before designing a pipeline that fills it.

Specifications

Official quantisations: FP8 from Qwen, NVFP4 from NVIDIA ModelOpt, plus community AWQ, MLX, and GGUF builds.

Capabilities

201 Languages on a 248K Vocabulary

The two numbers belong together.

A 248,320-token vocabulary is unusually large — most models in this catalogue run between 130K and 160K.

That size is what carries the language count. Covering 201 languages means covering scripts with fundamentally different structures: Latin, Cyrillic, Arabic, Devanagari, Han, Hangul, and dozens more. A smaller vocabulary handles them by splitting words into many tokens, which costs context and degrades quality on exactly the languages least represented in training.

The practical effect for non-English work: fewer tokens per word, which means more text fits in the window and each token carries more meaning.

Using Qwen3.6-35B-A3B on DEVUP AI

Base URL: https://api.devupai.com/v1 · Model ID: Qwen/Qwen3.6-35B-A3B

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DEVUP_API_KEY"],
    base_url="https://api.devupai.com/v1",
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B",
    messages=[
        {"role": "user", "content": "Hello world!"}
    ],
    max_tokens=1024,
)

print(response.choices[0].message.content)

Node.js

import DevupAI from "devupai";

const client = new DevupAI({
  apiKey: process.env.DEVUP_API_KEY,
});

async function main() {
  const response = await client.chat.completions.create({
    model: "Qwen/Qwen3.6-35B-A3B",
    messages: [{ role: "user", content: "Hello world!" }],
    max_tokens: 1024,
  });

  console.log(response.choices[0].message.content);
}

main();

cURL

curl -X POST "https://api.devupai.com/v1/chat/completions" \
  -H "Authorization: Bearer $DEVUP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-35B-A3B",
    "messages": [
      { "role": "user", "content": "Hello world!" }
    ],
    "max_tokens": 1024
  }'

Reading an Image

The vision encoder is part of the model type, not an attachment, so images and text interleave naturally.

import base64

with open("invoice.png", "rb") as handle:
    encoded = base64.b64encode(handle.read()).decode("utf-8")

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{encoded}"}},
                {
                    "type": "text",
                    "text": (
                        "Transcribe every line item with its quantity and amount, and report the "
                        "currency exactly as printed. Mark anything you cannot read cleanly as "
                        "unreadable rather than reconstructing it."
                    ),
                },
            ],
        }
    ],
    max_tokens=8192,
)

Send pages at full resolution. Downscaling before upload discards detail the encoder would otherwise use, and no prompt recovers characters it never received.

Long-Context Work

Where the hybrid attention pays off, and where to be careful.

from pathlib import Path

corpus = "\n\n---\n\n".join(
    f"### {path.name}\n{path.read_text(encoding='utf-8')}"
    for path in sorted(Path("contracts").glob("*.txt"))
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B",
    messages=[
        {
            "role": "system",
            "content": (
                "Review this set of agreements. Identify every obligation stated in one document "
                "that contradicts an obligation in another. Quote both clauses and name both files. "
                "Report nothing you cannot quote."
            ),
        },
        {"role": "user", "content": corpus},
    ],
    max_tokens=16384,
)

print(f"input: {response.usage.prompt_tokens:,} tokens")

Cross-document contradiction is the task that justifies a long window. A conflict between the third document and the eleventh is invisible to any pipeline that reads them one at a time.

Stay inside 262K unless you have measured past it. The extension to a million works; whether retrieval quality at token 900,000 matches token 200,000 is a question about your data, not about the architecture.

Self-Hosting

Well supported, and two version constraints matter.

Official checkpoints: BF16 base, Qwen's FP8, and NVIDIA's ModelOpt NVFP4.

The NVFP4 build has a vLLM version requirement. Version 0.28.0 or later — earlier versions load the checkpoint, but the optimised decode kernel is only selected from 0.28.0 on the relevant hardware. Loading successfully and running optimally are different things here.

Enable the MTP head if your serving stack supports self-speculative decoding. It is trained, included, and where the throughput gain lives.

Community builds cover AWQ 4-bit, MLX for Apple silicon, and GGUF at multiple precisions — which makes this one of the more accessible capable models for local deployment.

Where It Fits

Long-context work at low cost — 262K native on three billion active parameters is an unusual combination, and the hybrid attention is why it exists.

Multilingual deployment across 201 languages, with a vocabulary sized for the scripts rather than retrofitted.

Document and image understanding, through the integrated vision encoder.

Agentic work at volume, where three billion active parameters and self-speculative decoding keep per-call cost low.

Local and edge deployment, with official quantisations and community builds across formats.

Not for peak capability. Three billion active parameters is the compute ceiling per token; this is an efficiency model, and the larger sibling in the same generation is where the capability lives.

Practical Notes

Stay inside 262K unless you have measured the extension on your own data.

Send images at full resolution.

Enable MTP when self-hosting — it is trained and included.

Check the vLLM version if using the NVFP4 checkpoint.

Keep reasoning_content in its own field in both directions.

Consider the larger model in this generation when three billion active parameters is the limiting factor.

Limitations

Three billion active parameters. The sparsity buys speed and footprint; peak capability is bounded by what three billion parameters can compute per token.

Text output only. It reads images; it does not generate them.

262,144 is native; a million is extension. Behaviour at the far end is worth measuring rather than assuming.

Thirty-five billion parameters must be loaded even though three run per token. The MoE saving is in compute, not in memory.

The NVFP4 checkpoint needs a recent vLLM for its optimised kernel — an older version runs it suboptimally without failing.

Reasoning traces are working notes. Unpolished, sometimes exploring abandoned branches, and occasionally contradicting the answer that follows.