Run Qwen3.6-35b-a3b at home

This is ~70% as good as Qwen3.6 27b, but it runs on a potato

Any ~8GB+ Nvidia card

https://github.com/ggml-org/llama.cpp/releases/download/b10213/llama-b10213-bin-win-cuda-13.3-x64.zip
https://github.com/ggml-org/llama.cpp/releases/download/b10213/cudart-llama-bin-win-cuda-13.3-x64.zip

unzip both of those into the same folder. Like c:\ai\llama

Get this model (~17GB): https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf

Put the model in c:\ai\models

Save this in c:\ai as a powershell script named: “start-llama-server.ps1”

# Do Start | Run | powershell -ExecutionPolicy Bypass -File "C:\ai\start-llama-server.ps1"
#
# Server will be reachable at http://127.0.0.1:8080 (and on your LAN, since --host is 0.0.0.0).

Set-Location "C:\ai\llama"

.\llama-server.exe `
    --model "C:\ai\models\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf" `
    --alias "Qwen3.6-35B-A3B" `
    -ngl 99 `
    --n-cpu-moe 32 `
    -c 32768 `
    --flash-attn 1 `
    --cache-type-k q8_0 `
    --cache-type-v q8_0 `
    -np 1 `
    -t 8 `
    -b 2048 `
    --ubatch-size 512 `
    --reasoning off `
    --temp 0.7 `
    --top-p 0.8 `
    --top-k 20 `
    --min-p 0.00 `
    --presence-penalty 1.5 `
    --host 0.0.0.0 `
    --port 8080

…That’s it. You can access it via localhost:8080 or on any computer / ipad / phone /web device via http://ip.address:8080