We wanted to run Kimi K3 on AMD hardware. Not because it was easy, but because it was the right thing to figure out.
Kimi K3 is a 2.8 trillion parameter mixture-of-experts model from Moonshot AI, with approximately 104 billion active parameters per token. It ships in MXFP4 quantization, a format designed for the newest AMD silicon. The interesting question for us was simple: can you actually run this model on the AMD GPUs that exist in the market today, on a single node, with open source tooling, and get something useful out of it?
The answer is yes. But getting there required a specific sequence of engineering decisions that I want to write down, because I think they are more generally useful than just for this one model.
Here is the core tension.
Kimi K3 uses MXFP4 quantization for its expert weights. MXFP4 is a 4-bit floating point format with microscaled exponents. It is designed for AMD MI355X, which uses the gfx950 architecture (CDNA4). The MI355X has native MFMA instructions that can consume MXFP4 values directly in hardware.
The problem is that MI355X is not widely available yet. The AMD GPU you can actually rent today, on a public cloud, is the MI325X. The MI325X is gfx942, which is CDNA3. It does not have native MXFP4 support.
So if you load Kimi K3 on an MI325X and try to use the native MXFP4 kernels, vLLM raises a runtime error because the hardware does not have the instructions those kernels expect. You cannot just run the model as-is.
This is the kind of problem that sounds like a blocker until you look at it carefully.
We had two realistic AMD options. Here is how they compare:
| GPU | Architecture | HBM per GPU | GPUs Needed for Kimi K3 | Nodes Required |
|---|---|---|---|---|
| AMD MI300X | gfx942 (CDNA3) | 192 GB | 16 | 2 |
| AMD MI325X | gfx942 (CDNA3) | 256 GB HBM3e | 8 | 1 |
| AMD MI355X | gfx950 (CDNA4) | 288 GB HBM3e | 8 (native MXFP4) | 1 |
The MI300X and MI325X share the same architecture family (gfx942), so both require the same MXFP4 workaround. The difference is memory. The MI325X has 256 GB of HBM3e per GPU, which gives you 2 TB across an 8-GPU node. The model in MXFP4 format is approximately 1.5 TB on disk, distributed across 96 safetensors shards. With 2 TB of total HBM, you can hold the model and still have several hundred GB of headroom for KV cache on a single node.
The MI300X has 192 GB per GPU. You can fit the model, but you sacrifice too much KV cache headroom. To run Kimi K3 comfortably on MI300X, you need two nodes and 16 GPUs, which doubles your networking complexity and your cost.
The practical version is the MI325X on a single node. One machine, eight GPUs, 2 TB of total HBM. That is the configuration we went with, on a DigitalOcean GPU droplet running Ubuntu with ROCm.
This is the part I find most interesting.
The MI325X cannot run MXFP4 natively. But it can run int4 groupwise quantized weights very efficiently using the AITER FlyDSL kernels that already exist for AMD MoE inference. The insight in vLLM PR #50319 is that you can convert MXFP4 expert weights into groupwise int4 at model load time, then run the standard bf16-by-int4 MoE path that AITER already supports.
At a high level, the conversion works like this:
The conversion happens once, at load. It adds a few minutes to startup. After that, inference runs on the int4 path with no per-token overhead from the conversion.
You trigger this path in vLLM with a single CLI flag:
--quantization-config.moe.weight int4_per_group_32That flag tells vLLM to use the int4 requantization path instead of trying to run native MXFP4. Internally, a function called _use_k3_situ_int4_gfx942() checks for this flag and switches the MoE backend to the AITER FlyDSL int4 kernels with a tuned configuration table (kimik3_a8w4_tuned_fmoe.csv).
This is where most people will get stuck, so I want to be specific.
You cannot use a released vLLM Docker image. You cannot cherry-pick a few files from the PR. The changes in PR #50319 are too deep for file-level patching against any released image. You have to build vLLM from the PR source.
The build sequence has two stages:
Stage 1: Build vLLM from PR #50319 source
rocm/vllm-dev:base image as your base.Dockerfile.rocm_k3_gfx942 added in the PR, with these build args: REMOTE_VLLM=0, NIC_BACKEND=none, PYTORCH_ROCM_ARCH=gfx942, and --target vllm-openai.Stage 2: Build AITER with PR #4471
This is the one that will bite you if you skip it.
AITER is the AMD Triton-based kernel library that provides the FlyDSL MoE kernels. The released AITER (v0.1.19) hardcodes SiLU activation in the packed-int4 MoE stage 1 GEMM. Kimi K3 does not use SiLU. It uses SiTU-GLU (referred to as SiTUv2 in the code), a custom activation with two beta parameters: situ_beta=4.0 and situ_linear_beta=25.0.
If you use the stock AITER, the MoE stage 1 GEMM applies the wrong activation function. The model will load, it will produce tokens, and those tokens will be garbage. This is the worst kind of bug: it fails silently.
ROCm/aiter PR #4471 fixes this by adding act, situ_beta, and situ_linear_beta parameters to compile_moe_gemm1(). With that PR applied, AITER compiles the MoE kernels with the correct SiTU-GLU activation. The PR is currently open and not yet merged into an AITER release, so you need to build from the PR branch directly.
One operational detail: the PR includes a multi-stage Dockerfile that builds both vLLM and AITER in a single docker build pipeline. You do not need to manually rebuild AITER inside a running container or use docker commit. The Dockerfile handles the layering correctly. If you do choose to rebuild AITER separately inside the container, make sure your custom image preserves the vLLM entrypoint.
Here is the environment we set before launching vLLM:
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_MLA=0
export VLLM_ROCM_USE_AITER_FP4BMM=0
export HIP_FORCE_DEV_KERNARG=1
export SAFETENSORS_FAST_GPU=1
export NCCL_MIN_NCHANNELS=112
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7A few of these need context:
VLLM_ROCM_USE_AITER=1 and VLLM_ROCM_USE_AITER_MOE=1 enable the AITER FlyDSL MoE kernels, which are required for the int4 MoE path.VLLM_ROCM_USE_AITER_MLA=0 disables AITER for the MLA (multi-head latent attention) layers. PR #50319 does include AITER MLA support for MI300X-class hardware, but we kept it disabled during our initial bring-up to isolate variables. You may want to test with it enabled.VLLM_ROCM_USE_AITER_FP4BMM=0 disables the MXFP4 block matmul kernels, which is correct since we are running the int4 path, not native MXFP4.HIP_FORCE_DEV_KERNARG=1 and SAFETENSORS_FAST_GPU=1 are standard vLLM ROCm optimizations.NCCL_MIN_NCHANNELS=112 matches the expert parallelism degree (896 global experts / 8 GPUs = 112 experts per GPU), giving NCCL enough channels for all-to-all communication.And the key CLI flags:
vllm serve /mnt/scratch/kimi-k3/ \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--max-model-len 32768 \
--max-num-seqs 256 \
--gpu-memory-utilization 0.95 \
--enforce-eager \
--quantization-config.moe.weight int4_per_group_32A few of these flags deserve explanation because they are not optional:
--enforce-eager is required. AITER kernels are incompatible with CUDA graph capture on gfx942. If you leave eager mode, vLLM tries to capture CUDA graphs and the AITER kernels fail inside the graph capture path. This is not a performance choice. It is a correctness requirement on gfx942 today.--enable-expert-parallel distributes the 896 global experts across the 8 GPUs, giving 112 local experts per GPU. With tensor parallelism at 8 and expert parallelism enabled, each GPU holds its slice of the shared layers plus its 112 experts.--gpu-memory-utilization 0.95 pushes HBM utilization to 95 percent, leaving just enough room for the model weights (distributed across 8 GPUs after int4 requantization), the int4 expert weights, and a functional KV cache within the 256 GB per-GPU HBM3e pool.--quantization-config.moe.weight int4_per_group_32 must use dot notation. We found that passing the same configuration as a JSON string did not trigger the int4 requantization path when the model config declares quantization=mxfp4. The model loaded in native MXFP4 mode, which fails on gfx942. Use the dot-notation form to be safe.For context, here is what Kimi K3 actually is under the hood:
KimiK3ForConditionalGenerationsitu_beta=4.0 and situ_linear_beta=25.0The SiTU-GLU activation is the detail that makes the AITER PR #4471 non-optional. Most MoE models use SiLU or GELU, which AITER supports out of the box. Kimi K3 is the first widely deployed model we have seen that uses SiTU-GLU in the MoE path, which is why the AITER fix had to land before inference would produce correct outputs.
A short list of things that cost us time, in case they save you some:
quantization=mxfp4. This is a vLLM argument parsing behavior, not an AMD issue, but it will cost you an hour if you do not know about it.Dockerfile.rocm_k3_gfx942) that handles both vLLM and AITER builds. Use docker build with this Dockerfile rather than trying to cherry-pick changes into an existing image.On a single 8-GPU MI325X node, with enforce-eager and the int4 requantization path, we measured (these are our own measurements, not externally benchmarked):
These are not peak numbers. The enforce-eager requirement means we are leaving throughput on the table that CUDA graph capture would normally recover. As AITER and vLLM add graph-compatible kernel paths for gfx942, those numbers will improve without hardware changes.
The point is not the specific tokens per second. The point is that a 2.8 trillion parameter MoE model, shipped in a format designed for next-generation hardware, runs correctly and usefully on the AMD GPUs you can rent today, on a single node, with open source tooling and two community PRs.
I think the lesson here is broader than Kimi K3 or AMD.
When a model ships in a quantization format that targets hardware you do not have yet, the first instinct is to wait. Wait for the new GPUs. Wait for the upstream framework to add backward compatibility. Wait for someone else to figure it out.
The more useful instinct is to ask: what path does the hardware I already have support, and can I convert the model into that path at load time?
In this case, the answer was: gfx942 already has fast int4 MoE kernels via AITER FlyDSL. MXFP4 can be converted to groupwise int4 deterministically. The conversion is a one-time cost at load. After that, inference runs on a path the hardware already knows.
That is a general pattern. It will apply to the next model that ships in a new format, and the next hardware generation that introduces a new instruction set. The tools to bridge the gap are usually already there. You just have to find the right two PRs.
What is Kimi K3?
Kimi K3 is a 2.8 trillion parameter mixture-of-experts model from Moonshot AI, with approximately 104 billion active parameters per token. It uses 896 global experts with top-16 routing and a custom SiTU-GLU activation function in the MoE layers. It ships in MXFP4 quantization, a 4-bit floating point format with microscaled exponents.
Why does Kimi K3 not run natively on MI325X?
The MI325X uses the gfx942 architecture (CDNA3), which does not have native MXFP4 MFMA instructions. MXFP4 was designed for gfx950 (CDNA4), which ships in the MI355X. On gfx942, vLLM raises a runtime error when you try to use native MXFP4 kernels because the hardware does not support the required instructions.
How does the int4 requantization path work?
At model load time, vLLM reads the MXFP4 expert weights, dequantizes them to bf16, and re-quantizes them to groupwise int4 with group size 32. Inference then uses AITER FlyDSL bf16-by-int4 MoE kernels, which gfx942 supports efficiently. The conversion happens once at load and adds no per-token overhead.
What is SiTU-GLU and why does it matter?
SiTU-GLU (called SiTUv2 in the vLLM and AITER source code) is the activation function used in Kimi K3 MoE layers, parameterized by situ_beta=4.0 and situ_linear_beta=25.0. It is not SiLU. Stock AITER kernels hardcode SiLU in the int4 MoE GEMM, which produces incorrect outputs for Kimi K3. ROCm/aiter PR #4471 adds SiTU-GLU support to the AITER MoE compilation path.
Can I run Kimi K3 on MI300X?
Yes, but you need two nodes (16 GPUs) because the MI300X has 192 GB of HBM3 per GPU versus 256 GB of HBM3e on the MI325X. The same int4 requantization path applies since both are gfx942. A single MI325X node is simpler and avoids multi-node networking complexity.
Why is enforce-eager required?
CUDA graph capture is incompatible with AITER kernel compilation on gfx942. If vLLM attempts to capture CUDA graphs, the AITER kernels fail inside the graph capture context and the server crashes. Eager mode is a correctness requirement, not a performance tuning choice, until AITER adds graph-compatible kernel paths for gfx942.
What throughput should I expect?
On a single 8-GPU MI325X node with enforce-eager, we measured approximately 10 tokens per second decode and 5.6 tokens per second prefill for a single request. Throughput will improve as AITER and vLLM add CUDA graph compatibility for the int4 MoE path on gfx942.
Do I need to build vLLM from source?
Yes. The changes in vLLM PR #50319 are too extensive to cherry-pick against any released vLLM image. You need to clone the PR branch and build the full ROCm Docker image using Dockerfile.rocm_k3_gfx942 with PYTORCH_ROCM_ARCH=gfx942.
Is this production-ready?
It is functional and correct for inference. The enforce-eager requirement means you are not getting peak throughput, and the build chain requires tracking two community PRs (vLLM #50319 and AITER #4471). For production deployment, you would want to pin both PR commits and build a reproducible Docker image. The model produces correct outputs. The infrastructure around it is still maturing.
Will this get easier?
Yes. The vLLM and AITER PRs are on track to merge upstream. Once they do, released vLLM ROCm images will include the int4 requantization path and SiTU-GLU support, and the build chain collapses to pulling an image. The underlying approach (convert at load, run on the hardware path you have) will still be the right pattern for future format mismatches.
