Note: If you have a multi-GPU SM120 Blackwell system (RTX 50/Pro), try my vLLM fork to resolve P2P / TP=2 issues (Pending PR into upstream).

https://github.com/Gadflyii/vllm/tree/main

GLM-4.7-Flash NVFP4 (Mixed Precision)

This is a mixed precision NVFP4 quantization of zai-org/GLM-4.7-Flash, a 30B-A3B (30B total, 3B active) Mixture-of-Experts model.

Quantization Strategy

This model was made via custom quantization and calibration (128 samples, 2048 max seq len, neuralmagic/calibration, all 64 experts) scripts based on NVIDIA's approach for DeepSeek-V3. It uses mixed precision to preserve accuracy:

Performance

Usage

Requirements

• vLLM: 0.14.0+ (for compressed-tensors NVFP4 support)
• transformers: 5.0.0+ (for glm4_moe_lite architecture)
• GPU: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace)

Installation

pip install vllm>=0.14.0
pip install git+https://github.com/huggingface/transformers.git

Inference with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    "GadflyII/GLM-4.7-Flash-NVFP4",
    tensor_parallel_size=1,
    max_model_len=4096,
    trust_remote_code=True,
    gpu_memory_utilization=0.85,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(["Explain quantum computing in simple terms."], params)
print(outputs[0].outputs[0].text)

Serving with vLLM

vllm serve GadflyII/GLM-4.7-Flash-NVFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --trust-remote-code

Model Details

• Base Model: zai-org/GLM-4.7-Flash
• Architecture: Glm4MoeLiteForCausalLM
• Parameters: 30B total, 3B active per token (30B-A3B)
• MoE Configuration: 64 routed experts, 4 active, 1 shared expert
• Layers: 47
• Context Length: 202,752 tokens (max)
• Languages: English, Chinese

Quantization Details

• Format: compressed-tensors (NVFP4)
• Block Size: 16
• Scale Format: FP8 (E4M3)
• Calibration: 128 samples from neuralmagic/calibration dataset
• Full Expert Calibration: All 64 experts calibrated per sample

Evaluation

MMLU-Pro Overall Results

MMLU-Pro by Category

MMLU-Pro by Subject

Citation

If you use this model, please cite the original GLM-4.7-Flash:

@misc{glm4flash2025,
  title={GLM-4.7-Flash},
  author={Zhipu AI},
  year={2025},
  howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}
}

License

This model inherits the Apache 2.0 license from the base model.