Introduction

In the current era of artificial intelligence, efficient training and inference of large models based on domestic computing power stands as a strategic priority for national development. DeepSeek-V4-Pro is a leading representative among industry large models in terms of efficiency and performance. While DeepSeek-V4-Pro has already been successfully deployed for inference on domestic computing infrastructure, its training pipeline has not yet fully migrated to domestic hardware.

Against this background, the AI Training Platform project team at Shenzhen Loop Area Institute (SLAI), in collaboration with Harbin Institute of Technology, Shenzhen (HITSZ), Shenzhen Research Institute of Big Data (SRIBD), Huawei Global Technical Services (GTS), Computing Product Line, 2012 Labs, and the Shenzhen Smart City AI Computing Platform, conducted joint research to tackle key technical challenges in large model training on domestic computing infrastructure. Within just one month, the project achieved stable full-parameter continual pre-training and SFT for DeepSeek-V4-Pro on an Ascend 910C domestic computing cluster, sustaining over 1,500 steps of uninterrupted training, with model FLOPs utilization (MFU) exceeding 30% and a 14% improvement in the efficiency of key training operators.

To the best of our knowledge based on publicly available information, this work represents the first documented instance of a third-party institution completing full-parameter post-training of DeepSeek-V4-Pro on a domestic computing cluster. It marks a significant milestone in the evolution of domestic AI infrastructure: advancing beyond inference deployment and lightweight fine-tuning toward full-parameter post-training of ultra-large models.

01 Why is "Full-Parameter Post-Training" of Trillion-Parameter Models Such a Hard Nut to Crack?

DeepSeek-V4-Pro is a 1.6-trillion-parameter MoE open-source flagship model that incorporates novel mechanisms such as CSA + HCA hybrid sparse attention and mHC connections. Compared to its previous generation, DeepSeek-V3/R1, V4-Pro poses extreme, multifaceted challenges to domestic training frameworks:

Sparse MoE structure: Expert routing introduces cross-node communication overhead that is orders of magnitude greater than that of traditional dense models.

Hybrid sparse attention: Dynamic switching between attention patterns is extremely sensitive to operator efficiency and memory management;

Trillion-parameter scale: Weights, gradients, activations, and optimizer states together require multiple terabytes of VRAM for a single replica alone.

02 Core Progress: 1,500+ Steps of Stable Training on a Thousand-Card Cluster

Through joint efforts, the project has successfully achieved stable full-parameter post-training of DeepSeek-V4-Pro on a thousand-card-scale Ascend 910C domestic computing cluster.

Sustained SFT training: Completed over 1,500 iteration steps with zero skipped iterations and zero NaN iterations.

Training efficiency: MFU reached approximately 30%, with the computational efficiency of key training operators improved by approximately 14% over the initial version;

Peak performance: MFU stabilized at 34.9% on the Ascend SuperPoD,.

In parallel, the full-parameter continual pre-training and SFT pipeline for DeepSeek-V4-Flash has also been successfully integrated.

DeepSeek-V4-Pro Training on Ascend SuperPoD, MFU Stabilized at 34.9%

A single set of numbers speaks volumes: At comparable parameter scales, there are virtually no publicly documented cases of full-parameter post-training on domestic computing infrastructure. Yet our team ran a 1.6T MoE model on a thousand-card cluster at a steady pace of 27 seconds per step for over 1,500 consecutive steps. This is not a one-off laboratory demonstration;, it is a stable, reproducible capability ready for production-grade delivery.

More importantly, the platform has rapidly proven its value in vertical domains. Focusing on industrial-scale automated operations research modeling, the team completed a closed-loop validation within weeks, spanning data production, sample selection, training pipeline integration, and performance evaluation. This implication is clear: domestic computing platforms can not only "train large models" but "train domain-specific models well"—building enhanced capabilities for specialized tasks at lower cost and in shorter cycles.

03 Three Key Technical Breakthroughs: From "Able to Run" to "Trainable, Stable, andOptimal"

This project focused on full-parameter post-training of DeepSeek-V4-Pro, rather than lightweight fine-tuning methods such as LoRA. The joint team achieved systematic breakthroughs across three layers:

1. Distributed Placement: The "VRAM Jigsaw" of 1.6T Parameters

Insight: Handling trillion-parameter models is not just about having enough VRAM; it is about placing data intelligently.

The project successfully devised a distributed placement scheme for weights, gradients, activations, and optimizer states, orchestrating the interplay of data, tensor, pipeline, and expert parallelism. Determining what goes on each card, how it is arranged, and how scheduling adapts dynamically—solving this "VRAM jigsaw" is the foundation upon which stable training is built.

2. Sparsity and Communication: Preventing Experts from "Quarreling" and Attention from "Gridlocking"

Insight: The two greatest threats to MoE model training are expert load imbalance and cross-node communication storms.

The team performed deep adaptation and optimization of key training operators, including hybrid sparse attention, MoE routing, normalization, and matrix computation, improving operator efficiency by 14% over the initial version. Meanwhile, a real-time expert load monitoring and balancing mechanism was established to prevent scenarios where some experts are overloaded while others sit idle.

3. Long-Horizon Monitoring: Who Keeps Watch Through Days and Nights of Training?

Insight: The worst nightmare in full-parameter post-training is not slow speed, but abrupt mid-training crashes.

The joint team built a comprehensive monitoring system covering loss curves, gradient norms, expert loads, VRAM utilization, and automatic anomaly recovery. Every metric is visualized, alertable, and self-healing. Throughout the 1,500+ training steps, not a single instance of loss divergence or NaN values occurred, serving asthe most direct proof of long-horizon stability.

04 Practical Verification: A Significant Leap in Mathematical Modeling Capability Through Post-Training

To verify the real-world value of full-parameter post-training of DeepSeek-V4 on the Ascend cluster, the project designed a rigorous experiment that involves enhancing the mathematical modeling capability of a large model.

The team built an SFT modeling data production workflow that generated 3,000 high-quality SFT samples for mathematical modeling tasks, covering four target tasks types and three problem formats.DeepSeek-V4 then underwent full-parameter post-training on these samples.

Optimization of Modeling SFT Data Flywheel Process

The training curves delivered clear signals:

LM loss dropped rapidly from the initial level, ultimately converging to 0.2056;

MTP-1 loss converged to 0.2538;

Gradient norms decreased steadily with no oscillation or divergence;

Per-step time held stable at approximately 27 seconds.

Overview of 3K SFT Training Process

The most compelling evidence, however, came from benchmark comparisons:

All four key metrics significantly surpassed the original model, with ORGEval WL improving by over 5 percentage points. Together, these results demonstrate that full-parameter post-training on domestic computing infrastructure can not only ensures "stable operation" but also delivers "superior training outcomes"—significantly enhancing the model's capabilities in complex reasoning tasks.

05 Forging Talent Cultivation Through Real-World Challenges: Cultivating Experts Who Can Truly Train Large Model

Another distinctive value of this project lies in its role as a paradigm experiment in talent cultivation.

SLAI turned this trillion-parameter model training project into a "training ground", embedding students directly in real-world domestic computing scenarios. To date, the project has cultivated 42 students and established a collaborative mentorship model in which young faculty provide guidance, doctoral students tackle core technical challenges, and engineering teams offer hands-on support.

Throughout this process, students were not merely passive participants —they serve as active frontline practitioners with concrete responsibilities, including construction of training data, analysis of sample quality, verification of distributed parallel strategies, follow-up on long-horizon monitoring and anomaly recovery, as well as production of technical reports and engineering documentation.

One training launch. One error traced. One parameter tuned., one post-mortem reviewed. Through these granular, real-world engineering tasks, students genuinely progressed from "knowing how to call a large model" to "understanding and participating in training one". Their growth is reflected in three dimensions:

Developed a systematic understanding of the full pipeline for large model training on domestic computing infrastructure;

Acquired hands-on proficiency spanning the entire workflow from domain data preparation to model capability enhancement;

Built engineering literacy in problem decomposition, experimental design, training post-mortems, and team collaboration within live production environment.

Moving forward, these real-world tasks will be consolidated into course cases, practical training resources, and student research projects that continuously support SLAI’s mission of cultivating high-level interdisciplinary AI talents who "understand models, understand systems, can engineer solutions, and dare to tackle hard problems"

Project Team Conducts Technical Review and Student Hands-on Training

06 Future Outlook: From Full-Parameter Post-Training to Agentic RL and Ultra-Long Context

In the next phase, SLAI will continue to collaborate with existing partners to advance three priority tasks building on the established full-parameter continual pre-training andSFT pipeline.

1. Further Breakthroughs in Training Efficiency

Continue optimizing the training framework and key operators to further improve training efficiency (i.e. MFU) and reduce the computing cost of training trillion-parameter models.

2. Ultra-Long Context Training

Enable training with ultra-long contexts ranging from 512K to 1M tokens, strengthening long-document comprehension and long-chain reasoning capabilities for complex professional tasks.

3. Reinforcement Learning Post-Training Closed Loop

Break through reinforcement learning post-training for DeepSeek-V4-Pro and build a complete Agentic RL pipeline around tasks such as mathematical modeling optimization, code agents, and long-context reasoning. This pipeline integrates rollout generation, tool execution, reward computation and verification, policy updates, and evaluation feedback into a seamless loop.

In parallel, the project is dedicated to advancing open-source initiatives and consolidating talent development outcomes, including:

Phased open-source release of training configurations, evaluation scripts, synthetic data, technical reports, and related model and framework capabilities;

Transformation of real training tasks, data construction methodologies, troubleshooting cases, and evaluation workflows into course materials and hands-on training modules;

Continuously refinement of the talent cultivation pathway defined by "domestic computing support as the foundation, real-world tasks as the driver, student teams learning through practice, and the systematic accumulation of engineering capabilities".

Looking further ahead, the project will leverage this training ground to explore the foundational theories, new paradigms, and novel architectures for next-generation artificial general intelligence—including the organic integration of symbolism, connectionism, and behaviorism, and modeling exploration from single agents to swarm intelligence and ultimately to human-machine integration.

Conclusion

Achieving full-parameter post-training of DeepSeek-V4-Pro on domestic computing infrastructure represents more than an isolated engineering breakthrough.

It validates a viable pathway: domestic open-source flagship models, domestic AI computing infrastructure, elite training teams, and technical support from domestic manufacturers — a four-pillar sustainable, replicable, and reliable close loop.

It makes a declaration: from this point forward, domestic AI infrastructure is no longer confined to "able to infer," but truly trainable, stable and optimal.

More importantly, it ignites a hope: on the journey toward next-generation artificial general intelligence, Chinese universities, research institutions, and young students can stand on their own computing soil, conduct artificial intelligence research, and train their own trillion-parameter models with their own hands.

This is not the finish line. It is the starting gun.

The project will progressively open-source its technical reports, training configurations, and evaluation scripts. Please stay tuned for official updates from SLAI.