StepFun Open-Sources Step 5 Preview: 600B MoE Model with Million-Token Context for Long-Horizon Agents

StepFun Open-Sources Step 5 Preview Model Weights, Focusing on Long-Horizon Agent Tasks with a 600B Sparse MoE Architecture Supporting Million-Token Context and Multimodal Input.

Dual-precision weights (BF16/FP8) are released; the model natively supports cross-file code modification-execution-verification loops; it emphasizes goal consistency and intermediate state management in long contexts.

Target audience: AI architects, agent system engineers, and LLM deployment engineers.

On October 15, StepFun announced the official release of Step 5 Preview weights, including BF16 and FP8 versions, along with a technical blog post, accompanying open-source development tools, and deployment guides. The model weights are now available on Hugging Face and ModelScope.

Hugging Face URL: https://huggingface.co/stepfun-ai/Step-5-Preview

ModelScope URL: https://www.modelscope.ai/models/stepfun-ai/Step-5-Preview

According to the introduction, Step 5 Preview is designed for long-horizon agent tasks. It employs a sparse Mixture-of-Experts (MoE) architecture with a total parameter count of 600B and 27B active parameters per token. It supports a context window of 1 million tokens, natively handles text and visual inputs, and targets application scenarios such as software engineering, professional knowledge work, and finance. The official model description lists cross-file code modification, execution, and verification, as well as iterative tool calling and result-based action correction, as key capability directions.

These tasks impose different requirements on models compared to single-turn Q&A. Taking software engineering as an example, an agent must first understand requirements and code, then modify files, execute tests, check errors, and deliver results after multiple attempts. As the task progresses, code, tool return information, and intermediate decisions accumulate continuously. The model must handle increasingly long inputs while retaining early constraints to prevent subsequent modifications from deviating from the original goal.

Therefore, evaluating the actual value of long-horizon agents requires simultaneously examining task completion rates, execution time, resource consumption, and the number of human interventions.

The technical blog disclosed the design philosophy of Step 5 Preview, including a 92-layer narrow-deep architecture, Sparse GQA, training stability optimizations, million-scale verifiable environments, and reinforcement learning for long-horizon tasks. Several specific choices reflect how the team addresses computational and training challenges in long tasks.

Architecturally, the model uses a 92-layer Transformer, alternating between gated sliding-window attention and sparse global attention at a 3 ratio. Sparse GQA merges adjacent 8 tokens into index blocks; according to the team, this reduces indexing and top-k selection overhead to one-eighth of the original. Regarding training stability, the team uses FP32 for hidden states, related gradients, and normalization scaling parameters, and implements fine-grained gradient clipping.

It is necessary to distinguish between local operator optimization and overall performance gains. A reduction in indexing and selection overhead does not equate to an eight-fold speedup in the complete inference pipeline, nor can it directly imply that service costs decrease by the same proportion. Actual deployment also incurs costs for weight loading, attention computation, cross-device communication, and tool execution. For developers, more meaningful validation involves observing the time and resource usage for a complete task under identical hardware, context length, and concurrency conditions.

Total parameters of 600B and active parameters of 27B should be understood separately: sparse activation reduces the number of parameters participating in computation per token, but expert weights not activated by the current token still constitute model resources that the deployment system must manage. Therefore, one cannot equate its deployment threshold to that of a dense model of equivalent scale based solely on the 27B active parameters.

Regarding training data, the team claims to have constructed million-scale verifiable environments covering scientific reasoning, software engineering, machine learning R&D, and professional work. The blog also notes that synthetic agent data tends to bias towards simple, repetitive tasks, requiring explicit control over difficulty and diversity, and checking whether the model exploits evaluation loopholes to gain rewards.

This issue relates to whether agent training results can transfer to real-world work. Passing tests might mean the code is correct, or it might mean the tests did not cover critical conditions; improved task scores might stem from capability enhancement, or from the model finding shortcuts in the scoring rules. For long-horizon tasks, the value of training environments depends not only on quantity but also on the ability to distinguish truly completing tasks from superficially meeting metrics. After releasing scale information about million-level environments, their coverage, verification quality, and differences from actual business scenarios remain worth continuous observation.

Another noteworthy design is context compression. Step 5 Preview incorporates this into end-to-end policy training, optimizing with the same task reward, enabling the agent to organize work into a recoverable state when the context budget is nearly exhausted, and then continue execution.

From an application perspective, this means context management itself becomes part of what the model needs to learn. Even with a million-token window, continuously running agents still need to decide which information is worth retaining: initial requirements, confirmed facts, failed solutions, and unresolved issues may all influence subsequent actions. Overly aggressive compression may omit critical constraints; retaining too much redundancy increases subsequent computational burden.

Therefore, there remains a gap requiring engineering validation between long-context capacity and long-task reliability. Window size describes how much information the model can receive at once, while whether a task can proceed continuously depends on other factors.

Open weights enable enterprises to further validate how well a model fits their specific workflows. For example, teams can test cross-file modifications in internal code repositories, check information traceability within business documents, or evaluate recovery capabilities after tool-calling failures. In these scenarios, domain-specific tests beyond general benchmarks often provide better insights into whether deployment is worthwhile.

Regarding developer adoption, according to OpenRouter data visible as of October 15, 2026, Step 5 Preview ranks second globally on the platform’s weekly and daily token usage charts. The results below summarize Step 5 Preview’s performance in coding, agentic tasks, specialized knowledge work, and finance.