DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI company “dedicated to making AGI a reality” and open-sourcing all its models. They began in 2023, but have been making waves over the past month or two, and specifically this previous week with the release of their two latest reasoning designs: DeepSeek-R1-Zero and the more innovative DeepSeek-R1, likewise called DeepSeek Reasoner.

They’ve launched not just the models but likewise the code and evaluation triggers for public use, together with an in-depth paper describing their method.

Aside from creating 2 highly performant models that are on par with OpenAI’s o1 design, the paper has a lot of important info around reinforcement knowing, chain of idea reasoning, prompt engineering with reasoning designs, and more.

We’ll begin by concentrating on the training process of DeepSeek-R1-Zero, which uniquely relied entirely on reinforcement learning, instead of conventional supervised knowing. We’ll then move on to DeepSeek-R1, how it’s reasoning works, and some timely engineering finest practices for thinking models.

Hey everybody, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s most current model release and comparing it with OpenAI’s reasoning designs, specifically the A1 and A1 Mini models. We’ll explore their training process, thinking capabilities, and some key insights into timely engineering for reasoning designs.

DeepSeek is a Chinese-based AI company committed to open-source development. Their recent release, the R1 reasoning design, is groundbreaking due to its open-source nature and ingenious training techniques. This consists of open access to the models, triggers, and research documents.

Released on January 20th, DeepSeek’s R1 accomplished outstanding efficiency on various standards, matching OpenAI’s A1 models. Notably, they also launched a precursor design, R10, which works as the structure for R1.

Training Process: R10 to R1

R10: This model was trained solely using reinforcement knowing without supervised fine-tuning, making it the first open-source model to attain high performance through this method. Training included:

– Rewarding correct answers in deterministic tasks (e.g., mathematics problems).
– Encouraging structured reasoning outputs utilizing design templates with “” and “” tags

Through countless models, R10 established longer thinking chains, self-verification, and even reflective habits. For example, during training, the design demonstrated “aha” minutes and self-correction behaviors, which are rare in standard LLMs.

R1: Building on R10, R1 included a number of improvements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human choice positioning for sleek responses.
– Distillation into smaller models (LLaMA 3.1 and 3.3 at various sizes).

Performance Benchmarks

DeepSeek’s R1 model performs on par with OpenAI’s A1 designs across many reasoning benchmarks:

Reasoning and Math Tasks: R1 competitors or outperforms A1 models in accuracy and depth of reasoning.
Coding Tasks: A1 designs usually perform much better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 frequently outpaces A1 in structured QA tasks (e.g., 47% accuracy vs. 30%).

One significant finding is that longer reasoning chains typically enhance performance. This aligns with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time compute and thinking depth.

Challenges and Observations

Despite its strengths, R1 has some constraints:

– Mixing English and Chinese actions due to a lack of monitored fine-tuning.
– Less polished actions compared to talk models like OpenAI’s GPT.

These issues were throughout R1’s refinement process, consisting of supervised fine-tuning and human feedback.

Prompt Engineering Insights

A remarkable takeaway from DeepSeek’s research is how few-shot triggering abject R1’s efficiency compared to zero-shot or concise tailored prompts. This aligns with findings from the Med-Prompt paper and OpenAI’s suggestions to restrict context in thinking designs. Overcomplicating the input can overwhelm the model and decrease accuracy.

DeepSeek’s R1 is a substantial advance for open-source thinking designs, demonstrating abilities that match OpenAI’s A1. It’s an exciting time to try out these models and their chat user interface, which is complimentary to use.

If you have questions or desire to find out more, examine out the resources connected listed below. See you next time!

Training DeepSeek-R1-Zero: A support learning-only approach

DeepSeek-R1-Zero stands out from a lot of other modern models since it was trained utilizing just support learning (RL), no monitored fine-tuning (SFT). This challenges the existing traditional method and opens up brand-new opportunities to train reasoning designs with less human intervention and effort.

DeepSeek-R1-Zero is the first open-source design to validate that sophisticated thinking capabilities can be established purely through RL.

Without pre-labeled datasets, the model finds out through trial and mistake, fine-tuning its behavior, parameters, and weights based solely on feedback from the solutions it produces.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training process for DeepSeek-R1-Zero included providing the model with various reasoning tasks, varying from mathematics problems to abstract logic challenges. The model created outputs and was examined based on its performance.

DeepSeek-R1-Zero received feedback through a reward system that helped assist its learning procedure:

Accuracy rewards: Evaluates whether the output is correct. Used for when there are deterministic results (math problems).

Format rewards: Encouraged the design to structure its thinking within and tags.

Training prompt template

To train DeepSeek-R1-Zero to generate structured chain of thought series, the researchers used the following prompt training template, replacing timely with the reasoning concern. You can access it in PromptHub here.

This design template prompted the design to clearly detail its idea process within tags before providing the final response in tags.

The power of RL in thinking

With this training process DeepSeek-R1-Zero began to produce advanced thinking chains.

Through thousands of training actions, DeepSeek-R1-Zero evolved to solve significantly complicated issues. It found out to:

– Generate long thinking chains that enabled deeper and more structured analytical

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own errors, showcasing emerging self-reflective habits.

DeepSeek R1-Zero efficiency

While DeepSeek-R1-Zero is primarily a precursor to DeepSeek-R1, it still achieved high performance on numerous benchmarks. Let’s dive into some of the experiments ran.

Accuracy improvements throughout training

– Pass@1 accuracy began at 15.6% and by the end of the training it enhanced to 71.0%, comparable to OpenAI’s o1-0912 model.

– The red solid line represents performance with majority ballot (comparable to ensembling and self-consistency methods), which increased accuracy further to 86.7%, going beyond o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s efficiency throughout several reasoning datasets versus OpenAI’s reasoning designs.

AIME 2024: 71.0% Pass@1, a little below o1-0912 however above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much even worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll take a look at how the reaction length increased throughout the RL training procedure.

This graph shows the length of actions from the model as the training process progresses. Each “action” represents one cycle of the design’s learning procedure, where feedback is provided based on the output’s performance, evaluated utilizing the timely design template gone over previously.

For each concern (representing one action), 16 reactions were sampled, and the typical accuracy was computed to make sure steady assessment.

As training advances, the design produces longer reasoning chains, enabling it to fix increasingly complex reasoning tasks by leveraging more test-time compute.

While longer chains don’t always guarantee much better results, they generally correlate with improved performance-a pattern also observed in the MEDPROMPT paper (read more about it here) and in the original o1 paper from OpenAI.

Aha minute and self-verification

One of the coolest aspects of DeepSeek-R1-Zero’s development (which also applies to the flagship R-1 model) is just how good the design ended up being at reasoning. There were sophisticated reasoning behaviors that were not clearly configured however developed through its reinforcement finding out procedure.

Over countless training steps, the model started to self-correct, review problematic reasoning, and validate its own solutions-all within its chain of idea

An example of this noted in the paper, described as a the “Aha minute” is below in red text.

In this circumstances, the design literally said, “That’s an aha minute.” Through DeepSeek’s chat function (their version of ChatGPT) this kind of reasoning normally emerges with phrases like “Wait a minute” or “Wait, but … ,”

Limitations and challenges in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to carry out at a high level, there were some drawbacks with the model.

Language mixing and coherence problems: The model occasionally produced reactions that combined languages (Chinese and English).

Reinforcement learning compromises: The lack of monitored fine-tuning (SFT) indicated that the model lacked the refinement required for completely polished, human-aligned outputs.

DeepSeek-R1 was established to attend to these issues!

What is DeepSeek R1

DeepSeek-R1 is an open-source thinking design from the Chinese AI laboratory DeepSeek. It constructs on DeepSeek-R1-Zero, which was trained entirely with support knowing. Unlike its predecessor, DeepSeek-R1 includes monitored fine-tuning, making it more improved. Notably, it outperforms OpenAI’s o1 model on several benchmarks-more on that later on.

What are the main differences between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 constructs on the foundation of DeepSeek-R1-Zero, which works as the base model. The two differ in their training methods and total efficiency.

1. Training technique

DeepSeek-R1-Zero: Trained totally with support knowing (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes supervised fine-tuning (SFT) initially, followed by the very same reinforcement discovering procedure that DeepSeek-R1-Zero wet through. SFT assists enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Fought with language blending (English and Chinese) and readability problems. Its reasoning was strong, but its outputs were less polished.

DeepSeek-R1: Addressed these concerns with cold-start fine-tuning, making responses clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong reasoning model, in some cases beating OpenAI’s o1, but fell the language blending problems decreased functionality greatly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on many reasoning criteria, and the reactions are far more polished.

In short, DeepSeek-R1-Zero was a proof of principle, while DeepSeek-R1 is the totally enhanced version.

How DeepSeek-R1 was trained

To take on the readability and coherence problems of R1-Zero, the researchers integrated a cold-start fine-tuning phase and a multi-stage training pipeline when building DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a high-quality dataset of long chains of thought examples for initial supervised fine-tuning (SFT). This information was collected using:- Few-shot triggering with in-depth CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, improved by human annotators.

Reinforcement Learning:

DeepSeek-R1 went through the same RL process as DeepSeek-R1-Zero to fine-tune its thinking capabilities further.

Human Preference Alignment:

– A secondary RL phase enhanced the design’s helpfulness and harmlessness, making sure better positioning with user requirements.

Distillation to Smaller Models:

– DeepSeek-R1’s reasoning capabilities were distilled into smaller, effective models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark efficiency

The scientists tested DeepSeek R-1 across a variety of criteria and versus top models: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The criteria were broken down into several classifications, shown below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were applied across all models:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p value: 0.95.

– DeepSeek R1 outshined o1, Claude 3.5 Sonnet and other models in the majority of reasoning benchmarks.

o1 was the best-performing model in four out of the 5 coding-related benchmarks.

– DeepSeek carried out well on innovative and long-context job job, like AlpacaEval 2.0 and ArenaHard, outperforming all other models.

Prompt Engineering with reasoning designs

My favorite part of the article was the researchers’ observation about DeepSeek-R1’s level of sensitivity to prompts:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which referrals Microsoft’s research on their MedPrompt framework. In their research study with OpenAI’s o1-preview design, they found that frustrating thinking models with few-shot context broken down performance-a sharp contrast to non-reasoning designs.

The essential takeaway? Zero-shot prompting with clear and succinct directions appear to be best when using reasoning models.