When NOT to add “think step by step”
For years, asking an AI to think step by step was the safe default. For today's reasoning models it often adds length without adding quality — and sometimes it makes the answer worse. The trick is knowing which kind of model you are talking to.
What changed
Chain-of-thought prompting was built for models that answered in one shot: spelling out the steps helped them stay on track. Newer reasoning models already reason internally before they answer. When you also hand them a step-by-step procedure, they may repeat that internal reasoning in the visible answer (more output, same quality), or squeeze their own exploration into the procedure you wrote. Either way you pay for words that did not help.
The practical advice from current practitioner guidance is to control reasoning depth through the model's own settings where they exist — reasoning-effort or thinking-budget options — instead of through wording in the prompt.
What the evidence says
One relevant study is the Wharton Generative AI Labs report by Lennart Meincke, Ethan Mollick, Lilach Mollick and Dan Shapiro, titled “The Decreasing Value of Chain of Thought in Prompting”. Its test set was GPQA Diamond: 198 PhD-level multiple-choice questions in biology, physics and chemistry, each run 25 times under three conditions — answer directly, no special instruction, and “think step by step”. The numbers below describe that test, not a promise for every task.
For the non-reasoning models, step-by-step prompting lifted average scores, with the biggest rises for Gemini Flash 2.0 (13.5%) and Sonnet 3.5 (11.7%); GPT-4o-mini's 4.4% rise was not statistically significant. But the strictest yardstick told a different story: on getting every trial right, results were mixed, and Gemini Pro 1.5 dropped 17.2%. The researchers' reading is that extra reasoning can help with hard questions while also adding variation that trips the model on easy ones. Step-by-step requests also took far longer than direct requests on these models — 35–600% more time (5–15 seconds in the study's runs).
For the reasoning models tested (o3-mini, o4-mini, Gemini Flash 2.5), the average-score changes were small: o3-mini up 2.9%, o4-mini up 3.1%, Gemini Flash 2.5 down 3.3%, while step-by-step requests needed 20–80% more time (10–20 seconds). The report's conclusion is narrow and worth repeating: whether step-by-step helps depends on the model, the task, and what trade-offs you accept between accuracy, consistency and response time.
One more scope note: the study found many models already reason by default when given no special instruction, so “no instruction” and “step by step” are closer than their names suggest. Treat the checklist below as starting heuristics to test on your own task, not universal rules.
Decision checklist
- Is it a reasoning model? Start by dropping the step-by-step nudge. Say what a good answer looks like — format, audience, constraints — and let the model reason on its own. Use the effort/thinking setting if you want more or less reasoning, and compare against the version with the nudge before deciding.
- Is it a non-reasoning model and the task needs several steps? Try one short reasoning instruction first, for example asking for the working before the final answer, and check whether it actually improves your task before adding anything else.
- Are you stacking tricks? Avoid stacking chain-of-thought plus few-shot plus self-consistency plus least-to-most as a default. Each added scaffold is another chance to steer the model onto your solution path instead of its own; if you combine techniques, test the combination against the simpler version.
- Do you need the reasoning visible? If the trail itself is the deliverable (an audit, a worked solution), ask for it explicitly and accept the extra length. Remember that visible steps are a checkable explanation, not a guaranteed record of the model's actual internal process — for an audit, verify the calculations, evidence and conclusions independently. If only the answer matters, do not ask for the trail.
- Is the task format, not reasoning? Examples still help for format-following and classification. Keep them for format, and be cautious about using them to teach the model how to think.
What to do instead
Replace the reasoning nudge with the things that still move quality: one task, one audience, one desired outcome; a real example or constraint; and a clear description of the final answer. Then check the output against that description. If the answer is wrong, change one thing — the model, the effort setting, or one instruction — and run it again, rather than piling on more scaffolding.
This is the same rule as this site's overprompting advice: cut what does not earn its place. A step-by-step nudge is most likely to earn its place on a non-reasoning model doing multi-step work. On a reasoning model, it usually does not — but test that on your own task rather than taking it on faith.
Templates on this site where this applies
- Code Explainer — ask for the explanation you want, not for a step-by-step method the model must follow.
- Debugging Helper — give the error, the code and what you already tried; try one short reasoning request on a non-reasoning model, and try none first on a reasoning model — then compare the two.
- Explain a Concept Simply — the template already asks for analogy, core idea and example; adding a step-by-step scaffold on top usually adds words, not clarity.
Sources
- The Wharton Generative AI Labs report itself (Meincke, Mollick, Mollick and Shapiro), “The Decreasing Value of Chain of Thought in Prompting”: Wharton report summary and methodology. All figures on this page come from this report; check it for full method and context.
- Coverage of that report on how chain-of-thought gains are fading on reasoning models: Prompt Engineering Techniques Shift as Models Absorb CoT.
Model behaviour changes as models and vendor guidance change. If your tool's own documentation says a reasoning instruction is unnecessary or harmful for that model, follow the documentation over any general rule on this page.