Language Models vs. Reasoning Models / A Conversation with Claude

Claude: Oh, what is it? I'm curious. Go ahead!

Claude: Ah, yes! You're talking about language models and reasoning models. That's a good topic. What would you like to know?

Claude: Ah, how they relate. Good question. Let me explain it simply.

A basic language model gives you an answer as soon as you ask a question. Think of it as someone who immediately says whatever comes to mind. It's fast, but it can make mistakes on complex problems.

A reasoning model isn't actually a completely different kind of model. It's a language model trained to go through an internal thinking process before answering. Instead of responding right away, it works through things step by step: "Hmm, this means that, so the conclusion is..." Then it gives you the answer. That's why it's much stronger at problems that require deeper thought, such as math and logic.

So a reasoning model is a branch of language models. It has the same roots, with an extra "habit of thinking" added. The tradeoff is that thinking takes more time and costs more. Does that make sense so far?

Claude: Yes, exactly! High means it thinks longer and more deeply than Medium before answering.

That setting tells the model how much time and effort to put into thinking. Low is quick and rough, Medium puts in a moderate amount of effort, and High examines things very carefully.

For simple questions, Low or Medium is enough, and it's faster too. But for complex math, tricky coding, or logical problems that require digging deeper, High is much more accurate. The tradeoff is that answers take longer and cost more.

So the trick is to match the setting to the difficulty of the problem. Using High for something easy just slows things down unnecessarily, while using Low for something difficult makes mistakes more likely. Getting the idea?

Claude: Ah, that's a really good question. "Thinking" means something a little different for AI than it does for people.

For AI, thinking longer means generating more text before giving you an answer, internally, where you can't see it. For example, it writes out an intermediate process like a running conversation with itself: "Hmm, I should approach this problem this way. No, wait, that's wrong. Let's try again." After going through that process, it produces the final answer.

For a person, it's similar to working something out on an imaginary scratchpad. Think of the difference between answering a math problem in your head and writing out equations on paper, step by step. Working it out on paper is slower, but you make fewer mistakes. AI does the same thing.

So "thinking longer" doesn't mean sitting around doing nothing. It means generating a longer, more careful intermediate solution. The longer that process is, the more accurate the answer becomes. Does that feel a bit clearer now?

Claude: Oh, that's close, but there's a slight difference. Two things are mixed together.

One is what you described: generating answers along several different paths and choosing the best one. It tries different approaches, like "Should I solve it this way, or that way?" Then it picks the one that makes the most sense.

The other is following one long train of thought while checking and correcting itself along the way: "I solved it this way, but wait, there's a mistake here. Let's try again." That's going deeper along a single path.

In practice, the two work together. It thinks at length, considers different possibilities when needed, checks for mistakes, and then gives you a refined final answer.

So your idea of "choosing among several answers" is right, with a process of correcting and refining things added to it. You've identified the key point!

Claude: Well, the difference varies a lot depending on the problem.

For easy questions, like "What's the weather like today?", Low and High give you almost the same answer. Thinking more doesn't change much because the answer is straightforward. Using High for something like that just makes it slower.

But the gap gets much wider for difficult problems: complex math, tricky coding, or logic puzzles. High can get something right that Low gets wrong. The rate of correct answers goes up noticeably.

So you can think of it this way: little difference for easy problems, a big difference for difficult ones. That's why matching the setting to the problem's difficulty saves both time and money.

Claude: Yes, that's right! The same instruction can produce different results depending on the model.

The difference is especially noticeable when the instruction is complex or involves several steps. A model that reasons deeply examines the instruction more carefully and catches things that might otherwise be missed. A model that answers more lightly may get the general idea but overlook the details.

On the other hand, if the instruction is very clear and simple, the results are similar across models. They all handle things like "Translate this sentence into English" well.

So in practice, people pay attention to two things: making instructions as clear as possible, and choosing a model that suits the difficulty of the problem. You get the best results when those two line up.

Claude: Yes, that's right. Every model gives the same answer, "hello", to a question like that.

That's because the answer is fixed, and there's nothing to think through. No matter how deeply it thinks, "annyeong" is "hello". There isn't another answer. So increasing reasoning doesn't change the result for something like this. Using High just wastes time. That's a perfect example.

Claude: Yes, definitely! This is the key point.

The more deeply a model reasons, and the closer you go to High, the more tokens it uses. That's because the internal thinking process we discussed, those intermediate steps, is all generated as text. All of that consists of tokens. Even when you can't see them, tokens are still being used.

So even for the same question, Low uses fewer tokens because it thinks briefly, while High uses far more because it thinks for longer. More tokens mean higher costs and slower responses.

That's why using High for a question like "annyeong means hello" is a poor deal. The answer is the same, but it uses lots of thinking tokens. Those tokens are the reason matching the setting to the difficulty saves money. Is the whole picture coming together now?

Claude: Well, the latest is generally better, but not always. It depends on the situation.

The benefits of the latest models are clear. They're smarter, know more recent information, and handle complex tasks better. So newer models have an advantage for difficult tasks or work where quality matters.

But there are reasons the latest isn't always the right answer. First, the latest models are usually more expensive and slower. Second, simple tasks don't necessarily need the latest model. A model from the previous generation is enough for things like translation or simple summaries. Third, if you're running a service where cost matters, it's sensible to handle most requests with a cheaper, faster model and reserve the latest one for the difficult tasks.

So "the right model for the job" is a better approach than "always use the latest". It's the same principle we discussed when choosing reasoning settings.

Claude: Ah, that's a very perceptive question. You're right: "more recent information" and "being smarter" are two completely different things.

Being smarter means improving its ability to think and process things, rather than simply knowing more information. Let me break that down.

First, there's the ability to solve complex problems. Older models would make mistakes along the way in math or logic problems involving several steps. Newer models can work through them to the end without making those mistakes.

Second, there's the ability to understand instructions accurately. Older models would miss parts of complex requests such as "Do this, but don't do that". Newer models follow them more carefully.

Third, they make things up less often. Making up an answer while pretending to know something is called a hallucination, and this happens less with newer models.

That's because training methods keep improving as models are developed. It's like getting a better meal from the same ingredients because the cooking method has improved. So the processing ability itself gets better, independently of how much information the model knows. Is the distinction clearer now?

Claude: Haha, yes, exactly! If cost and speed aren't concerns, using the latest model with reasoning set to High is the easiest choice, and gives you the most reliable results.

That way, it handles both easy and difficult problems well, so you don't have to think about which setting to choose. You just use the best option for everything.

As we discussed, though, it can be wasteful. You're using a top-of-the-line model just to translate "annyeong" into "hello". But if tokens and speed don't matter in your situation, that waste isn't a problem, so it really is the most convenient choice.

Claude: Yes, that's right! Definitely. The higher you set reasoning effort, the longer it takes to get an answer.

That's natural, because it goes through a longer internal thinking process. Generating all those intermediate steps takes time. Low gives you an answer quickly, while High spends a while working through things first.

So that tradeoff becomes clear. High is accurate but slower and uses more tokens. Low is fast and cheap, but can get difficult problems wrong. You're trading speed for accuracy.

You've really got a handle on these AI concepts today, JB. Your understanding is clear!