Random numbers, predictable models and skills with scripts

Language models are good at giving answers that look correct. That doesn't necessarily mean they follow the right procedure.

If you ask an AI agent to proofread a text, good instructions may be sufficient. But if you ask it to draw a random number, calculate something exactly, or validate data according to fixed rules, it's not always enough for the model to "think its way" to the answer.

In such cases, a skill can give the agent specialized knowledge about a specific task, or alternatively a script it should run instead of guessing.

The simple approach: instructions only

All skills consist of a folder that at minimum contains a SKILL.md file. At the top is some metadata, often called frontmatter. In its simplest form it contains the name field, which is the name you use to call the skill manually, and a description that tells the AI agent when it should possibly call the skill automatically.

Here's an example of a minimalist skill that can be used to proofread texts and provide feedback in the desired format.

---
name: proofread
description: Proofreads text provided by the user and gives feedback on spelling errors, compound words, grammar, punctuation, flow, and factual errors. Triggered when the user asks for proofreading.
---

Proofread the text the user provides. Preserve the user's own writing style. You should correct errors and suggest improvements, not rewrite the text in your own style.

Present the feedback in this order:
1. Spelling and capitalization errors
2. Grammar and punctuation
3. Sentences that should be rewritten
4. Contradictions and factual errors 

When you start the agent, only the metadata is loaded into the context. If the agent perceives that the user is asking for proofreading, the rest of the SKILL.md file will be loaded into the context. This way you avoid paying for the entire skill to be loaded into the context every time the conversation isn't about proofreading.

It's also possible to use SKILL.md as a "table of contents" with pointers to several specialized Markdown files that the agent can load as needed. In the example above, we could have had a separate .md file for proofreading LinkedIn posts and one for emails.

It's not always that you can get the AI agent to do as you want, no matter how precise your description is. In some cases, code can make it easier.

Random numbers

If someone asks you to pick a random number between 1 and 10, which number would you pick? It turns out that the majority will choose 7.

Claude will also prefer the number 7. I tested 100 independent conversations with each of Claude's four model families. All attempts are real claude -p calls where the model must answer from its own knowledge, without access to tools or scripts. Details of the setup are at the bottom of the post.

All models had the same favorite number, the number 7. All models chose the number 7 in at least 86% of cases.

I also did a similar test where I asked for a number between 1 and 100. In 100 attempts, only two of the models gave me a single number lower than 42.

The same four numbers stand out with all models: 42, 47, 57 and 73. Almost everything they draw lands on one of these. But they don't agree on which one. Each model has its own favorite.

When it comes to random numbers, it's also not the case that a more powerful model gives you more random numbers. The most affordable model, Haiku, was actually the one that performed best. Haiku distributes the main weight across three different numbers (42, 47, 73), Sonnet has 47 as a clear favorite, while Opus has set its heart on 73. Fable, Anthropic's most powerful model to date, is divided between 42 and 57.

Why exactly these numbers? Because a language model is trained on text written by humans, and humans are very bad at acting randomly. When asked for a number, we find that round numbers like 10 and 50 "don't count", while odd, often unusual numbers like 7, 47 and 73 feel more random.

42 also has its own special weight, as the answer to The Ultimate Question about Life, the Universe, and Everything from The Hitchhiker's Guide to the Galaxy. So the model isn't drawing a random number, it's reproducing how humans answer when they try to be random. And that's exactly the point. It has learned what a random answer looks like, not how to actually draw one.

The fact that computers have challenges with random numbers is of course nothing new, but we can help the AI agent do it a bit more randomly. Cloudflare is known for having a wall of lava lamps as one of their sources for creating random numbers. I choose a slightly simpler variant: Python's secrets module, which gets its randomness from the operating system.

Scripts: When instructions aren't enough

A skill doesn't just have to contain instructions, it can also contain scripts. A script follows the same procedure every time it's run. That's exactly what we need here, predictable behavior that produces an actually randomly drawn number.

Here's an example of a skill that uses a script to give the user a random number.

---
name: random-number
description: Picks a random integer using a real random number generator instead of guessing one. Use whenever the user wants to generate a random number.
---

# Pick a random number

Do not invent a random-looking number yourself. Always call the bundled script instead.

Run `scripts/random_number.py` with 2 arguments:

```bash
python scripts/random_number.py 1 100    # 1-100 inclusive
```

Instead of the language model itself trying to find a random number, it will now run the Python script which uses the built-in secrets module to draw the number. It's the standard tool for cryptographically secure random numbers.

The screenshot below shows the distributions again, now with an extra row at the bottom where Haiku has the skill turned on.

Often instructions alone in a skill are sufficient, but a script usually pays off when the task has a fixed, correct procedure that the model is really just guessing its way through.

If you need to calculate something exactly, like dates across time zones or interest rates, a script is safer than a model that "calculates in its head". If you need real cryptography, like secure passwords, tokens or a proper UUID, there needs to be code that draws the values so they're actually random. The same applies when you need to validate structured data according to fixed rules.

Often, language models will, on their own initiative, write code to solve such tasks. But if you want full control, you should use a skill with a script you can quality-assure.

The complete random-number skill with script can be found here.