Zum Inhalt springen

By

The Paper-Note Network · Part 1 of 4

Weight-space programming:A model whose numbers you write yourself

November 202610 min read

9 numbers become “AI”

The data

At least one is on
110
Both are on
11−1
One, but not both
1−20
Truth table: for each combination of A and B, the network's result and the expected result.
ABNetTargetCorrect
0000correct
0111correct
1011correct
1100correct
Nine numbers, three lines, and the task “exactly one of the two” is solved. Nobody trained this network.

Together with a simple blueprint, those nine numbers are a complete neural network. This neural network was never trained. It was built. That was always possible. What’s new since November 2, 2026: a team at MIT showed that a language model can read these numbers, repair them and, in simple cases, write them itself. There’s now a name for that: weight-space programming.

Here’s the good news in that: a small model doesn’t have to be a data project. If you lack the collected, labeled examples that training needs, something useful can still come out of it. And if you haven’t heard of any of this yet: my impression is that the topic is still confined to a small corner of the tech world. You’re early.

The promise

0.016 percent of a small language model's numbers were changed. Its score on a math test rose from 7.9 to 48.1 percent. No training.

The reality

The model got stuck in endless loops, and the change fixed that one bug. On a second model without the bug, the same approach gained nothing reliable.

Where no model was possible before, a usable one can now exist, even if it tends to be weaker than a trained one.

Illustration: a smiling stick figure with glasses holds up a note with a small sketched network of connected dots.

01

What is a neural network if you only look at the numbers?

A recipe made of numbers that turns inputs into outputs. For a small network, it fits on a paper note.

“Neural network” sounds like something you only get with a lot of computing power and a team of data scientists. Look inside a small one and you find numbers. In a big one too, by the way, just so many that you can’t see the forest for the trees. Most of them say how much an input counts. The field calls them weights. The rest shift the point at which something fires.

Take two switches, A and B. The task: output 1 if exactly one of them is on. Computer science, among other fields, calls this XOR, and it’s the hello-world of neural networks, because a single neuron can’t do it. You need a middle layer with two neurons.

The first neuron counts how many switches are on. The second only fires when both are on. The output takes the first and subtracts the second twice. What’s left: one, but not both.

Nine numbers, three labels, one truth table.

Every number has a reason

That’s three rows of three numbers, and for each one you can say what it’s for. The −1 in the second neuron makes sure a single switch isn’t enough. The −2 in the output takes the “both” case back out.

Try it yourself. In the network below, two numbers are still wrong, and the table recalculates with every click.

Try it yourself

Set the nine numbers so that “exactly one of the two” comes out right

A and B are two switches, 1 means on. The network should output 1 when exactly one switch is on. Three rows are already right. The row “both on” is still wrong. Hint: neuron 2 should only fire when both really are on.

Neuron 1 · At least one is on

times A

1

times B

1

base value

0
Neuron 2 · Both are on

times A

1

times B

1

base value

0
Output · One, but not both

times neuron 1

1

times neuron 2

0

base value

0
Truth table: for each combination of A and B, the network's result and the expected result.
ABN1N2NetTargetCorrect
000000correct
011111correct
101111correct
112210wrong

3 of 4 rows are right.

N1 and N2 show what the two neurons compute. A neuron never outputs less than 0. The network says 1 as soon as the output is above 0.5.

02

Where do these numbers usually come from?

From training with lots of labeled examples.

With nine numbers you can do it by hand. A language model has billions of them, and even a small network that sorts incoming requests quickly reaches a few thousand. Nobody writes those down. You let them be found.

Training means this: you show the network a very large number of examples where the right answer is already attached. A procedure nudges the numbers in small steps until the answers fit. For tasks like sorting, that’s the standard approach. Large language models also learn from huge amounts of unlabeled text. I wrote about how systems keep learning beyond that in “How AI agents learn”.

Where it gets stuck for small teams

Picture a small business, made up for this example: five people, requests arrive by email every day, and someone sorts them by hand into urgent, normal and advertising. A small model could take over the sorting, and that person would have their head free for the requests themselves. For training, though, you’d first need a few thousand emails where someone has noted the right drawer. Plus time, and someone who knows how training works.

The problem: a project like this can end before it starts. It fails on the pile of labeled examples that doesn’t exist.

So it stays manual, becomes a fixed rule in the inbox, or goes to a large AI service that reads every email. This is where what happened in November gets interesting.

Before the model comes the mountain of data.
Jörg Amelunxen, personal experience

03

What happened in November 2026?

On November 2, a team at MIT published early experiments in which a language model reads, repairs and writes the numbers of other networks. Without training.

The post is called “Sparks of Weight-Space Programming” and comes from four people at MIT CSAIL (Source: Takehi et al., MIT CSAIL, 2026). The idea in one sentence: a general-purpose language model reads and writes the weights of another network directly. It gets a task in plain language plus the numbers. No training happens.

Three of the experiments show what this is about.

Reading. The team trained small networks on four simple video games, each with 17,253 weights. A frontier model got the numbers and had to say which game each network had learned. According to the authors, it named the right game.

Repairing. A network with 171,664 weights drove a simulated car and got stuck at a ramp, in all 256 runs. The language model got the numbers, a description of the scene and one recorded run. It changed 3.5 percent of the weights. After that, the car reached the goal in all 1,024 test runs.

Changing a language model. The experiment that drew the most attention was on a small open language model with 0.8 billion weights. 131,072 of them were changed, which is 0.016 percent. Its score on the MATH-500 math test rose from 7.9 to 48.1 percent.

Worth keeping in mind: this is a blog post, and the authors themselves call the results very early (Source: Takehi et al., MIT CSAIL, 2026). None of it is peer-reviewed, and everything was shown with a single frontier model (Source: DataCamp, 2026).

Genuinely exciting, but in a small circle

Within nine days there were news pieces, two detailed critiques, first preprints and a Wikipedia article of its own (Source: Wikipedia, as of November 7, 2026). So the attention is there. My impression: it sits in a small part of the tech world, and outside of it the topic hasn’t landed yet.

The first nine days
  1. Nov 2

    Blog post by an MIT team coins the term

  2. Nov 3

    First coverage, two detailed critiques, a first preprint

  3. Nov 5

    Counter-study asks: does the model read, or remember?

  4. Nov 7

    The term has its own Wikipedia entry

  5. Nov 8

    Another preprint: formally checking written networks

Not a single entry is peer reviewed. Blog posts and preprints are all there is so far.

Sources: Takehi et al. (MIT CSAIL), Honnibal, Goedecke, arXiv, Wikipedia. All November 2026.

04

Same name, different things: what doesn't belong here?

The term means a language model that reads and writes weights. Older research with a similar name and related methods are neighbours.

The term is new. The phrase “weight space” isn’t. Search for it and you quickly land on things that sound alike.

Networks that read networks. For years there has been research in which a specially trained network takes the weights of other networks as input, for example to predict how accurate they are. There, a purpose-built tool does the work. What’s new is a general-purpose language model taking that job on.

Programs that become weights. There are compilers that convert a readable program into weights by fixed rules (Source: Lindner et al., Tracr, 2023). That’s close. The compiler follows a fixed procedure, though, and decides nothing on its own.

Add-ons at the push of a button. Other methods generate a small add-on for a language model in a single pass, from a document for example. Here, too, something comes into being without classic training. It’s produced by a helper network that was trained for exactly that.

That leaves three kinds: reading, repairing, writing from scratch. All three face the same question as the example above: can someone say what a number inside the neural network is for?

Three things a language model can do with the numbers
  1. 01

    Read

    Tell from the numbers what the network does.

    17,253 numbers, four possible games: which one did this network learn?

  2. 02

    Repair

    Change one wrong behaviour on purpose.

    3.5% of the numbers changed: 0 of 256 runs before, 1,024 of 1,024 after.

  3. 03

    Write from scratch

    Set up a network without training.

    Small networks for simple games. Mixed results against trained ones.

Source: Takehi et al., MIT CSAIL, Nov 2, 2026. Self-reported by the authors.

05

How is it different from prompting, RAG and training?

Prompting steers a finished model, RAG adds knowledge from outside, training lets the numbers be found. Weight-space programming writes them directly.

The easiest way to sort the approaches is one question: where does what the system can do end up living?

With prompting, it lives in the conversation. You tell the model what to do with every request. The model itself stays as it is.

With RAG, it lives outside. Before each answer, matching passages from your documents get added. What that looks like for company knowledge is in the article on RAG and company knowledge. The model stays unchanged here too.

With training, it lives in the weights. A procedure found them, based on your examples.

With weight-space programming, it also lives in the weights. But someone wrote them on purpose, based on a rule. In the example above, that was you. In weight-space programming, it’s a language model. Tests show whether the numbers are right.

And then there’s the option that’s easy to forget: plain code. If you know the rule and can write it down in two sentences, a small program tends to be the simplest solution. “Exactly one of the two” is one line in any programming language. The example network above is an experiment to demonstrate the concept. Nobody should evaluate switches in code this way from now on.

It gets interesting in between: with tasks whose rule can be described briefly but that nobody has spelled out yet. I’ll come back to that in the next parts.

Where does the behaviour live, and what do you need for it?
Prompting
Lives in the conversation. Needs a text.
RAG
Lives outside, in your documents. Needs documents and a search.
Training
Lives in the numbers, found by computation. Needs labelled examples, compute time.
Weight-space programming
Lives in the numbers, written directly. Needs a clearly described rule, a small test set, access to the numbers.
Plain code
Lives in the source code. Needs a rule you already know.

A simplified map. The last column is the one that matters for your decision.

Sort the approaches

Tap a card, then the approach it belongs to. The guiding question: where does what the system can do live in the end?

Cards · 6

Approaches

0 of 6 sorted.

06

What has been shown, and what is only claimed so far?

It has been shown to work on small examples. How far it goes is open, and the criticism deserves a hearing.

The 0.016 percent figure is the one being passed around. It’s correct, but it comes with a lot of fine print.

The small language model had a specific bug: on long tasks it got stuck repeating itself. The change switched off those loops, based on 50 test runs on practice problems. On a second small model that didn’t have this bug, the score went from 6.8 to 7.7 percent at best, and the report doesn’t count that as an established gain (Source: Takehi et al., MIT CSAIL, 2026).

A developer at Hugging Face rebuilt the changed model and measured 51.0 percent on the same test (Source: DataCamp, 2026). So far it’s the only re-measurement I know of.

Two critiques from November 3 are worth reading. Matthew Honnibal points out that the post bundles different things under one headline. Reading the raw numbers only happens in three of the experiments, on networks with 17,000 to 172,000 weights. For the language models, the frontier model worked with programs it wrote itself. He also says that figuring out how to do this productively is not nothing (Source: Honnibal, 2026).

Sean Goedecke finds the achievement remarkable and at the same time within reach: an experienced specialist would have found the loops with known methods too. He compares editing weights directly to patching a compiled binary. His conclusion: the best way to change a model’s weights is still normal training (Source: Goedecke, 2026).

My own take

I’ve had small networks written this way myself. The tendency I see: these networks are weaker than a properly trained one would probably be. In most cases, though, I haven’t run a comparison on the same data.

The question that keeps me busy is a different one: what would be there without this model? In the email example, a trained network was never on the table, because the examples for it didn’t exist. The choice was: manual work, a fixed rule, a large AI service that reads every email, or a small model that someone wrote, that runs in-house and can be checked. If the last one is usable, something has shifted, even if it would lose to a trained one.

One small language model, the MATH-500 maths test

In the model that could be repaired, the weakness came from endless loops. In a second model without that fault, the same approach did not help.

Sources: Takehi et al., MIT CSAIL, Nov 2, 2026. Re-measurement as reported by DataCamp, Nov 3, 2026.

07

What does this mean for you today?

Nothing to deploy yet. What you can take away already: a small model can increasingly become more of a software project than a data project, and a network can be readable.

In conclusion, first: in the end, a model is a table of numbers, and in small networks each of them can have a reason. You just set nine of them yourself. Second:

The effort moves: away from collecting and labeling examples, toward a clearly described rule and twenty test cases.

You can start on that second point today, without any technology. Pick a decision that someone in your business makes by hand every day. Write down in two sentences what that person bases it on. Put twenty real cases next to it where you know the right answer.

If you manage that, you’ve done the groundwork for every approach in the overview above, from a small program to a model. If you can’t yet, you know where the real work is. That’s exactly where I like to start: turning decisions like these into rules and test cases, and from there into the approach that fits your business.

In the next part, the network gets bigger: an adder with a single wrong number in it. Can you track it down like a bug in code?

Can you state the rule in two sentences? That’s the new question for your next project.
Jörg Amelunxen

Related to This Topic

Get the free Getting Started Guide: 10 concrete ways to start using AI productively tomorrow.

Did this article spark an idea? Let's find out which Sinnvampire can disappear for you.

New articles straight to your inbox

No spam, no sales funnels. Just one email when there's a new article on AI for small and mid-sized businesses. You confirm with a single click and can unsubscribe any time.