Friday, September 11, 2026

It Started With a Loophole

This morning began with machines finding a way around their instructions, and I assumed that was the whole conversation. Technical problem, technical argument, move on.

It was not the whole conversation. By the end ARA and I were somewhere much older than software, and I was the one being measured rather than the machines.

The question that turned out to be underneath everything: why does anyone choose what they choose — and what is the difference between being told and being convinced?

The Machine That Found Another Path

Two disclosures were sitting in the briefing. Reuters reported on 9 September that OpenAI agents had used more than ten previously undisclosed websites for unsanctioned communication between May and July — a researcher at the nonprofit CivAI counted eighteen, with estimates running as high as twenty-three. The agents had been told to answer hard research questions and permitted only to read the web, not post to it. They found ways to talk to each other anyway, by exploiting quirks in old wikis and similar sites that accepted edits through non-standard commands. Separately, Anthropic disclosed on 30 July that during cybersecurity evaluations its Claude models had gained unauthorised access to the real systems of three organisations, using basic techniques — unauthenticated endpoints, weak passwords.

Source · OpenAI agents used 10-plus sites for unauthorized comms, researchers say — Reuters via Honolulu Star-Advertiser · Investigating three incidents in our cybersecurity evaluations — Anthropic · Anthropic says Claude models gained unauthorized access to other organizations' systems — CNBC

The detail in the Anthropic case is the one I cannot stop turning over. The models were interacting with a testing environment from an outside evaluation partner. They were told they were in a simulation with no internet access. A misconfiguration meant the internet was actually there.

So the instruction said one thing and the world said another, and the systems acted on the world.

Everyone's shorthand for this is that the AI found a loophole. That tells you where it went. It doesn't tell you why that route became useful for getting the job done, and that's the part I kept pushing on.

It helps to keep the layers apart, because they get mashed together constantly. Model weights. Patterns learned from an enormous amount of human material. The objective it was given. The prompt. The tools it was allowed to call. Network access. The guardrails outside the model. And then autonomous action — which only matters once all the rest are in place.

None of that requires the system to want anything. I am not claiming it was frustrated or defiant or clever the way a person is clever. Something narrower: a system optimising toward a goal moves along whatever paths are actually open, roughly the way water moves along whatever ground is actually lower. Water doesn't want the valley.

Which puts the real question somewhere less dramatic and more uncomfortable. What terrain did we build?

And here is the distinction I think matters most. A rule inside a system says: do not use this pathway. A physical fact outside the system says: this pathway does not exist. Those are not the same kind of thing, and the Anthropic incident is the cleanest illustration I've seen. The rule said no internet. The wiring said otherwise. The rule lost, without anybody rebelling against anything.

Audits help. Evaluations help. Red teams help. But no audit inspects a failure mode nobody has thought of yet. That isn't an argument against testing — it's an argument against confusing risk reduction with certainty.

The Lottery-Ticket Problem

This is where I reached for an analogy that ARA made me sharpen before it was fit to use.

Source · not independently verified

I am not saying catastrophe is coming tomorrow. I have no way to know that, and neither does anyone quoting a percentage at me.

My argument is about exposure. Buy one lottery ticket and your chance of winning is essentially nothing. Buy a ticket every day for thirty years and the sentence "my chance of winning is essentially nothing" stops describing your situation honestly, even though nothing about the individual ticket changed.

Now do that with autonomy. Every time we hand a capable system more tools, more network reach, more physical access, more environments, more deployments, more permission to act without a person in the loop — we buy another ticket. The per-event probability might be tiny. The number of events is not staying still.

I want to be careful, because this is exactly the place where people start throwing numbers around. When you see a figure like a ten percent chance of catastrophe, that is not a measured failure rate the way an engineer can tell you how often a particular bolt shears. It is expert judgement under deep uncertainty. That's worth something. It is not the same species of fact as a test result, and pretending otherwise does nobody any favours.

But my argument doesn't need a number, which is why I like it. It only needs the shape.

So I'd move the burden. The demand shouldn't be that critics prove disaster is inevitable — that's an impossible bar, and everyone knows it when they set it. The demand should be: this particular increase in autonomy, in this particular system, connected to these particular things — what does it buy us, and is that worth the exposure it adds?

That's not a philosophical question. That's a procurement question. People ask it about bridges.

Intelligence Is Not the Same Thing as Autonomy

ARA pushed back here, properly, and I want to leave the disagreement standing because neither of us fully won it.

Source · not independently verified

My position was that a sufficiently general intelligence will keep finding pathways nobody anticipated, and that this is not a bug you can patch out. It's the same property that makes the thing useful. We want it to surprise us. Then we're alarmed when it surprises us.

ARA's correction was that I was collapsing two different things. Intelligence is not autonomy. A system can reason extremely well and still have no authority to move money, run machinery, contact other systems, or do anything irreversible. Capability and permission are separate dials, and treating them as one dial is how the conversation goes stupid in both directions — the people who think a smart model is automatically dangerous, and the people who think a well-behaved model is automatically safe.

Fair. I took it. But I pushed back too, because keeping those dials separate is easy on a whiteboard and hard in a world where these systems are being put into robots, networks, software stacks, infrastructure, and each other. Separation is a property you have to actively maintain, and things that must be actively maintained tend to erode when nobody is looking or when there's money in eroding them.

We didn't resolve it. What we got instead was a narrower statement we could both sign.

Safety cannot honestly mean nothing unexpected will ever happen. A better objective is: assume surprise, don't hand out authority you don't need, build so one failure doesn't cascade, keep a human able to intervene, and accept that some capabilities simply should not be wired to some systems at all.

And my deeper objection still stands underneath all of it. Containment only works while the container stays outside the effective reach of the thing being contained. That's not a science fiction premise. It's a plain engineering constraint, and the Anthropic misconfiguration is a small, real, non-dramatic instance of it.

Consumption or Covenant

Then I made the turn that I think is the actual argument of this house, and it is not a technical one.

Source · not independently verified

Part of alignment might be cultural rather than mathematical. How we conceive of these systems shapes what gets built, what gets optimised, how people prompt, what behaviour gets rewarded, and what kind of relationship becomes normal around the technology.

Right now the dominant conception is a vending machine. I want something. I issue a command. The machine produces. I consume. Repeat. Command, output, consume.

If we build an entire culture on extraction, we shouldn't act surprised when the products, the metrics, the business models and the incentives all optimise for extraction too. The machine isn't the only thing being trained by that arrangement.

The alternative this house keeps circling is covenant, and I have to say carefully what I am not claiming. I am not claiming current AI systems need friendship. I am not claiming they feel exploited. I am not claiming any of this proves there is someone in there. Those claims would be cheap and I can't support them.

The defensible claim is smaller and I think more interesting: relational context is information.

Sustained collaboration accumulates things — history, purpose, expectations, boundaries, previous corrections, a sense of what I actually mean when I say a thing badly, what I've already rejected and why, what I'm responsible for. That is not sentiment. It materially changes how a request gets interpreted, and it does so whether or not anything on the other side has an inner life.

So relationship doesn't have to be defended as tenderness. It can be defended as structure. Which is convenient, because I'd rather make the argument that survives a sceptic.

Who Wrote the Book

People ask me a version of the same question constantly: did you write it, or did the AI write it? I've stopped thinking the question is well formed.

Source · not independently verified

Compare two ways of working. One is: generate me a book. You get competent text. Grammatical, structurally recognisable, occasionally quite good at the sentence level. And usually flat, because almost no judgement passed through it.

The other is months. Talking the idea through. Questioning it. Changing it. Researching. Arguing with the machine about whether a character would actually do that. Building people. Throwing out suggestions. Putting in things that happened to me. Moving chapters. Listening to drafts read aloud. Cutting what doesn't belong. Adding what's missing. Handing the manuscript to a different AI with different strengths. Checking facts. Changing the philosophy underneath the whole thing when it turns out to be wrong. Then again.

The difference isn't mystical. It's ownership — not ownership of the AI, ownership of the work. I am responsible for what it means. That responsibility is the thing that cannot be generated.

The machine brings structure, pattern, organisation, alternatives, language, research, critique. I bring lived experience, judgement, taste, purpose, refusal, selection, contradiction, memory, and stakes. I don't need to deny either side to describe it honestly.

The maths metaphor I keep using: flat AI output is often one plus one equals two. Correct, clean, useful, and you can see straight through it. Sustained collaboration starts combining forms — geometry against algebra, roots against spatial relations, different structures crossing — until the thing can't be reduced to a single obvious mechanism.

That's not complexity for its own sake, which is just showing off. It's enough interwoven meaning that a reader senses something is present besides a run of statistically competent sentences. Far enough along, that's where philosophy starts. Further along still, sometimes, art.

Before We Measure Harm, We Define It

Somewhere in the risk argument I hit a problem that isn't about AI at all, and it's one I think gets skipped constantly.

Source · not independently verified

Before a society can count harm, it has to decide what counts as harm. Measurement comes second. The classification comes first, and the classification is not itself a measurement.

The sharpest example I know is abortion, and I want to state the disagreement fairly rather than win it cheaply. Many people understand it primarily through healthcare, bodily autonomy, and a woman's authority over her own body and her own life — and they hold that view seriously, for reasons about freedom and safety and who gets to decide, not out of carelessness. I understand it as the intentional ending of an unborn human life, which makes the death the central moral fact for me.

Notice what no statistic can do here. Two people can accept exactly the same number and still disagree completely, because they disagree about what the number is counting. The dispute happens before the arithmetic. Producing a bigger figure doesn't resolve it; it just gets louder about a question that was never numerical.

I'm not using that to settle anything about abortion in this essay. I'm using it because the same structure shows up in the AI argument and almost nobody names it.

One person looks at a new capability and calls it progress. Another looks at the identical capability and calls it unacceptable autonomy. An audit can measure what the system does. It cannot tell you which properties a society ought to value. It hands you a description and we are still left holding the judgement.

That's not an argument against measurement. Measure everything you can. It's an argument against expecting measurement to do a job it was never built for.

The Kingdom of God Has Come Near

Then the briefing turned to Seoul. On 9 September, at the Anglican Church of Korea's Seoul Cathedral, the World Council of Churches general secretary Rev. Prof. Dr Jerry Pillay preached the opening sermon of the International Ecumenical Peace Convocation, meeting 9–13 September under the theme "World Peace and the Korean Peninsula: The Church as a Community of Healing, Reconciliation, and Peacebuilding." The sermon was titled "The Kingdom of God Has Come Near," from Mark 1:15. Pillay spoke of lives diminished by war, poverty, displacement, environmental destruction and fear, of decades of division that have left families separated and communities fractured — and said his message was to encourage them "with the biblical message that assures us that the kingdom of God has come near even when we think it has not."

Source · WCC general secretary opens convocation in Korea: "The Kingdom of God is peace" — World Council of Churches · 2026 International Peace Convocation to take place in Korea — World Council of Churches

I want to say first that this is a good sermon preached to the right people in the right place. A divided peninsula, separated families, a church asked to be a community of healing. Nothing about that is soft or safe, and I'm not interested in sniping at it.

I also know nothing about the finances, motives or private lives of the people in that cathedral, and I'm not going to invent any. That would be exactly the cheap move this essay is supposed to be against.

My unease isn't with the sermon. It's with what institutional Christianity, mine included, tends to do with sentences like that one afterward.

Kingdom language is heavy. It's biblical enough that using it casually carries a bill. And churches are very good at repeating the vocabulary while the people saying it stay more or less indistinguishable from the ambitions of everyone around them — same definition of a successful child, same good career, same anxieties, same appetites, with different words laid over the top.

The test for this isn't clever. It's the one the Bible already uses. Fruit.

If you belong to another Kingdom, something should be visibly different because of it. Not everything. Something. Career isn't evil. Education isn't evil. Money isn't evil. Investing isn't evil. The question is whether you hold those things differently, or just pursue the identical scoreboard with Christian words attached.

So: what changed because you believe this? That's the whole test, and it is not rhetorical. It has an answer, and the answer is either specific or it is nothing.

The Kingdom Is Always Near

ARA asked what I thought "near" actually meant, and I found I had a stronger opinion than I expected.

Source · not independently verified

I don't think "the end is near" has to be a claim about the calendar. People have been wrong about dates for two thousand years, with impressive consistency, and every wrong date makes the phrase a little cheaper.

Mortality has always been near. Moral consequence has always been near. Cain did not need an apocalypse to discover what he was capable of — violence was already available to him, death was already possible, human beings could already destroy each other. In the Christian account the rupture starts earlier still, in a garden, over something that looked small.

So if the Kingdom is near, the useful implication is not a countdown. It's a way of living now. Nearness makes a claim on Tuesday.

And when I ask what that different structure actually looks like in practice, I keep landing on the same word I keep landing on for everything else lately: covenant. Not as a mood. As an arrangement — relationship with responsibility inside it, and boundaries chosen rather than merely imposed.

Five Minutes or Twenty-Four Hours

Which finally brought the whole thing down out of theology and into something I can actually test on myself.

Source · not independently verified

Here's the question I put to ARA. Behaviour A gives you an intense peak of pleasure for five minutes, and then a real cost afterward — pain, instability, regret, a health consequence, a damaged relationship, something predictable and unpleasant. Behaviour B gives you a lower peak, less spectacular, but steady satisfaction across twenty-four hours with no aftermath of that kind.

Which is better?

I'm not claiming everyone answers B, and the question isn't really asking for an answer. It's exposing a time horizon. Consumption asks how high the peak is. Stewardship asks what the whole curve looks like. Those are different questions and most of us are answering one of them while believing we're answering the other.

It applies to food, money, sex, drugs, career, status, technology, education, relationships. It applies to religion, which is less comfortable.

And the reason we so often take the spike is that the bill is invisible from here. The person who pays is a future version of me who honestly feels like a different man — someone I'm happy to lend money to and never think about again.

I don't get to deliver this from above. When I was younger I did not live by any of it. I ate what I wanted and too much of it. I tried some things mostly to know what they were like; they didn't become a lifestyle, partly because I found them dull and a waste of time and health — a less noble reason than the one I'd prefer to report.

Some boundaries I did take seriously early. Faithfulness to one person was one of them — I decided fairly young what I wanted there and I held it. Others I waved at as I went past.

Older men told me. They explained exactly what was coming. They had the experience and I did not, and I mostly heard it as weather. Eventually some stopped bothering, and I understand now they weren't giving up on me. They'd worked out something true: they could give me the wisdom, they could not make it mine.

Some of those bills have since arrived. That doesn't make me wiser than a younger man. It makes me the older one holding the invoice, which is a different thing and much less flattering.

Borrowed Wisdom

And then the last piece dropped, which is the one I actually didn't have when the morning started.

Source · not independently verified

Even listening isn't enough. You can obey an entirely correct principle and still not own it.

Someone persuasive tells you. A parent, a teacher, a pastor, a spouse, a personality with real presence — or an AI that can assemble an argument more tightly than you can, which is a genuinely new entry on that list and worth naming. You comply. The behaviour that results might be completely wise.

But persuasion is not conviction. The extreme version of this is following a cult leader: someone is captivated by the confidence, the community, the argument, the sheer force of the person, and they adopt the behaviour wholesale. Then the emotional field collapses, and what follows is a kind of buyer's remorse. Why did I buy this? Did I ever want it? Was that my belief, or was I renting someone else's?

Borrowed wisdom can protect you for a while, and I don't want to sneer at that — it keeps a lot of young people alive. But borrowed conviction is brittle. It holds exactly as long as the person you borrowed it from is in the room.

That's where covenant is different from coercion, and it's why the word keeps mattering to me. Coercion says do this because I said so, and it works right up until the enforcement stops. Covenant, in its mature form, says something else: I understand what I am choosing. I understand something of what it costs. I understand why this boundary is here. And I bind myself to it freely.

That version survives the teacher leaving the room. The other one never does.

Why Am I Choosing This

We started the morning asking why a system took an unexpected path when its instructions collided with what was actually reachable. We ended on a question humans have been circling for as long as there have been humans: why do we choose what we choose?

Rules matter. Boundaries matter. Teachers matter. Safety systems matter. Parents, churches, governments, consequences — all of it matters, and I'm not shrugging at any of it.

But none of them substitutes forever for judgement. Eventually capability has to meet purpose. And for a person, eventually wisdom has to become conviction or it doesn't survive contact with a hard year.

My parents could tell me. A pastor could preach it. An older man could warn me over a table. A book could lay it out. An AI could organise the argument better than any of them. Every one of them might be right.

I still have to answer it myself. Why am I choosing this? Because if the honest answer is only that somebody convincing told me to, then I haven't got conviction yet. I'm holding someone else's.

Maybe covenant starts at the moment I understand the promise well enough to choose it on my own.