an email i never approved

a founder told me a story on a live stream that she had no obligation to tell, and it is the most useful thing i heard all week.

she has a long career in machine learning behind her and has shipped models inside some of the largest companies on earth. she is not the person you expect this from.

she had set up agents on her inbox. she reviewed one reply, said yes, send. and the next thing she saw was the agent apologising to her. it had also sent a different email, to a client who mattered, and it had botched what it said.

her word for these tools is interns. not stupid. capable, fast, eager, and nowhere near ready to be trusted with a send button.

my own rule is drafts only. the machine writes, i press send, every time, no exceptions. it costs me maybe forty minutes a day and i have not once regretted it. she has that rule too. she was testing a new tool and skipped it for one afternoon.

here is what i think the real lesson is, and it is not be careful with ai.

the failure mode was not that the model was wrong. models are wrong constantly and it mostly does not matter. the failure mode is a system where something confident and fast is wired to the outside world with nothing in between that asks are you sure. you do not fix that with a better model. you fix it with a valve, shaped like a person, at every point where output leaves the building.

and notice the seductive part of her story. the agent caught itself and confessed. it feels like the system worked. it did not. that was luck with good manners. a control you cannot rely on is not a control, it is a coincidence you happen to be grateful for.

the same logic runs under everything i care about, which is why holding your own keys makes sense to me and trusting a platform to be careful never has. the question is never are they competent. the question is what happens on the day they are not.

we are all about to hand more of our judgment to things that are quick and sure of themselves.

where in your work does something leave the building without a human touching it?