Lloyd Blankfein just put his finger on why even Goldman Sachs is wary of AI agents

Lloyd Blankfein spent decades at Goldman Sachs learning how to manage risk at scale. He watched the firm navigate the 1987 crash, the dot-com bust, the 2008 financial crisis, and the post-crisis regulatory overhaul that reshaped Wall Street. So when the Goldman senior chairman and former CEO says something worries him about AI, it’s worth paying attention to what, exactly, that thing is.

It’s not superintelligence or autonomous weapons. It’s a much more mundane — and in some ways more frightening — problem.

The problem with AI is “not because it’s smarter than us and going to turn us into pets,” Blankfein said in a new interview on Andreessen Horowitz’s The a16z Show, published Monday, “but because we don’t have the ability to test whether it’s right or not.” When you’re running a big institution, he explained, you can’t make mistakes and numbers really matter.

Alluding to AI in particular but technological advancement in particular, he said, “everything is whirring behind the scenes,” and you don’t really get a close look at the thought process of the technology on which you’re relying. “Now you can leave a piece of software, [and it] could go out and do 70,000 transactions,” he said, explaining that when he started on the trading floor, everyone could hear every mistake, and the room would get quiet at the smallest slip-up.