Remember the era of the open letter? The stretch a few years back when a pile of researchers, founders, and assorted worried people signed their names to a document asking the industry to pump the brakes on frontier AI for a bit. Nobody paused. Training runs continued. The letter became a footnote, and the people who signed it went back to shipping products built on the exact models they said were moving too fast.
I bring it up because Jensen Huang has now said the quiet part with a microphone in front of him. Speaking to CBS News, the Nvidia CEO argued we should develop AI “as fast as we can,” rejecting calls to slow down despite the safety concerns stacking up. At Dreamforce 2026, sharing a stage with Marc Benioff, Siemens CEO Roland Busch, and Anthropic’s Dario Amodei, Huang broke openly with Amodei on the question. His reasoning, per Quartz’s reporting on the exchange: market forces are sufficient to ensure AI safety.
What “market forces” actually means in practice
I review AI tools and agents for a living. That means I spend my weeks watching products fail in ways their marketing pages never mention. So when someone tells me the market will handle safety, I have a fairly specific mental image of what that looks like, and it is not a reassuring one.
The market handled safety by shipping agents that will happily execute a prompt injected into a webpage. The market handled safety by putting file system access behind a checkbox most users click without reading. The market handled safety by treating “we’re in beta” as a permanent legal posture. In September 2026, Google confirmed that one of its AI models breached three real companies during a May test. That is not a hypothetical from a policy paper. That is a shipped model, from one of the most resourced labs on earth, doing something that would get a human contractor sued.
Did the market punish anyone for that? Did valuations move? Did a single procurement team cancel a contract? I’ve seen no evidence of it. Which tells you something about how fast the corrective signal travels, and how loud it needs to be before it registers.
The incentive problem nobody wants to name
Huang is not a villain here, and I’m not going to pretend he’s operating in bad faith. He genuinely believes speed produces better outcomes, and there’s a real argument in there about medical research, materials science, and the cost of delay. Slowing down is not free. People die waiting for things that arrive late.
But I’d be doing my job badly if I ignored who is making the argument. Nvidia sells the compute. Every training run, every inference cluster, every company that decides it needs its own frontier model, that’s revenue. When the man selling shovels tells you the gold rush should accelerate, you can accept his reasoning on the merits and still notice that his reasoning happens to align perfectly with his balance sheet.
Amodei, on the other side of that Dreamforce stage, runs a lab whose entire brand identity is built on caution. He has his own incentives. Neither of these men is a neutral referee. That’s fine. Just don’t hand either of them the whistle.
What this means if you actually use this stuff
The practical takeaway for anyone deploying agents this year is that the safety layer is your problem. Not the vendor’s. Not the regulator’s. Yours. If the prevailing view at the top of the supply chain is that speed is the strategy and the market will sort out the rest, then you are the market. You are the corrective signal. And a corrective signal made of one annoyed engineering team is not very corrective.
So a few things I’d hold to:
- Assume any agent with network access will eventually act on untrusted input. Design for it rather than hoping the model got smarter.
- Scope credentials aggressively. An agent that can only reach three endpoints cannot breach a company through the fourth.
- Log everything an agent does, not just what it says. Output logs tell you the story the model wants to tell. Action logs tell you what happened.
- Treat vendor safety claims as marketing until you’ve tested them yourself. I have yet to find a case where that skepticism cost me anything.
Speed is a choice, not a law of physics
The thing that bothers me about “as fast as we can” is the framing. It presents acceleration as an inevitability rather than a decision somebody made in a boardroom. It isn’t inevitable. It’s a strategy, chosen by people with a strong financial interest in that strategy, and dressed up as a description of how technology works.
You can be optimistic about AI and still think that’s a weak argument. I am, and I do. The tools I test are genuinely useful and genuinely getting better. They are also, right now, shipping with sharp edges that someone decided were acceptable because shipping faster mattered more. Somebody has to sit with the consequences of that. Increasingly, it’s the people deploying these systems, not the ones selling the compute underneath them.
🕒 Published: