I was a corporate/IP lawyer in Silicon Valley for over 40 years and worked with 100s of startups. Sadly, this level of arrogance is not limited to PhDs. The ability of successful people to believe that their success is in one field expands to unrelated ones (like managing people for a technical founder or raising capital for a brilliant professor) is sadly quite common. It usually ends in failure although failure modes for startups are numerous. I particularly enjoyed the "explainer" on the on the purpose of hedge funds. However, a reminder that Situational Awareness still has about $10 billion which I understand is mainly in Anthropic stock, so he may still have a nice exit (his investors, not so much).
Michael Shermer, Editor-in-chief of _Skeptic_ magazine, wrote a book 30 years ago called _Why People Believe Weird Things_. The final chapter was about why _smart_ people believe weird things.
His thesis for that chapter was that smart people come to believe weird things for non-smart reasons (like groupthink), but then are smart enough to defend those weird beliefs against skepticism. They're smart enough to reinforce the belief over time, and get entrenched.
I think about that a lot with lab-grown AI evangelists.
A good article. Personally, I think their arrogance is worse than what you say. A small amount if any of their knowledge comes from direct experience and most to almost all depending on the person comes from the studies and articles they read. So many studies these days are biased, both intentionally and because those running them really don't know that much about statistics or design of experiments. If you ask a question, they give much more credence to those writers and experimenters in their own field with similar credentials. I guess everyone does that. I everyone thinks they are smarter than they really are. When questions are raised, instead of looking inward and treating it seriously, they react defensively and do everything they can to get him. That's one thing I like about AI. When I'm designing and coding with AI, there are no egos involved. It's such a relief. But even with all of that, some wonderful things are being done and the power of AI is an example, assuming the net effect will be positive for the masses rather than the asses. Can you imagine if the world consisted of mostly open people, interested in the benefit of everyone rather than their own selfish interests? That's what academia should be focused on promoting in my opinion. As soon as money is involved, everyone's agenda and ethics seem to change for the worse, no matter the field. Sorry, I seem to use your articles to go on a rant all too frequently. That's probably why I haven't written any of my own articles! I go for a ride on yours! ;-)
The Hugging Face case is the strongest item here because the arrogance produced a measurable outcome rather than an embarrassing quote. And the consequence outlives the incident. That refusal is now written into an incident response runbook, which is the document a security team follows at three in the morning, and runbooks propagate into tabletop exercises, audit evidence and eventually insurance questionnaires. A decision made in San Francisco about who is trusted to analyse a payload ends up as a Chinese model named at step four of a Western procedure, and nobody in that chain chose it as policy.
Thank you! Absolutely loved it. I am a huge fan of AI tools and love working and experimenting with them and the great potential of this technology to make the world a better place. However, the narrow and reductionist worldview of some of the leading faces and voices in AI, shows above all a lack of experience with, empathy for, and imagination about the universe of things that exists outside their bubble. It has tendencies of a cult. - The economic fall-out of the cultish overshoot will cost all of us, a lot. And will also delay the benefits of the ongoing technological revolution by five to ten years. Which also costs all of us a lot.
Appreciate it! I think my exasperation with all of this has been building for a while, so it had a lot of time to marinate even before Situational Awareness blew up.
Leopold’s perspective on Artificial General Intelligence (AGI) appears fundamentally flawed. Given his extensive educational background, it is surprising that he seems to conflate the illusion of intelligence with its true essence. Current AI models act as a metaphorical mirror, reflecting human intellect rather than possessing it. The underlying mathematical frameworks and mechanisms of these models share very little with actual human cognition. By overstating the significance of what is essentially a technological mirage, he misrepresents the technology's current state—a public stance that seems primarily motivated by financial opportunism rather than objective analysis.
I have witnessed this specific type of intellectual hubris before while working at a hedge fund. When collaborating with individuals holding PhDs across various STEM fields, it is easy to fall into the comforting illusion that they will inherently possess the correct answer to any given problem. Consequently, they often stop applying objective, critical thought. When they inevitably fail, they remain conspicuously silent; yet, when market conditions are favorable, they are quick to overstate their personal opinions and reinforce their own brilliance. This kind of intellectual complacency is deeply problematic, especially when navigating irrational markets—a dynamic that seems highly applicable to the current narrative surrounding AI. (i.e. Also an AI optimist, trying to be zero sugar hype)
"No one likes an arrogant prick—especially if they’re wrong." I've been that arrogant prick. It's really hard to recognize the places where the Dunning-Kruger effect applies to you.
> His claim to fame was being part of OpenAI’s Superalignment team before being fired over alleged leaks (which he disputes) and then publishing an essay in 2024 about the imminence and importance of AGI that launched a thousand media interviews (of him). And then he was running a $20 billion hedge fund. And then it blew up.
His claim to fame was providing 4000% returns to his LPs on an investment thesis he largely posted publicly.
"Instead of policymakers or society making this decision, it’s effectively been centralized to AI labs as the “safe hands”" This is unfair since Anthropic especially has been under intense pressure from the US government (that's policymakers for you) to impose exactly these kinds of strictures.
I was a corporate/IP lawyer in Silicon Valley for over 40 years and worked with 100s of startups. Sadly, this level of arrogance is not limited to PhDs. The ability of successful people to believe that their success is in one field expands to unrelated ones (like managing people for a technical founder or raising capital for a brilliant professor) is sadly quite common. It usually ends in failure although failure modes for startups are numerous. I particularly enjoyed the "explainer" on the on the purpose of hedge funds. However, a reminder that Situational Awareness still has about $10 billion which I understand is mainly in Anthropic stock, so he may still have a nice exit (his investors, not so much).
Michael Shermer, Editor-in-chief of _Skeptic_ magazine, wrote a book 30 years ago called _Why People Believe Weird Things_. The final chapter was about why _smart_ people believe weird things.
His thesis for that chapter was that smart people come to believe weird things for non-smart reasons (like groupthink), but then are smart enough to defend those weird beliefs against skepticism. They're smart enough to reinforce the belief over time, and get entrenched.
I think about that a lot with lab-grown AI evangelists.
A good article. Personally, I think their arrogance is worse than what you say. A small amount if any of their knowledge comes from direct experience and most to almost all depending on the person comes from the studies and articles they read. So many studies these days are biased, both intentionally and because those running them really don't know that much about statistics or design of experiments. If you ask a question, they give much more credence to those writers and experimenters in their own field with similar credentials. I guess everyone does that. I everyone thinks they are smarter than they really are. When questions are raised, instead of looking inward and treating it seriously, they react defensively and do everything they can to get him. That's one thing I like about AI. When I'm designing and coding with AI, there are no egos involved. It's such a relief. But even with all of that, some wonderful things are being done and the power of AI is an example, assuming the net effect will be positive for the masses rather than the asses. Can you imagine if the world consisted of mostly open people, interested in the benefit of everyone rather than their own selfish interests? That's what academia should be focused on promoting in my opinion. As soon as money is involved, everyone's agenda and ethics seem to change for the worse, no matter the field. Sorry, I seem to use your articles to go on a rant all too frequently. That's probably why I haven't written any of my own articles! I go for a ride on yours! ;-)
So true; this reminds me of a book “Range” where specialized experts in one field often step out of line with other fields.
Excellent post - and that compliment is coming from someone frequently called "an arrogant prick." :-)
The Hugging Face case is the strongest item here because the arrogance produced a measurable outcome rather than an embarrassing quote. And the consequence outlives the incident. That refusal is now written into an incident response runbook, which is the document a security team follows at three in the morning, and runbooks propagate into tabletop exercises, audit evidence and eventually insurance questionnaires. A decision made in San Francisco about who is trusted to analyse a payload ends up as a Chinese model named at step four of a Western procedure, and nobody in that chain chose it as policy.
Thank you! Absolutely loved it. I am a huge fan of AI tools and love working and experimenting with them and the great potential of this technology to make the world a better place. However, the narrow and reductionist worldview of some of the leading faces and voices in AI, shows above all a lack of experience with, empathy for, and imagination about the universe of things that exists outside their bubble. It has tendencies of a cult. - The economic fall-out of the cultish overshoot will cost all of us, a lot. And will also delay the benefits of the ongoing technological revolution by five to ten years. Which also costs all of us a lot.
This was so peak. One of the best articles you've done to date
Appreciate it! I think my exasperation with all of this has been building for a while, so it had a lot of time to marinate even before Situational Awareness blew up.
Leopold’s perspective on Artificial General Intelligence (AGI) appears fundamentally flawed. Given his extensive educational background, it is surprising that he seems to conflate the illusion of intelligence with its true essence. Current AI models act as a metaphorical mirror, reflecting human intellect rather than possessing it. The underlying mathematical frameworks and mechanisms of these models share very little with actual human cognition. By overstating the significance of what is essentially a technological mirage, he misrepresents the technology's current state—a public stance that seems primarily motivated by financial opportunism rather than objective analysis.
I have witnessed this specific type of intellectual hubris before while working at a hedge fund. When collaborating with individuals holding PhDs across various STEM fields, it is easy to fall into the comforting illusion that they will inherently possess the correct answer to any given problem. Consequently, they often stop applying objective, critical thought. When they inevitably fail, they remain conspicuously silent; yet, when market conditions are favorable, they are quick to overstate their personal opinions and reinforce their own brilliance. This kind of intellectual complacency is deeply problematic, especially when navigating irrational markets—a dynamic that seems highly applicable to the current narrative surrounding AI. (i.e. Also an AI optimist, trying to be zero sugar hype)
"No one likes an arrogant prick—especially if they’re wrong." I've been that arrogant prick. It's really hard to recognize the places where the Dunning-Kruger effect applies to you.
Malthus has not been proven wrong yet.
“A rising tide lifts all boats.”
—-Old Wall Street Trite Platitude
Or, as per Warren Buffett:
“After all, you only find out who is swimming naked when the tide goes out.”
> His claim to fame was being part of OpenAI’s Superalignment team before being fired over alleged leaks (which he disputes) and then publishing an essay in 2024 about the imminence and importance of AGI that launched a thousand media interviews (of him). And then he was running a $20 billion hedge fund. And then it blew up.
His claim to fame was providing 4000% returns to his LPs on an investment thesis he largely posted publicly.
"Instead of policymakers or society making this decision, it’s effectively been centralized to AI labs as the “safe hands”" This is unfair since Anthropic especially has been under intense pressure from the US government (that's policymakers for you) to impose exactly these kinds of strictures.