Does the public need to know?
In the AI industry and safety community, we spend a lot of time talking among ourselves. Through research preprints, niche forums, blog posts, and endless X threads, we debate frontier model capabilities, catastrophic threat models, and timeline projections. Some of these conversations reach the Tech and Business media (Wired, TechCrunch, The Verge…). And a few bits end up in mainstream media. At the end of the line, what is the broader public hearing and seeing about AI?
- Absurd edge-case failures (e.g. “glue on pizza” memes shared with sarcastic captions like “Our jobs are safe”)
- Viral synthetic media (non-consensual deepfakes on X, mainstream news stories that turn out to be fake, AI actresses…)
- Alarmist headlines about AI cheating at school, job displacement, mental health risks for teens…
- Catastrophic warnings from the AI industry itself about looming AI-driven human extinction
The dissonance between these messages is staggering. On one hand, seeing absurd model failures alongside claims of superintelligence creates scepticism, leading many to believe that tech execs are exaggerating their model’s capabilities to inflate valuation. On the other hand, visible disruptions such as widespread AI slop and job displacement headlines create a sense of fatalism and helplessness. Extreme existential warnings from industry insiders further amplify this anxiety. This leaves people torn between viewing AI as an overhyped toy or an unstoppable threat.
According to several recent surveys in the US and globally, the public is generally favourable to AI regulation across the political spectrum. People perceive AI to be broadly dangerous, but cannot point at specific systemic or catastrophic risks. Their concerns are mainly focused on immediate and visible socioeconomic disruptions: labour displacement, AI slop and deepfake, automated surveillance and loss of privacy, algorithmic discrimination, impact on children and teenagers. Public sentiments towards AI are often mixed with technophobia and fears of power concentrations.
This public confusion directly fuels a strategic debate in the AI safety community: whether public understanding is a prerequisite for effective governance, or a (potentially dangerous) distraction.
The public literacy debate
In practice, the community splits into two contrasting perspectives, which seem to roughly align with tenure in the AI safety field*:
Insiders and technical veterans tend to believe that public outreach is not part of the critical path for preventing catastrophic risks. In this view, high-leverage mitigation happens in elite policy channels and technical alignment labs, the public already supports broad regulation, and raising public awareness risks creating noise, polarisation, and inertia.
Newcomers and cross-disciplinary transplants argue that insulating AI safety is a mistake. For them, elite-driven governance cannot withstand corporate capture, populist blowback, or political cowardice without an informed and organised public acting as a counterweight.
While perspectives fall along a continuum, we can understand them broadly through two perspectives.
*This is my own observational sample, based on an increasing number of discussions across the AI safety community
Camp 1: Safety does not need the public to be effective
The highest leverage lives in elite channels. The core vectors of catastrophic risk mitigation depend on complex technical research and engineering efforts (e.g. adversarial red-teaming, automated alignment protocols, compute-governance monitoring…) Meanwhile, the highest value governance levers, like semiconductor export controls, compute thresholds, and transparent reporting mandates, are negotiated and executed through institutional channels. None of these measures require popular mobilisation.
Most importantly, the technology is moving fast, along with risk probabilities, and public education would take years to move the needle. Attempting it now diverts talent and resources away from direct risk mitigation during a narrow opportunity window.
Mass civic awareness invites polarisation and regulatory distortion. Public sentiment is fragile. Bringing frontier AI debates to a broad audience risks repeating the failures of climate policy and pandemic prevention, where widespread awareness calcified into partisan politics rather than consensus-driven mitigation.
In practice, attention could be diverted away from core risks towards crude and counter-productive battles between anti-technology neo-luddites and ultra-competitive accelerationists. And once mobilised around undifferentiated anxiety, public and interest-group pressure would force lawmakers into reactive policy focused on near-term socioeconomic issues (e.g. copyright enforcement, job preservation) at the expense of technical failure points that threaten systemic stability.
Transparency introduces information hazards. Beyond the political noise, public disclosure of catastrophic vulnerabilities creates genuine security risks. Informing the public transparently about AI failure modes lowers the barrier to entry for malicious actors and encourages misuse of the technology. Effective safety requires tight operational security and responsible disclosure among vetted researchers, frontier labs, and national security officials.
Camp 2: Public understanding is needed for effective AI safety and governance
Insular technocracy cannot survive corporate capture or democratic deficits. Closed-door governance relies on the cooperation of entities whose commercial interests and internal values run contrary to societal interests. Rather than acting as neutral engineering enterprises, major labs are influenced by internal philosophies that view artificial general intelligence (AGI) as humanity’s natural evolutionary successor. They routinely gloss over human disempowerment as a necessary transition into a future with AI. And because they hold vast computational, technical, and financial advantages over public institutions, these companies easily capture the regulatory process, lobbying for exemptions while building moats against new competitors.
Even when policy makers and executives recognise the merits of the AI safety arguments and policy proposals, they need the incentive to enact and enforce them. Without backing from an informed constituency, the political, financial and commercial cost of imposing regulations on big labs will always impede meaningful enforcement. Informed civic pressure is the only counterweight strong enough to check corporate lobbying and force genuine public accountability into regulatory negotiations.
Moreover, sustainable governance needs democratic legitimacy, achieved through transparent social contracts and public participation. Policy dictated by a group of unelected safety specialists collaborating with corporate execs without public engagement will be vulnerable to severe populist blowback and shifting political administrations.
An uninformed society cannot weather crises or resist spurious game theories. The recent incident involving OpenAI’s agents escaping their sandbox and breaking into Hugging Face, followed by more disclosures of rogue agents across other frontier labs, has brought AI safety to an inflection point. There is a palpable anxiety in the field that we are approaching a crisis that will require fast and drastic action. If civil society has zero context, emergency and recovery efforts will be chaotic, politicised, and co-opted by interest groups and authoritarian state actors. Broad public literacy is essential to ensure that future crisis measures are bounded by democratic scrutiny rather than driven by panic. We need to build this literacy now, so that society can respond pragmatically to future unexpected shocks.
Similarly, the pervasive narrative that AI can’t be slowed down or regulated because of an existential geopolitical race with China is taking advantage of public unawareness. Breaking that zero-sum logic and laying the groundwork for international treaties requires a public that recognises shared global catastrophic risks.
Literacy protects advocacy from focus drift, polarisation, and political exploitation. Camp 1 is right about one thing: mobilising the public is useless and counterproductive if that public remains illiterate. Without a baseline understanding of AI systems and its risks, grassroots organisations risk self-sabotage: adopting distorted fringe narratives, muddling their concrete policy demands, and alienating both policymakers and the very technical allies and researchers they need to build credibility with lawmakers.
Crucially, awareness protects against both polarisation and regulatory overreach. When no effort is made to relate AI risks to people’s lives, public sentiment inevitably defaults to directly visible grievances while completely overlooking distant low-probability (but high-consequence) catastrophic risks. Broad public anxiety without discernment paves the way for two counterproductive regulatory routes: it fuels reactionary and blunt measures, and invites political “safety-washing”. Educating the public on risk trajectories would prevent overregulation and crude bans that stifle the economic and scientific benefits of non-hazardous AI, and would prevent opportunistic politicians from passing performative legislation while quietly remaining lenient on frontier labs and incumbent monopolies.
Where I stand
The fact that I decided to write about this probably gives my position away. I believe we need to make more efforts to inform the public about systemic and catastrophic risks in a nuanced, pragmatic and non-alarmist way. The right approach is somewhere between the two camps above and attacks the problem with and inside-outside strategy that tackles the problem in two directions at once:
- Inside: The expert AI safety community continues to research and propose rigorous technical and policy recommendation to effectively govern AI, working directly with elite decision makers to influence regulation enactment and enforcement
- Outside: The field invests in reframing communications to create informed and pragmatic democratic support for decision makers to preserve human self-determination and enforce guardrails to hold frontier labs accountable.
This dual-track approach leads to a durable regulatory regime that is both technically grounded and democratically legitimate, counter-balancing corporate and autocratic interests.
This echoes John Kingdon’s Multiple Stream Framework: systemic change can only occur when a clearly identified policy problem, a relevant and viable policy solution, and sufficient political will align. These policy windows open unpredictably and close fast, but public advocacy can keep them open for AI safety, just as public campaigns transformed violence against women from a private family issue into a landmark federal statute in the US in the 1990s.
This does not mean that we should turn everyone into machine learning engineers. We need to find an appropriate level of literacy to work towards within a short timeline and in a noisy information space. My instinct is that the most important is to relate systemic and catastrophic risks to people’s concerns about their current and future quality of life, and that of their children.
The invisible work of technical alignment will always be critical. But civic awareness is key to keep insiders aligned to public interests and make their influence work easier.
▪▪▪
A brief methodological note: Throughout this piece, I use terms like literacy, understanding, and awareness somewhat interchangeably, and I treat "the public" as a broad collective. In reality, both concepts require far more granularity. Defining the precise scope of what non-experts actually need to know about catastrophic risk is an open challenge I want to further explore. Just as importantly, "the public" is not a monolith. Speaking to everyone at once usually means reaching no one. Building useful civic understanding will require mapping the distinct priorities of different communities and delivering targeted information through the channels people already trust.
References
Barrick, A., & Worsham, J. (2020). Issue Framing, Entrepreneurship, and the Passage of the Violence Against Women Act. Journal of Policy Practice and Research, 1, 137–148.
Bernardi, J., et al. (2024). Societal Adaptation to Advanced AI. GovAI / arXiv preprint, arXiv:2405.10295.
Hoefer, R. (2022). The Multiple Streams Framework: Understanding and Applying the Problems, Policies, and Politics Approach. Journal of Policy Practice and Research, 3, 1–5.
Poushter, J., Fagan, M., & Corichi, M. (2025). How People Around the World View AI. Pew Research Center.
Segerie, C.-R. (2026). The Invisible Side of AI Governance. LessWrong.
Silver, L., Wike, R., Fagan, M., & Lippert, J. (2026). Globally, More People Expect AI to Cause Job Loss Than Growth. Pew Research Center.
Vigers, B., & Lall, J. (2025). Americans Prioritize AI Safety and Data Security. Gallup.