One unfortunate side effect of “The Singularity” is that the name can conjure up a very static and shallow sense of the option space. If there is A Moment when Transformative AI Arrives, then surely there’s A Solution: a breakthrough alignment technique, or a landmark legislative framework that sets the rules of the road, or a treaty with China to coordinate on risks.
Needless to say, I suspect that there will be important legislative frameworks and perhaps even treaties, and I certainly hope for breakthrough alignment techniques. But I worry this implicitly frames the solution space as a one-shot game awaiting a master plan — humanity’s own Move 37.
There may in fact be moments where decisive action is necessary. But I suspect getting AI right is going to feel more like a chaotic, iterative process of institutions trying to make better decisions over time as the facts change underneath them.
This is, in part, because it’s surprisingly hard to know what technical red lines it would be a good idea to enforce today. Two or three years ago, for example, the consensus in AI safety was to include alignment research data in pretraining so that the models would deeply understand what we wanted from them. Now the emerging view is the opposite, because that same data can teach models to evade the mitigations it describes.
So people will disagree, often in good faith, update, switch sides, underreact, overreact, measure the wrong things, sometimes fail for very boring implementation reasons, and sometimes succeed because we got lucky. Factions will clash and counteract each other and discursively feel out the edges of a compromise. And the sudden arrival of new AI capabilities will render certain assumptions obsolete and scramble coalitions as the future announces itself in uneven, undeniable bursts.
Some disagreements over AI policy are rooted in value conflicts that benchmarks can’t settle. Those must be negotiated through the democratic process. But many are actually disagreements about which AI future we will find ourselves in. For those, we can hedge across different worlds. To that end, the most useful intervention is to build up institutions — within government and outside — that can adapt in real time to new information, command the legitimacy and capacity to act on what they learn, and preserve option value so that mistakes are fixable, not permanent.
The alignment problem, in this broader political sense, asks whether we can build wise, capable institutions that help us steer through the transition to a world of powerful AI systems. I take the catastrophic risks from advanced AI seriously, and I’m not at all confident we’re on the right track to handle them.
I recently joined Coefficient Giving as head of public policy, where I’m launching a new team focused on strengthening the institutions that will have to navigate this transition. My central belief is that we need to rapidly build wise, capable institutions that expand our optionality, while doing ambitious policy development now so that well-vetted ideas are on the shelf for the hard-to-predict moments when important policy changes will actually happen. And because nobody knows when those moments will arrive or who will hold power when they do, the work has to run through durable, cross-partisan coalitions engaging a wide range of AI-policy issues, without losing the thread on the debates that matter most.
I’m also excited to oversee the Abundance and Growth Fund and CG’s general government relations. That is a broad remit, but a coherent one. Because from an institutional perspective, good outcomes are correlated. Worlds in which government officials have the situational awareness to nimbly respond to new bio and cybersecurity vulnerabilities enabled by frontier models are more likely to be worlds where the FDA streamlines clinical trials for AI-designed drugs and where we have agile certification pathways for autonomous transit and energy grids. The underlying building blocks of institutional capacity are the same. Almost every policy goal you might care about, from AI to housing to energy to science to economic growth, runs through wiser, more capable institutions.
Scaling, uh, finds a way
I have lots of uncertainty about where AI is going and how fast. But the closer you get to the technology, the faster time (and timelines) seem to move. Researchers at the AI labs are saying there’s a realistic chance that we kick off a loop of recursive self-improvement in the next 12-24 months.
The tiny economist on my shoulder (who looks suspiciously like Tyler Cowen) keeps saying “bottlenecks are everywhere and reality has a surprising amount of detail”. And there are plausible reasons to expect AI could fail to transform the world in quite the way or on the timeline that prominent voices expect.
We are moving now beyond the realm of rich training data and it’s unclear whether sample efficiency is improving and, if it isn’t, how important that fact will be. As Tom Reed notes in a smart piece, lots of the relevant data for performing at or beyond human-level expertise doesn’t exist in standardized forms at all: “There is no GitHub for closing a Series B, winning a knife fight, or negotiating with Houthis.” More broadly, it may turn out that progress in many fields (including in AI R&D) is bottlenecked more by the capacity to run experiments in the real world than it is by raw intelligence.
And yet, despite all these very good arguments, almost every relevant measure of AI progress has gone up and to the right. There were plausible reasons AI progress could slow down three years ago, and two years ago, and one year ago, and today. But the line has gone up and to the right for long enough that the evidentiary tables have turned on skeptics of AI capabilities.
That said, you don’t have to be completely sold on the fast-takeoff story to recognize the urgency here. The question isn’t just “how fast?” but “what if?”. What will we wish we had the capacity to do if current trends hold? You still buy a fire extinguisher even if you don’t expect your house to burn down.
As a general-purpose technology, AI will affect foreign policy, tax policy, industrial policy, labor policy, monetary policy, healthcare, education, liability, scientific discovery, public administration, and the balance of power between firms, individuals, and the state. Tom Cunningham’s plea to economists to take AI seriously applies in equal measure to most policy professionals:
There is a substantial probability (>10%) that AI will exceed human-level performance on virtually all non-physical tasks within ten years.
This would be an unprecedented shock to human society.
The economics profession should treat it with an urgency comparable to WWII or COVID.
Building wise, capable institutions
The uncertainty over the pace of AI capabilities, and the second, third, fourth, and nth order effects of that progress, mean that we should bet on AI policy moves that increase option value, are robustly good across as many timelines as possible, and preserve societal and individual agency. As Christoph Winter and Charlie Bullock of LawAI write in Radical Optionality, “Instead of relying on uncertain predictions about exactly how transformative AI will be developed and what it will be capable of, governance efforts should focus on ensuring that, when and if important decisions need to be made, governments have the institutional capacity to make them well.”
What do I mean by wise, capable institutions?
Wise institutions exercise good judgment under uncertainty. They know what they do and do not know, and internalize all the ways they could be wrong so they can fail gracefully when possible. They are live players, capable of doing genuinely new things rather than executing a script. And they are decisive when the moment calls for it, taking ownership whether things go well or poorly. There is an old-fashioned virtue of statesmanship in all this that I think we need more of: prudence, seriousness, courage, and accountability.
If that sounds a bit mushy, remember that one of our most resilient alignment techniques is teaching models to exhibit good character, and consider whether you might be underrating virtue ethics.
Capable institutions can actually accomplish the goals they set. This is the more classic state capacity definition: do you possess the requisite legal authority, funding flexibility, technical talent, and implementation muscle to do the thing? It sounds simple, but a capable institution would not, for example, create a months-long loophole in semiconductor export control enforcement by accident.
Perhaps most important, wise, capable institutions require wise, capable people. Personnel is policy. Institutions are not just a collection of rules and procedures; nor are they freewheeling constellations of heroic individuals making things up as they go. At their best, institutions are durable systems for turning judgment into legitimate action: high-integrity people embedded in structures of authority, incentives, norms, tacit knowledge, and organizational memory that let them work together well over time.
And institutions are not just government agencies. METR is one of my favorite examples, and it lives entirely outside of government, squinting at hard-to-measure parts of AI progress better than almost anyone. We should aspire to make Congress, political parties, labs, media, and philanthropy itself behave more like wise, capable institutions. AI governance will depend on this whole ecosystem, not on one agency or one bill.
Sometimes the work here is uncontroversial. Regardless of what AI future you think we’re heading towards, there is essentially no downside to a well-funded, well-functioning Center for AI Standards and Innovation that can assess the pace of AI progress and help the government make sense of the trends. It would cost only ~$84 million a year to have a CAISI capable of executing Trump’s AI Action Plan, chump change in the world of government spending. Why hasn’t this happened already? Largely because of inertia, prioritization, and low political salience, which are exactly the kinds of problems that effective policy entrepreneurship can solve.
Beyond CAISI, what sorts of institutions might increase our odds of a good outcome here? A few examples:
Supercharged congressional capacity, both to help members understand trends in AI and, critically, to provide meaningful oversight of executive branch use of AI.1
A healthy ecosystem of third-party auditing organizations that can operate more nimbly than government and provide a credible independent assessment on the risks of both internal and external AI deployments. The Madisonian insight that “ambition must be made to counteract ambition” applies just as well to auditors checking labs as it does to Congress checking the executive.
A technical talent reserve that can rapidly provide clearances for national security relevant work and move engineers from the frontier labs into public service and back out again.
A “frontier capabilities observatory” that can spend inordinate amounts of inference compute to see how far our strongest models can be pushed on dangerous misuse tasks in carefully controlled environments. The goal would be to buy society a larger adaptation buffer with more time to harden our defenses.
An equivalent of the National Transportation Safety Board for AI that can impartially investigate accidents and near misses to reconstruct what went wrong and why.
Journals and think tanks in every nook and cul-de-sac of the political spectrum that can help wrestle with and metabolize the rapid pace of AI for a pluralistic set of political and philosophical traditions.
Ambitious R&D institutions that can shape the technology tree: accelerating defensive breakthroughs like AI verification and cyberdefense, but also the scientific infrastructure to actually capture the upside, using the full suite of innovation-funding mechanisms.
A central challenge is that many of the traditional avenues of institutional evolution will break down at the scale and speed of change that transformative AI may bring. We might not have the luxury of a decade of iterative trial-and-error; we may need to find ways to radically compress those learnings and iteration cycles into just a few years. That will entail ambitious new experiments in institutional form and function. Can we build and scale technical institutions like CAISI more nimbly outside of government and graft them onto the bureaucracy later if the moment calls for it? Can we charter two or three organizations to pursue the same mission with different theories of change, the way DARPA funds rival technical approaches to one problem?
Wise and capable should not be confused with staid and incrementalist. The point of building institutions that can see clearly is that they can act on what they see. If the evidence starts arriving quickly, we should stand ready to endorse interventions in a couple of years that would seem radical today.
Preparing for punctuated equilibria
What does all this mean practically? How should we prepare?
Political scientists sometimes talk about policy, especially legislative policy, as operating in a punctuated equilibrium: nothing happens for long stretches of time, and then suddenly everything happens at once. A classic example is the PATRIOT Act and the formation of DHS as a response to 9/11; regardless of whether you think these were good or bad moves, they constituted one of the biggest domestic security overhauls in decades. Most of our modern biosecurity architecture was written in the short window after the anthrax attacks, and the launch of Sputnik prompted the US to create NASA, DARPA, and the National Defense Education Act within a year.
It seems possible, if not likely, that we will see one or more similar moments in AI policy in the coming years. I don’t know what the triggers will be, and how much it will be a single dramatic moment vs. a tipping point after years of progress. It could be a major cyber attack, compelling evidence of misaligned behavior on internal systems, a child safety crisis, a shocking capabilities jump from a Chinese AI lab, or something that none of us are currently able to predict. The policy changes that follow these moments are likely to be much more significant than the incremental changes beforehand.
Being ready for that moment means investing in policy talent so there are technically competent people in the room to debate trade-offs and exhibit good judgment. It means well-vetted ideas on the shelf that legislators can pull from, implementation details and legislative language worked out ahead of time, and relationships with members, staff, agencies, labs, civil society, and intellectual communities who will matter in a crunch. Otherwise, when the window opens, the menu of options will be panic, symbolic legislation, industry-written rules, poorly-aimed overreaction, or whatever random provisions happen to be lying around.
Durable AI policy also has to be able to survive changes in political power. This doesn’t mean aiming for all sides to agree on everything, that would be both impossible and undesirable. AI will touch too many genuinely political questions. But if particular ideas become too identified with a single faction, the other side will often define itself in opposition. You get counterpolarization, Waluigi versions of the AI views you wanted to promote, and institutions built by one administration that are gutted by the next. The goal instead should be enough cross-partisan buy-in so that the underlying institutional infrastructure of AI policy can be inherited and revised rather than torn down and rebuilt every four years.
AI safety as a political system property
Michael Nielsen has made a useful analogy to fire safety: we did not make the world safe from fire primarily by making matches “aligned.” We made the world safer through building codes, materials science, detection systems, insurance, and professionalized response.2 Something similar may be true for AI. Technical alignment matters enormously, but “AI safety” can’t only mean making individual models less likely to behave badly. It also means making the surrounding world more resilient to the powers those models might unlock.
So our political system will need to have answers to the broad series of bottlenecks that will determine whether society responds “well”: labor transition, state capacity, kids’ safety, accelerating beneficial deployment, biosecurity, cybersecurity, regulatory reform, market structure, and concentration of power.
Some of these are areas that Coefficient Giving hasn’t done as much work on in the past. But if you believe AI policy will come to define almost every policy area, if you care about navigating the transition to AGI safely, then to be a successful policy actor, you need to be in conversation with practitioners on a wide range of these issues.
Additionally, there is an O-ring phenomenon where failure on certain policy dimensions can undo progress on others. Voluntary coordination on safety between AI labs could get squashed because of the legal uncertainty around antitrust laws. A major scandal involving government use of AI could poison the well on augmenting state capacity for years. And the classified channels between labs and government that everyone agrees we’ll need are already closed to many of our best researchers, because we have no pathway to rapidly vet and clear the foreign-born talent that founded and staff these labs.
Broadening the aperture of AI policy doesn’t mean losing sight of the importance of preventing worst-case scenarios; instead, it’s a recognition that the same Congress that will pass laws on catastrophic risk will also pass laws on many other topics, and the institutions, expertise, trust, and coalitions needed to navigate transformative AI can be built (or squandered) in the policy windows that open along the way.
Abundance & AI
One of my favorite things about working in policy is seeing the way in which different policy issues can bleed into each other. The connections, tactics, and knowledge you’ve developed in one area can create positive, serendipitous spillovers for other policy areas you work on. The staffer you worked with on H-1B reform turns out a year later to be exactly who you needed to get on board for a key metascience policy win. The expertise you built up on infrastructure permitting ends up being critical to understand bottlenecks on AI data center development.
This seems to be especially true, in my view, for AI policy and the abundance agenda.
AI poses serious risks that we need to tackle as a society. But there is also absolutely gobsmacking upside if we can nail the execution. And a proactive, forward-looking story may, in fact, be politically necessary to assemble the coalition to build a constructive and durable policy framework. This means we need a policy agenda with concrete reforms that improve people’s lives in meaningful and legible ways. It’s not enough to wave our hands and say we need data centers to “cure cancer”.
And by default, many beneficial applications may never materialize or end up severely delayed because market incentives don’t adequately align developer interests. If we want more AlphaFolds and not just chatbots, we need to think seriously about market shaping: using tools like prizes, procurement, Advanced Market Commitments, public data sets, test beds, standards, and research funding mechanisms to steer market investments for the public interest.
Similarly, beneficial applications may end up getting bottlenecked by regulatory processes that were developed for a different era. If AI makes it trivially easy to generate new scientific hypotheses but can’t simulate all parts of complex human physiology, then we may put many more plausible drug candidates into the pipeline without fixing the binding constraint of sluggish and expensive clinical trials.
If AI makes software and analysis cheaper, physical infrastructure, energy, transmission, housing, and permitting may become more binding. Abundance efforts should ask: What becomes scarce in a world where intelligence gets cheaper? What bottlenecks become more salient? What public goods are underprovided by default?
The human qualities required
In 1955, John von Neumann published an essay asking whether we could survive our own accelerating technology, in his case about nuclear weapons and climate engineering. He considered and dismissed the obvious master plans. Banning these dangerous technologies wouldn’t work; the useful and the harmful were too intertwined. Abolishing war would be nice, but the resolve for peace after 1918 had lasted only twenty years. The only safeguards he could endorse were “day-to-day — or perhaps year-to-year — opportunistic measures, a long sequence of small, correct decisions.”
He went on to argue that new political forms and procedures would be necessary, that these transformations were not predictable in advance, and that most first guesses about them would be wrong. And he was correct! The decades that followed produced the International Atomic Energy Agency, test ban treaties, the Non-Proliferation Treaty, and the Moscow-Washington hotline. Some of these efforts overshot (we made nuclear energy almost unbuildable for a generation), some of them undershot (it took eighteen years after Hiroshima to ban atmospheric testing), and on at least two occasions the whole system came down to the judgment of a single Soviet officer.
The nuclear era had a quarter century to assemble its institutional architecture and we’ll likely have far less. Our moment calls for nothing less than a revitalization of American government and civil society on a radically compressed timeline. Each of the small, correct decisions ahead will look small only in the sweep of the full historical picture. Up close, many of them will require heroic levels of effort and coordination.
Coefficient Giving is scaling rapidly to meet the moment, against a backdrop of what Nan Ransohoff has called the third wave of American philanthropy, a wave large enough to potentially fund thousands of new projects and organizations. The binding constraint is unlikely to be funding, it will be people: grantmakers and policy entrepreneurs with the judgment and ambition to build the institutions we wish we had. If you share this vision, please apply! And if you are building something that we’ll need in the years ahead, reach out.
Here I’ll end as von Neumann did: “Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required: patience, flexibility, intelligence.”
For example, I’m excited about proposals that leverage the fact that AI chains of thought that can be audited in a zero-knowledge-proof way by oversight committees and inspectors general to still allow executive branch confidentiality and normal day-to-day operations while providing some sort of external check at scale.
Although we did also, of course, invent the safety match, partly in response to a rash of fires caused by the accidental lighting of early “strike anywhere” phosphorus matches.
Congrats Caleb! And great essay.
Congrats, and appreciated the nuanced articulation of the political economy of AI policy!
Also, great to see we’re both fans of Joseph Wright’s paintings: https://republicofscience.substack.com/p/science-as-collective-sensemaking