Why I haven't tried to forecast AI progress but will soon
Summary
I have dedicated a lot of time and energy to better understanding and forecasting global catastrophic risks, but have largely neglected AI risk. I explain my reasoning for this and how it’s changed over time. Then I lay out my strategy for how I plan to start working in this area, and list what I expect to be publicly publishing along the way.
Introduction
If you were to look at my previous attempts at forecasting Global Catastrophic Risks (GCRs), you’d notice a pattern in the questions on AI progress… I didn’t answer them. I deferred to the wisdom of the crowd as much as possible, with some slight tweaking here and there where resolution criteria between forecasts didn’t match. More generally, I haven’t been forecasting much at all lately, choosing instead to focus on ways that the art of judgmental forecasting could be improved, especially on the topic of GCRs.
I’ve been thinking a lot about these and related stances, and I think my positions have gradually shifted in a number of places. This post will be structured as a list of topics, each followed by what I used to believe and why and then why I’ve changed my mind (if I have). The end result is that I’ll be pivoting a significant amount of my attention to trying to forecast AI progress, and I’ll get into what I expect my efforts to look like towards the end of the post.
Reasons I Haven’t Forecasted AI Progress
Forecasting AI progress is hard / Our current tools are inadequate
What I believed: I’ve written before on why I think the current judgmental paradigm is especially weak on topics that require deep expertise and predictions on long time horizons. AI seems like the GCR where these problems are worst, and you can add to the mix that this deep expertise is evolving ever more rapidly and the current experts may be completely irrelevant to the ultimately important questions.
I flat out do not believe that the status quo tools of simply asking generalist forecasters to predict likelihoods of AI disaster in the future are remotely likely to produce useful things. They might point to interesting gaps in knowledge, but I don’t expect them to be accurate and I don’t expect their associated rationales to have much value at all relative to discussions between experts.
Of course, something being hard on its own isn’t reason to avoid it. I was framing this as a choice between spenging my time focusing on other GCRs that seemed easier to forecast (though still not remotely “easy”) and focusing on AI, which I strongly believe to be the most challenging of the set. Since I didn’t feel our tools were adequate for thinking about something like nuclear war, it made sense to try and build them up to that level of adequacy first before moving on to an even more ambitious target.
How I’ve changed my mind: There are experimental tools on the fringes of the forecasting world that seem viable but haven’t really been tried yet. I am highly confident would improve our ability to forecast topics like this (though combined they might still be inadequate!). Furthermore, due to the level of attention and investment dedicated to forecasting AI progress relative to other GCRs, there will likely be lower activation energy required to out new approaches in this domain. Maybe even in part because the inadequacy of our current toolkit will be most obvious here. I’m thinking of things like conditional trees or improved methods of collaboration between domain experts and forecasters.
I’ve spent most of my time with this blog so far on trying to discover which interventions would be most effective in improving our forecasting ability. I think I’ve found some, and I’m optimistic that if I kept digging I’d find more. Where I’m less optimistic is in finding opportunities to put them to the test from my current position in the world. I’m mostly shouting into the void, hoping to find the right collaborators to discuss these things with. There are very few organizations/individuals actively trying to improve the art of judgmental forecasting, and their attention is split in many directions. I’m not sure there is a path from where I am now to meaningfully collaborating with or joining them without attaching more impressive accolades to my history in this area. The large incentives and attention allocated to forecasting AI progress should make impressive performance in this area a stepping stone to positions where I can have more impact on the topics I think are more impactful.
To be clear, I am still going to actively and aggressively pursue collaborations on the ideas that I’ve shared here, but I’ve grown increasingly confident that currently my pace of being able to discover and understand interventions greatly outpaces my ability to work on implementing them, so it makes sense to reallocate a lot of that time and energy.
AI might not even be a GCR
What I believed: AI risk seems different in kind to the other GCRs I’m used to thinking about like pandemics or nuclear war or asteroids. I’m highly confident that without additional countermeasures each of those three would eventually kill lots of people if we hadn’t already been killed by something else. The almost guaranteed need for these countermeasures means that there’s a clear objective of figuring out the best ways to mitigate these GCRs and then get resources allocated to them. Judgmental forecasting can definitely help with this!
It seems totally plausible that AI/AGI ends up not being a GCR at all. It’s certainly not right now, in its current form. It’s profoundly challenging to predict the direction of technological progress or where it will end up. This adds an additional layer of uncertainty, as if we were trying to predict the risks and possible countermeasures of a global nuclear war before WWII, where all of this forecasting might end up being worthless.
I understand counterfactual impact, I promise. But again, my frame was trading off time and energy from focusing on other GCRs to focus on this one. AI has been getting much more attention in the Effective Altruism world AND that attention could turn out to be worthless. It felt obvious, for me personally, that the right response was to focus on the simultaneously “more neglected” and more likely to be useful topics with the one thing I was going to focus deeply on. I found myself excited about working on the rest of the GCRs, which made it much easier to invest a lot of time and energy.
How I’ve changed my mind: Ok, I still believe all of this. The other ways I’ve changed my mind have just tipped my internal scales far enough that it now seems worth forgoing the increased likelihood of impact from focusing on more traditional GCRs.
I’m unqualified to work on AI stuff
What I believed: I’ve worked in an engineering discipline and I’ve written some code. I’ve never implemented a neural net, used PyTorch, or been through a programming interview. AI as a GCR, more than all the rest, feels out of reach for generalists. Translating ideas and information into more generally accessible forms takes time, and the field seems to be advancing so rapidly that I’d expect this delay to be significant. Other GCRs have much better established knowledge bases that have many more paths inside from a generalist perspective.
How I’ve changed my mind: I still believe all of this, I just think it’s a bad reason not to dig deeper. My ignorance should be cause to study a topic, not to continue ignoring it. How can I actually know the limits of my understanding without trying?
Furthermore, it’s increasingly looking like the ability to quickly recognize and adapt to the existence of powerful AI tools is becoming a critical lever for capacity to do work in the world, analogous to the ability to write well or query search engines. I expect deepening my understanding of at least LLMs to better prepare me for life in general, not just forecasting. In practice, I’ve been increasingly spending time tracking the latest AI developments, now it’s just time to turn a forecasting lens on the topic.
Lots of other people are already working on AI risk
What I believed: Though globally meaningful work on AI risk seems obviously neglected, within the world of Effective Altruism and judgmental forecasting it seems to equally obviously be the number one priority. From what I can tell, almost axiomatically, lots of effective altruists are trying to maximize their impact and thus flock to the number one priority. This obviously reduces the impact any additional person should expect to have on the area relative to areas that very few individuals are focused on. Even just in terms of personal motivation I find it easier to work in areas that I feel are more neglected, independent of expected value.
How I’ve changed my mind: I should suck it up and do it anyway. The vast EA resources are dedicated to the problem because lots of EA individuals and organizations believe it warrants that level of attention. While I obviously shouldn’t defer my decision making to them completely, I should put some real weight on the wisdom of the crowd, just as I’ve done with my previous AI forecasts.
These resources also mean that the current “hot thing” is where to find connections and opportunities. If AI doesn’t turn out to be a GCR, I’ll still have made those gains. I expect to do more impactful work in a timeline where working on AI forecasting allows me to access more meaningful opportunities earlier vs. one where this work continues to be posted to an unread blog because I’ve continued to omit AI forecasting from my portfolio.
My Plans
So, I’ve decided I want to start directly forecasting again, and that I want to focus on AI progress. But there are presumably lots of ways to do that. I want to direct my efforts more specifically into areas within that topic that I expect to be most useful. Namely…
I want to compete in tournaments / forecasting competitions.
The allocation of resources to prize pools indicates that other people think these questions are important and likely to be useful. This seems like a natural way to chose initial subtopics to learn about in a field I know little about. I want to start forecasting and then learn in service of those forecasts vs. do a bunch of generalist research and then start forecasting.
These tend to have quite clear scoring that will give me rapid feedback on my performance and understanding. I want to fail fast, in environments where I can easily compare my performance to others.
Cash earnings seem like the most universally comprehensible credential of forecasting performance/impact, and this could open more doors for me later on. Also, money is useful.
These competitions may be scarce / infrequent, in which case I will try to self select AI related questions from open forecasting platforms that seem most important.
I want to write and openly publish rationales for my forecasts.
I refuse to just forecast a likelihood. If I participate in a competition where that is the answer format, I will also write rationales and publish them here as soon as I am able. I believe the arguments behind a forecast, especially in such a fast paced and convoluted topic like AI, represent most of the value of the forecast. They allow others to learn and critique and that’s where advancement is going to come from, not the immediate value of a number that you can aggregate.
Writing these will force me to capture my current state reasoning and understanding, which should accelerate my improvement by letting me look back and finds sources of errors. Scoring will likely be delayed relative to forecasting and these post-mortems will be critical to my learning.
I want to better understand the AI progress space, possible futures, and critical points in the near future.
If I just forecasted questions like “what is the chance of human extinction from AGI by 2100”, I think the gap between my current understanding and an informed prediction is so vast that even with a rationale I’d just output useless gobbledygook and doing this repeatedly would take forever to converge on meaningfully improved understanding.
Instead, I want to make sure I’m forecasting nearer term events that are objectively scorable and that I can start building into a sort of scaffold of understanding for how I expect things to play out and work, ala conditional trees. Ideally, competitions that I participate in will be structured to at least include these kinds of questions if not be exclusively focused on them.
In practice, my expected next steps are something like…
Keep tabs on the space of forecasting competitions, such that I don’t miss AI related ones.
Curate a list of questions I want to initially forecast.
Research the field in an attempt to build my understanding around those questions.
Write and publicly publish forecasts with rationales.
Expect this content to appear on this blog soon.