OpenAI’s GPT-2 disclosure as inverted responsible-disclosure
In February 2019, OpenAI announced its GPT-2 language model but, citing risks of misuse (mass-generated disinformation, fake reviews, phishing), declined to release the largest version’s weights right away — a then-unusual departure from ML’s open-publication norms. By September a separate group had reproduced a model of similar size on a single GPU, intensifying debate over whether OpenAI’s strategy had accomplished anything. This is a reply to programmer-blogger Jeff Kaufman, who had defended OpenAI’s approach.
OpenAI’s strategy was unlikely to improve the overall outcome, especially given the ambiguity around the threat model, which entity or entities would be responsible for mitigation, and what exactly their response might be. In computer security, when you discover a new category of potential vulnerabilities (say, Heartbleed ), the standard procedure is something like: Create a working implementation of the vulnerability (eg., a dummy system that can be “hacked into”), to confirm that it’s real, and scope out what exactly it enables attackers to do.
Try to identify all the systems/software which might be vulnerable, insofar as you reasonably can.
Privately contact the software vendors, CPU makers, or whoever else can mitigate the vulnerability. Tell them that their systems can be hacked, and that they need to patch it or otherwise take action.
Once they have fixed it, or after a reasonable deadline (in case they drag their feet excessively), then tell the public about the problem and how they should fix their devices and protect themselves.
If we consider GPT-2 dangerous, we could think about it as a class of vulnerabilities — say, it might email people in an automated way, impersonate their bank convincingly, and convince them to give up their bank passwords. Compared to the standard “responsible disclosure” model, there are a bunch of issues with the OpenAI model of skipping to step #4: Widely publicizing the fact that something is secret, without the secret itself, has the well-known effect of generating interest and making it seem more important (see “Scarcity” on LessWrong ). You’d expect this to accelerate the amount of work done on GPT-2-like systems, including attracting a larger number of bad actors, increasing the number of ways bad actors could exploit it, and helping bad actors to make discoveries more quickly. Note that it’s usually easy to make technical schematics “public” (in a less-known academic journal, say) without a large audience being immediately aware of it, while the OpenAI approach seems designed for maximum awareness by everyone, including bad guys.
Announcing it while threats are still vague distributes the responsibility for identifying mitigation needs from the discoverer to each mitigator, separately and individually, which reduces the chances of fixing it because it requires so much more initiative. Suppose, for example, that it had been important to identify and block GPT-2-generated text message conversations. It would have been straightforward to list and then privately contact every relevant vendor (Google, Facebook, phone companies, etc.), demonstrate that this was a problem, and try to fix it with them. Instead, making the announcement to everyone first requires each of these orgs to separately realize that this is a problem, decide what to do, and assign someone mitigation responsibility. (I haven’t looked super-hard, but to my knowledge, nothing like that has happened.)
Trying to use the technical difficulty of model training as a substitute for disclosure lead time probably won’t affect bad guys with significant budgets, most notably state actors. (How long would it have taken the Chinese government to replicate, or even just to hack OpenAI’s laptops and steal the model directly, now that it’s been highlighted as a nice juicy target?) It’s also extremely unpredictable, since it depends on how many people try to work on making it cheaper and more effective, and how fast they can go.
The best argument I’ve heard for the OpenAI model is that doing it so publicly made every AI researcher aware of the possibility of non-disclosure, which is valuable. However, I’m pretty sure this is outweighed by the likelihood of other teams imitating OpenAI and making counterproductive, widely-publicized non-disclosures. This problem is bad enough that I’d be tempted to try to use it in reverse , and make safety-enhancing technologies that everyone should use secret (but widely publicized as secret), so that more people will actually adopt them (see “To Spread Science, Keep It Secret” ).