Encrypted applications as a leverage point
I often think in terms of ‘leverage points’ — places where one can push a little on one end, and get a huge movement on the other, like the famous Einstein-Szilard letter (or the similar-but-much-less-famous Frisch-Peierls letter). One possible leverage point I’ve seen come up, especially for those with technical/math backgrounds, is developing better encrypted applications. The consequences of bad computer security are already huge — most likely including the election of Donald Trump as the US’s president — and will only get larger over time, even before strong AI is built. It’s only a few years ago that we got widely available secure messaging software (Signal/WhatsApp), and there are still many missing pieces, most notably the lack of an equivalent to the Google/Microsoft office suite.
There is, of course, already a computer security industry, but it suffers from many bad incentives: It’s extremely hard for a non-expert to tell whether a product is secure (see eg.
Schneier on snake-oil cryptography ). You don’t know for sure until you’re compromised, and many compromises are discovered years later, if ever. In 2013, even Edward Snowden used CryptoCat to communicate with journalists, about the most sensitive use case there is. The next month, DecryptoCat was published, a tool allowing anyone to decrypt CryptoCat group conversations at will. Oops.
The benefits of security are uncertain, confusing, and far into the future; the costs of security are certain, clear, and immediate.
In many cases, there’s a principal/agent problem, where most of the cost of a breach is imposed on others, not on the people who did a poor job at security. Eg., Target paid $18.5 million for its 2013 security failure, which despite being the “largest settlement ever”, is only 45 cents for each person affected. It’s hard to know what the full damage amount was, but the vast majority almost certainly fell on random customers, not on Target.
Government regulations which try to impose security misunderstand it, and therefore tend to be ineffective. One software manager, at a company in a highly-regulated industry, once told me flat-out that it didn’t matter whether their software was secure or not. All the regulator cared about was the set of procedures in place for handling a breach, not whether the breach happened. Another told me that their company was proven to be secure, because their CPA did a computer security audit every quarter. They showed me an impressive-sounding list of checks performed, such as when the CPA proved that their data was only transferred over SSH by confirming that he could log in on a terminal via SSH. I wish I were making this up.
Because of all these factors, the commercial market for software which is actually secure (not fake-secure or passes-CPA-security-audit) is tiny. Security usually winds up getting bolted on to other products — products the customer chooses to buy or not based on other features, not on how secure they are. This is why Facebook Messenger only added end-to-end encryption last year, and even then their implementation had problems. While this is terrible, the flip side is that there should be much less competition for making better security than for making better image-sharing sites or video games.
Making secure software is difficult, of course, but I think it’s a different kind of difficulty than big projects normally face. A project to build a new subway, for example, is hard because it needs lots of workers, heavy machinery, and government permission. But the group behind Signal, widely considered the gold standard in secure chat, is a non-profit with something like half a dozen employees. This is a strong indication that other groups of this form will have a comparative advantage here. The difficulty comes, AFAICT, mostly in having individual people who are very skilled, not in having large numbers of people or vast resources. Therefore, for the right type of person, it seems like the only barrier is in spending human capital on those skills, rather than on something like fame or prestige or corporate rank. Not being a security professional myself, I can’t say how to become one, but if anyone is interested in this kind of thing, I can highly recommend the Cryptopals challenges as an easy introduction to the area.