Efail: A Postmortem
I’m writing only for myself: not for GnuPG, not for Enigmail, and absolutely not for my employer.
Less than a week ago, some researchers in Europe published a paper with the bombshell title “Efail: Breaking S/MIME and OpenPGP Email Encryption using Exfiltration Channels.” There were a lot of researchers on that team but in the hours after release Sebastian Schinzel took the point on Twitter for the group.
Oh, my, did the email crypto world blow up. The following are some thoughts that have benefited from a few days for things to settle.
Culture in Crisis
The vast majority of the OpenPGP community were champs. … You guys are awesome. I love you. I want to buy each and every one of you a beer.
They say that when there’s a fire in a nightclub you’re at more risk of dying from the stampede than the blaze. The panic kills both by crushing people underfoot, and by clogging the exits so that people have to stay in the club longer and breathe more hot smoke-filled air. The fire is a problem but the panic is worse. That’s what we saw here, and frankly I place a lot of blame for that at the feet of the Electronic Frontier Foundation.
I’ve sent a few very angry emails to Danny O’Brien taking his outfit to task for what I see as reckless conduct, and he’s been a gentleman in listening to them. That’s not to say he agrees with all of my criticisms, of course, but he gave me a fair hearing.
I’m sure the EFF will soon be posting their own post-mortem. I look forward to seeing what they think of how they handled things in benefit of hindsight.
I think they acted with the best of intentions. In my experience, most disasters begin with well-meaning people sincerely trying to do the right thing.
The same cannot be said of the trolls who sprang up in the aftermath. Danny says he’s had people accusing him of working to advance the CIA’s agenda. I believe him. The accusations thrown against Sebastian and his research group have been flat-out horrifying. Some of the things said about him, his research group, and his paper, left me feeling a little bit better about the kind of trolls I’ve had to deal with. I mean, mine have just been screaming at me that I’m a liar.
The vast majority of the OpenPGP community were champs. When this panic struck most of you hunkered down, waited for the storm to pass and the floodwaters to recede, and trusted that when it was all over things would be okay. You guys are awesome. I love you. I want to buy each and every one of you a beer. If you flag me down at a conference and say, “hey, Rob, I kept my head during Efail,” I’ll make good on it, too.
But the trolls, well. Normally trolls are the price we pay for vigorous public discourse. But when there’s a mass panic situation, trolls become harmful to everyone’s well-being. They encourage us to behave tribally and to make war on the other tribes. That’s a disaster for everyone.
It’s normal for tempers to flare in a crisis. But it’s at times like these that you have to decide whether you’re committed to working together to restore normalcy, or whether it’s more important to apportion blame and make those jerks pay. I encourage you to choose the former, and to remember those whom you saw choose the latter. You can count on the former in a crisis. Counting on the latter is taking your life in your hands.
Neither Fish Nor Fowl
OpenPGP is a victim of its own success — its own insane, deranged, unusual, off-the-charts, stellar success. There have been a lot of cryptographers with Ph.Ds after their names who have opined that OpenPGP is old, that its crypto is baroque, that it’s too hard to use, that it’s a clumsy fit for the problem space, and more. Most of it is kind of true. Almost all of it misses the point.
Recently I placed an order for a new laptop. It comes with a flavor of Debian installed on it. I’m benefiting from OpenPGP already and my laptop hasn’t even shipped yet. Every Debian package has an OpenPGP signature on it, and thus OpenPGP is making sure my new laptop isn’t pre-rooted with malware.
It’s kind of amazing if you think about it: OpenPGP’s number one use today, far and away, is to verify the authenticity of downloaded operating system packages. In our cloud present where we spin up new Amazon instances with the click of a button, OpenPGP is being continuously used to secure our software supply chains. For each OpenPGP-encrypted email I receive I probably update three or four dozen packages across several different systems. I’d guess that for me, an OpenPGP power user, under five percent of my OpenPGP usage relates to email.
When I see someone criticize OpenPGP as being best abandoned, I always check to see if they’re paying attention to all the strange little niches where OpenPGP is working invisibly behind the scenes and doing a surprisingly good job of things. Almost always they’re focusing myopically on email and only email. That tells me they haven’t thought about it deeply enough to have an informed opinion.
OpenPGP is a victim of its own success — its own insane, deranged, unusual, off-the-charts, stellar success.
If a time tourist visited the earth at the dawn of humanity it would be a pretty good bet the primates would become the evolutionary winners. But those strange mostly-hairless plains apes, who were far weaker than their chimpanzee relatives, less hardy than their mountain gorilla relatives, more homicidal than their bonobo relatives, whose only real advantages lay in the opposable thumb and a throat capable of making a variety of noises? Nah, those guys are toast. If that tourist jumped forward to 1969 to see Armstrong walk on the moon, our tourist would fall over with shock at seeing just how far opposable thumbs and language skills can take you — and how. (“Standing on top of the world’s biggest firecracker and daring it to blow you up is a shoddy way of going to the moon!” they’d say. Guilty. But it worked…)
Likewise, OpenPGP’s great success is its adaptability. Neither fish nor fowl, it somehow manages to swim just well enough to not drown and fly just well enough to not fall to its death. It’s got opposable thumbs and language skills and it’s not asking for your permission to succeed. It’s not just going places, it’s already gone places. Most of us just haven’t noticed.
The Big Problem With OpenPGP
Unfortunately it’s not all rosy. Criticisms about OpenPGP not really keeping up with the times are well-founded. The basic ideas behind OpenPGP haven’t changed all that much in a quarter-century. When I first started working with PGP in 1992 it was the cutting edge of design. It’s not any more. OpenPGP has problems, critical ones, ones that desperately need to be addressed. Unfortunately for a protocol that’s so important to the internet as a whole, hardly anybody seems to give a damn.
Oh, sure, every pundit has an idea. “If you just…” They always manage to not only know what the silver bullet is, but also fit it into a 288-character tweet. I’m quite fond of the observation that ‘just’ is a four-letter word. (I first heard this from Dan Kaminsky, but I don’t know where he heard it from.) Nothing is ‘just.’ Even the small things involve hard-fought battles.
Last year GnuPG tried to finally drop support for decrypting PGP 2.6 messages. Keep in mind PGP 2.6 is over a quarter-century old and isn’t even covered by the OpenPGP spec. It’s not even in GnuPG’s mission (“to provide the best free implementations of OpenPGP and S/MIME”) to read PGP 2.6 messages. You’d think we could drop that and reduce our codebase by a bit, right?
OpenPGP has problems, critical ones, ones that desperately need to be addressed.
No — the outrage from users was so vast that GnuPG had to cancel those plans.
If we can’t even drop support for PGP 2.6, just imagine what would happen if we dropped support for late-1990s OpenPGP traffic. That’s what people are implicitly telling us to do when they say, “you guys just need to refuse to decrypt non-MDCed traffic.” Can’t do it: our users will kill us. We’d like to, but we like not swinging from lampposts more. The 1% of GnuPG users who actually use it for email have some extremely vocal opinions about breaking backwards compatibility.
(Recently, Enigmail decided to make the lack of an MDC a hard fail. In a couple of hours the lead developer had five users complaining about the breakage.)
Quite often the pundit muses, “well, sometimes you have to do that for the good of the users.” At this point I generally smile and nod and get out of the conversation, because clearly this person doesn’t have much experience dealing with user revolts.
And yet, this isn’t really a problem at all. The real problem is the apathy of our users. And there, Gentle User, I have to respectfully suggest that there is something very important you need to be doing: you need to be getting active.
OpenPGP is a standard set forth by the Internet Engineering Task Force. You can get a copy for free with just the click of a mouse. The latest update was 2007. The spec is so old that one of its authors has since died (Hal Finney, who did legendary service to the community). The IETF’s OpenPGP Working Group, which we just call “the WG,” recently tried to overhaul the spec. The overhaul failed and the WG was shuttered due to the lack of community involvement.
It’s not bad crypto that’s killing OpenPGP. It’s apathy.
My May 14 tweeting of GnuPG’s official statement has been liked 332 times and retweeted 302 times as of this writing. That’s 634 reactions total. If just one tenth of those people had sent just one email each to the WG saying, “OpenPGP is important and I’m following this with interest, I don’t have the technical chops to contribute but it’s very important to me your work continue,” the WG would have never closed. Werner Koch has drafted an updated spec that has a lot of excellent new developments, but it’s not an official standard and—for the foreseeable future—won’t be due to the lack of community involvement.
It’s not bad crypto that’s killing OpenPGP. It’s apathy. Apathy is what makes us so beholden to backwards compatibility: because we lack a mandate from the users to make breaking changes. Apathy is what makes it impossible to modernize the spec: because there’s so little interest we can’t justify keeping the WG alive.
OpenPGP’s enemy isn’t bad crypto or aging protocols or anything else the pundits are saying. It’s apathy.
Get involved and give a damn.
But apathy can be fought. You can send an email to GnuPG-Users saying, “Break backwards compatibility already: it’s time. Ignore the haters. I trust you.” You can express interest in the WG and say it’s important to the future of the internet that OpenPGP be updated and backwards compatibility broken. You can ask your email plugin vendor, “Do you communicate directly with whoever does your OpenPGP implementation? Have they explained to you all of the details about how to correctly use it? Are you getting the answers you need?”
The bad news is that apathy is a very tough enemy to fight. The good news is we know exactly how to fight it: we need to get involved and give a damn.
The Balkanization of OpenPGP
As a consequence of OpenPGP being a package verification tool that started life as an email security tool and is now being repurposed back into being an email security tool, well, it’s not a terribly good fit for email any more. (And as a package verification tool it has its own problems, but that’s a different rant.) How many desktop email clients today have native support for OpenPGP? I don’t mean they call out to GnuPG, but I mean they actually have their own OpenPGP codebase: how many?
As near as I can tell, it’s zero. (Some webmail clients are fielding their own, usually based on OpenPGP.js.) S/MIME support is everywhere but desktop OpenPGP is the neglected stepchild. To the extent we have OpenPGP support in email clients it’s overwhelmingly third-party plugins, some of which are questionably supported, calling out to GnuPG. That means there are three complex pieces of software that all have to interoperate perfectly, despite the fact some of these teams may not even be aware of the existence of the others.
S/MIME support is everywhere but OpenPGP is the neglected stepchild.
This is not a recipe for reliable, high-quality software. This is a recipe for fragile pipelines that break easily and leave different teams saying, “well, we did our part correctly…”. (Which also ties into the OpenPGP community’s fear of breaking changes, as mentioned above. The moment GnuPG changes the way it does things, a dozen important data pipelines in obscure places break and the GnuPG community has no knowledge of it until the “WHAT DID YOU LUNATICS DO?” emails start pouring in.)
Efail
The Efail paper presents two attacks against OpenPGP and two on S/MIME. (The OpenPGP and S/MIME attacks on MIME parsing vulnerabilities are substantially similar.) With respect to the OpenPGP attacks, in neither case is OpenPGP being targeted directly. It’s email clients choosing to disregard warnings, or recklessly stitch together MIME messages, which is the problem.
I know sysadmins who are using OpenPGP to encrypt backups measuring dozens of petabytes.
(Some cryptographers have taken GnuPG to task for its streaming API, where GnuPG doesn’t return a success or failure flag until after it’s returned plaintext. They say GnuPG should buffer the entire email in memory and, in case of failure, return nothing. They say that by returning plaintext even if there’s a problem, it makes it too easy for email clients to do the wrong thing. The streaming API is, in their view, a misfeature. They’re right if one considers OpenPGP to be an email protocol: but as I’ve already mentioned, it’s not an email protocol any more. I know sysadmins who are using GnuPG to encrypt backups measuring dozens of petabytes. When you’re encrypting a petroleum company’s worldwide geological surveys, the data is too large to fit in memory. The streaming API exists to support things other than email crypto. It’s not a terribly good fit to the email space. Email clients use it anyway, some for good reasons, and some for less-than-good reasons.)
I have real problems with the title Sebastian’s group chose for their paper. They promised attacks on the OpenPGP protocol and I don’t think they delivered. Exploiting non-MDCed OpenPGP messages isn’t novel, interesting, or even much of an attack surface, given that every sane OpenPGP client uses MDCs by default and gives loud warnings (or fails outright) on MDC errors.
But there are two things that are getting overlooked in all the criticism of Efail. One of them is pretty darn important, and the other is near-apocalyptic in importance.
Ten of twenty-eight OpenPGP clients tested had at least some susceptibility to Efail. That’s jaw-dropping.
First, a surprising number of OpenPGP plugins ignore clear warnings from their OpenPGP engines. Ten of twenty-eight clients tested had at least some susceptibility to Efail. That’s jaw-dropping. I expected some clients were poorly written, but I wasn’t expecting almost a third! Further, the plugin I most expected to be immune (Enigmail) turned out to be one of them. This strongly suggests the GnuPG pipeline (Thunderbird to Enigmail to GnuPG and back again) is even more fragile than we thought. That’s a really important result. If you’re not shocked and appalled by this, you’re not paying attention.
Second, S/MIME has been hit right at the waterline. The Efail paper is almost a game-over on S/MIME, and that’s huge given how common S/MIME is in the workplace. We knew these S/MIME attacks were possible, but to my knowledge this is the first really comprehensive look at how to implement them in real-world attacks. I doubt this is the end of the road for S/MIME, but it’s an open question as to whether it should be. Contrary to first impressions there are some mitigations — particularly a supplementary RFC by Peter Gutmann that helps address the Efail attacks — but I don’t know if the various S/MIME vendors will all implement it. This is the sort of thing where a partial deployment is inadequate.
I have grave reservations about the hype of Efail, and I don’t think their attack on OpenPGP’s CFB mode merited inclusion in the paper. Those two things I’ll go to the grave believing. But their S/MIME work and their work discovering the sheer shoddiness of the email client ecosystem?
Those were important things that needed to be published, and my hat is off to them for doing it!
Lessons Learned
The first (and maybe most important) lesson is the importance of remembering we’re on the same side. There were a lot of people behaving badly. We need to be quick to call out bad behavior, especially when it’s done by people who claim to be on our side. Panic and terror is a bigger problem than the actual vulnerability.
The second is we could stand to have a renewed appreciation for OpenPGP’s importance to not just email crypto, but the global economy. GnuPG in particular is too critical to the information economy to be allowed to fail, and yet it’s under-resourced to a shocking degree. We need to either start prioritizing and resourcing it correctly, or replace it outright. The former will be much easier than the latter.
Remember: we’re on the same side.
The third is the importance of responsible disclosure. I don’t think the Efail authors did a particularly good job of it. After showing the GnuPG team a preprint which only addressed the weak and uninteresting CFB attack, the Efail authors then substantially extended their paper with some surprising and deeply interesting results on S/MIME and the email client landscape. I think they should have come back to GnuPG with the revised paper, because GnuPG is — in addition to being an OpenPGP client — also an S/MIME client. The OpenPGP side of GnuPG wasn’t impacted, but the S/MIME side potentially took a hammer blow. The GnuPG team deserved to have the Efail authors tell us what was coming. They didn’t. They say their November disclosure to the GnuPG team satisfied the gentlemen’s agreement of disclosure: I disagree.
That said, I find it hard to blame the Efail authors much. I referred earlier to the OpenPGP balkanization problem, where the ecosystem is made up of so many different parts that are only loosely coordinated. The Efail authors examined twenty-eight OpenPGP-aware email clients (and even more S/MIME clients). Suddenly they needed to coordinate with twenty-eight different groups. If you were to ask me to coordinate with twenty-eight different groups I’d screw up, too. That’s a highly nontrivial outreach task, and it’s unreasonable to expect the Efail authors to have done it perfectly.
Which leads us to the obvious next subject:
How Can We Improve?
I think a major cause of this bungled release was the lack of coordination between the different players in the OpenPGP space. KMail, Enigmail, GPGTools, GnuPG, and more, each needed to be informed. Often, people made unfounded assumptions about who was telling whom what. I’ve seen people argue that GnuPG was informed because Enigmail was informed, and I’m part of Enigmail, and ergo GnuPG was informed through me. It’s neat logic, but it assumes that I ever noticed the Efail entry in our bug tracker. (I didn’t. Neither did Olav, nor Dan.)
Coordination is important. So, to that end, here’s a suggestion for how to improve in the future. I propose that a trusted community participant — someone who’s in the OpenPGP world, but has no stake in any vendor — take the responsibility for chairing a Disclosure Group. The Group would have a private mailing list which vendors could join, assuming they passed whatever membership requirements the Group set forth. (“You must have actually shipped code” being an obvious first requirement.) Vendors who leaked emails from the mailing list would be ejected from it. From what I’ve heard, Openwall has some experience with this sort of disclosure group: we might be able to lean on them for some important lessons.
So what about it, Danny? Think it over. Let us know.
A lot of this current drama could have been avoided if instead of trying to coordinate with twenty-eight groups the Efail researchers could’ve sent out a single email. “A week from today we’ll be releasing a paper detailing major attacks against a large portion of the OpenPGP community. You’ll find a high-level description of the attacks attached to this email. If you need the complete preprint, talk to us. There’s going to be a lot of media attention, so be ready to explain to your users if they’re vulnerable and how to mitigate.”
It isn’t the fault of the Efail authors they failed to coordinate well with twenty-eight groups. It’s the fault of the OpenPGP community for making it so hard on them. This is a problem we can fix. Let’s fix it.
Despite my anger at the EFF and how they handled this disclosure, I think they’re the ones best-equipped to run an OpenPGP Disclosure Group.
So what about it, Danny? Think it over. Let us know.
And if you think I’m on to something here, tell the EFF. Email them, tweet at them, whatever, but tell them. Remember, apathy is killing OpenPGP!
In closing
Thanks for reading this. I know it’s long. Thanks for staying with OpenPGP for so long. Thanks for caring.
Never forget that we’re on the same side, and the people on the other side include tyrants and murderers. Let’s stick together.
[This essay has had minor edits for clarity and readability. There have been no substantive changes to the thinking or the opinions.]