Recent gnupg keyserver DOS attack

Post your bugs here.
User avatar
ilu
Posts: 2344
Joined: 09 Oct 2013 12:45

Recent gnupg keyserver DOS attack

Postby ilu » 29 Jun 2019 20:38

https://gist.github.com/rjhansen/67ab92 ... 8d6955275f
https://gist.github.com/rjhansen/f716c3 ... 96e54e4b7e
I don't think you need to read the whole horrible story and the questionable moaning about this very foreseeable attack if you are not a frequent keyserver user. This is the main part:
Any certificate may be poisoned at any time, and is unlikely to be discovered until it breaks an OpenPGP installation.

The number one use of OpenPGP today is to verify downloaded packages for Linux-based operating systems, usually using a software tool called GnuPG. If someone were to poison a vendor's public certificate and upload it to the keyserver network, the next time a system administrator refreshed their keyring from the keyserver network the vendor's now-poisoned certificate would be downloaded. At that point upgrades become impossible because the authenticity of downloaded packages cannot be verified. Even downloading the vendor's certificate and re-importing it would be of no use, because GnuPG would choke trying to import the new certificate. It is not hard to imagine how motivated adversaries could employ this against a Linux-based computer network.
Mitigations

At present I (speaking only for myself) do not believe the global keyserver network is salvageable. High-risk users should stop using the keyserver network immediately.
Users who are confident editing their GnuPG configuration files should follow the following process:

(1) Open gpg.conf in a text editor. Ensure there is no line starting with keyserver. If there is, remove it.
(2) Open dirmngr.conf in a text editor. Add the line keyserver hkps://keys.openpgp.org to the end of it.

keys.openpgp.org is a new experimental keyserver which is not part of the keyserver network and has some features which make it resistant to this sort of attack. It is not a drop-in replacement: it has some limitations (for instance, its search functionality is sharply constrained). However, once you make this change you will be able to run gpg --refresh-keys with confidence.
Mitigation no. 1 is easy if I understood correctly, everybody should remove keyserver entries from their systems as quick as possible.. But where would be the second dirmngr.conf on our system and what happens without it?

Also be very careful when importing keys from other repositories. Check key size before importing them. Also remember that pip, npm, pear, composer, ... use keys as well.

This might get bad really quickly if someone else with more malicious intent picks this up.

User avatar
Arjen Balfoort
Site Admin
Posts: 9155
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Recent gnupg keyserver DOS attack

Postby Arjen Balfoort » 30 Jun 2019 08:44

That was an interesting read. Thanks.

I've removed the gnugpg configuration in solydxk-system, but I cannot do anything about the user's configuration. I could, however, blog on our main site to simply remove ~/.gnupg.

Those who use gpg would never consider removing that directory and for those people we could advise to remove the keyserver parameter from gpg.conf.

Is that okay with you?


SolydXK needs you!
Development | Testing | Translations

kurotsugi
Posts: 2194
Joined: 09 Jan 2014 00:17

Re: Recent gnupg keyserver DOS attack

Postby kurotsugi » 30 Jun 2019 10:44

creating a blog about how to mitigate the issue would be nice. we should leave the keyring mechanism left for the user since deleting it would broke things. that post tell us about how to mitigate the issue, not to solve the issue. at one point the issue will get resolved and user could activate the keyserver again.

anyway, this issue happened on server side so once someone attack the keyserver, none of our user would be able to upgrade their system. I still could upgrade my system so we can sure that our keys are safe, at least for now. we might also warn the user not to refresh their gpg cache until the issue got resolved. theorically reset the repo cache should solve the issue but I don't have any chance to try it

User avatar
ilu
Posts: 2344
Joined: 09 Oct 2013 12:45

Re: Recent gnupg keyserver DOS attack

Postby ilu » 30 Jun 2019 11:25

What's the defaults if one removes the conf? The first answer here https://unix.stackexchange.com/question ... -directory says ~/.gnupg will get recreated with defaults (but owned by root).
Also the dirmngr service is still loaded but inactive, I'm not exactly sure what that means.

Code: Select all

systemctl --user status dirmngr
● dirmngr.service - GnuPG network certificate management daemon
   Loaded: loaded (/usr/lib/systemd/user/dirmngr.service; static; vendor preset:
   Active: inactive (dead)
     Docs: man:dirmngr(8)
Edit: This was the wrong command, it should have been:

Code: Select all

systemctl --user status dirmngr.socket
● dirmngr.socket - GnuPG network certificate management daemon
   Loaded: loaded (/usr/lib/systemd/user/dirmngr.socket; disabled; vendor preset
   Active: active (listening) since Mon 2019-07-01 17:52:33 CEST; 2min 32s ago
     Docs: man:dirmngr(8)
   Listen: /run/user/1000/gnupg/S.dirmngr (Stream)
If you removed the skel conf then the latest solydx10 ISO has the default gnupg config and that includes a sks keyserver setting. I think this now needs to be preconfigured (no matter what Jonas said) to explicitly remove the keyserver from the config. Sometimes things change quickly ...

We should announce the recommended fix which is to modify gnupg.conf (and not remove it) as rjh and dgk say in their papers - as maintainers they should know.

I'm worried about repository keys that are not served via packages (as debian does) but get imported. Are keyservers ever used for updating them? Ubuntus keyserver runs with sks btw. All I can find is that the update needs to be triggered by a manual command so as long as you don't do this ... The poisoned keys were a lot bigger than 1MB. So if people check the size of a repo key manually before importing it they should be fine.

Edit: Since debian distributes its key as a package the debian key itself is safe. Hows the Solydxk key distributed?

kurotsugi
Posts: 2194
Joined: 09 Jan 2014 00:17

Re: Recent gnupg keyserver DOS attack

Postby kurotsugi » 01 Jul 2019 04:49

solydxk keys also provided as packages. debian and it's children by default using packages. the only exception would be ubuntu's PPA.

AFAIK dirmngr is dormant (i.e it's not used until user manually use it). it could be activated by systemd socket or similar mechanism. btw, if we decided to modify the conf we need to remember to revert the configuration once the problem is solved.

User avatar
grizzler
Posts: 2114
Joined: 04 Mar 2013 15:45
Location: The Hague, NL

Re: Recent gnupg keyserver DOS attack

Postby grizzler » 01 Jul 2019 10:59

Regarding dirmngr, you need to reload/restart it after making changes to gpg.conf and dirmngr.conf (just create it if it doesn't exist - it should be in the same directory as gpg.conf). Otherwise it won't pick up the changes.

As far as packaging is concerned, if you need to import a key that isn't packaged, never use the old method with apt-key. Just download the keyfile and put it in the /etc/apt/trusted.gpg.d directory. That way you can easily see how big it is and getting rid if a poisoned key is simply a matter of deleting the file.
Frank

SolydX EE 64 - tracking Debian Testing

User avatar
Arjen Balfoort
Site Admin
Posts: 9155
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Recent gnupg keyserver DOS attack

Postby Arjen Balfoort » 01 Jul 2019 13:45

What about this default gpg.conf in /etc/skel:

Code: Select all

# Set the remote key server
#keyserver hkps://keys.openpgp.org

# Use the GPG agent for key management and decryption
use-agent

# Specify the hash algorithms to be used for new keys as available
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

# Set the list of personal digest preferences. Use gpg2 --version to get a list of available algorithms.
personal-digest-preferences SHA512

# Use SHA512 as the hash when making key signatures
cert-digest-algo SHA512

# Set the default key for signing
#default-key  [keyname]

# Use the default key as default recipient if option --recipient is not used
#default-recipient-self



SolydXK needs you!
Development | Testing | Translations

User avatar
ilu
Posts: 2344
Joined: 09 Oct 2013 12:45

Re: Recent gnupg keyserver DOS attack

Postby ilu » 01 Jul 2019 15:50

grizzler wrote: As far as packaging is concerned, if you need to import a key that isn't packaged, never use the old method with apt-key. Just download the keyfile and put it in the /etc/apt/trusted.gpg.d directory. That way you can easily see how bit it is and getting rid if a poisoned key is simply a matter of deleting the file.
This sounds like very good advice which we need to communicate because every repo instruction uses the apt-key method.
grizzler wrote: Regarding dirmngr, you need to reload/restart it after making changes to gpg.conf and dirmngr.conf (just create it if it doesn't exist - it should be in the same directory as gpg.conf). Otherwise it won't pick up the changes.
Huh, the maintainer did not say anything about that. But maybe for people who use gpg the service runs regularly? Reboot is not enough?

Code: Select all

systemctl --user restart dirmngr.socket
systemctl --user status dirmngr.socket
● dirmngr.socket - GnuPG network certificate management daemon
   Loaded: loaded (/usr/lib/systemd/user/dirmngr.socket; disabled; vendor preset: enabled)
   Active: active (listening) since Mon 2019-07-01 17:52:33 CEST; 2min 32s ago
     Docs: man:dirmngr(8)
   Listen: /run/user/1000/gnupg/S.dirmngr (Stream)

Jul 01 17:52:33  systemd[2625]: Stopping GnuPG network certificate management daemon.
Jul 01 17:52:33  systemd[2625]: Listening on GnuPG network certificate management daemon.
Seems I went about it in the wrong way before. Is there a way to test whether it picked up the change?
kurotsugi wrote: if we decided to modify the conf we need to remember to revert the configuration once the problem is solved.
It's most probably not going to be solved with these tools. They'll come up with band-aids at best. The whole architecture is broken. And if somebody wanted to use gpg they would need to configure it themselves.

Pre-configuring the gnupg default conf by commenting out the keyserver as arjen did in his post seems best to me.

User avatar
grizzler
Posts: 2114
Joined: 04 Mar 2013 15:45
Location: The Hague, NL

Re: Recent gnupg keyserver DOS attack

Postby grizzler » 01 Jul 2019 17:25

ilu wrote:
01 Jul 2019 15:50
grizzler wrote: Regarding dirmngr, you need to reload/restart it after making changes to gpg.conf and dirmngr.conf (just create it if it doesn't exist - it should be in the same directory as gpg.conf). Otherwise it won't pick up the changes.
Huh, the maintainer did not say anything about that.
Maybe not, but I ran some tests yesterday and I noticed that simply changing the files wasn't enough. On my machine a gpgconf --kill dirmngr was required (a --reload didn't cut it, though in theory it should).
But maybe for people who use gpg the service runs regularly? Reboot is not enough?
Well, that's even more drastic, so I would assume that would work as well... :lol:
Seems I went about it in the wrong way before. Is there a way to test whether it picked up the change?
Do a gpg --search-keys for a known key and check the first line of the response (gpg: data source: ...)? That's how I found out the same keyserver was being used after the files had been changed.

Edit
This sounds like very good advice which we need to communicate because every repo instruction uses the apt-key method.
A couple of years ago I did a small project as a kind of "proof of concept" to replace apt-key with another tool: akm - a shell script which would only put keys in trusted.gpg.d. It worked, but getting it distributed seemed a bit of a hopeless task. Everyone uses apt-key and convincing people to change their ways on such a scale... Nope, sorry, too much to do as it is.
I just dug it up and added a check for file size (configurable). Maybe we could suggest our users to use that instead of apt-key (after some thorough testing by others than me, of course...).
Frank

SolydX EE 64 - tracking Debian Testing

User avatar
ilu
Posts: 2344
Joined: 09 Oct 2013 12:45

Re: Recent gnupg keyserver DOS attack

Postby ilu » 03 Jul 2019 10:28

grizzler: where is it? The poisoned keys have 45 MB. That's not easily missed.

arjen: We should put up a note saying:
1. Open /home/<username>/.gnupg.conf in a text editor. Ensure there is no line starting with keyserver. If there is, remove it or put a "#" before it. Save. Enter in terminal: gpgconf --kill dirmngr
2. If you need to import a repository key that isn't packaged, never use the old method with apt-key. Just download the keyfile and put it in the /etc/apt/trusted.gpg.d directory. That way you can easily see how big it is and getting rid if a poisoned key is simply a matter of deleting the file. Key files should be small, definitely smaller than 1 MB.

As an aside: Cleaning a key is possible if you know what you are doing: https://daniel-lange.com/archives/159-C ... g-key.html. Takes almost 2 hours though.

User avatar
Arjen Balfoort
Site Admin
Posts: 9155
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Recent gnupg keyserver DOS attack

Postby Arjen Balfoort » 04 Jul 2019 07:28

There must be something more to do.
If I kill dirmngr I get this:

Code: Select all

gpgconf --kill dirmngr
gpgconf: error running '/usr/bin/gpg-connect-agent': terminated
gpgconf: error running '/usr/bin/gpg-connect-agent KILLDIRMNGR': General error
But I have to do that each time I login.

[EDIT]
Forgot to mention: I also have to do this if I create an empty ~/.gnupg/dirmngr.conf


SolydXK needs you!
Development | Testing | Translations

User avatar
Arjen Balfoort
Site Admin
Posts: 9155
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Recent gnupg keyserver DOS attack

Postby Arjen Balfoort » 04 Jul 2019 10:41

This obviously has to do with gpg-agent but because I'm not running gpg-agent, I suppose I can ignore these messages.

Explicitly putting no-use-agent in gpg.conf does not prevent the message.


SolydXK needs you!
Development | Testing | Translations

User avatar
grizzler
Posts: 2114
Joined: 04 Mar 2013 15:45
Location: The Hague, NL

Re: Recent gnupg keyserver DOS attack

Postby grizzler » 04 Jul 2019 11:38

Curious.. I can't reproduce that here, with nor without a running gpg-agent.

Could be something that has been fixed over time. I remember seeing quite a number of gnupg related updates since stretch became stable.

ilu wrote:grizzler: where is it?
You mean the akm script? I'm working on some additional features. Should be done today or tomorrow. I'll post an url when it's ready.
Frank

SolydX EE 64 - tracking Debian Testing

User avatar
Arjen Balfoort
Site Admin
Posts: 9155
Joined: 26 Jan 2013 19:36
Location: Netherlands
Contact:

Re: Recent gnupg keyserver DOS attack

Postby Arjen Balfoort » 04 Jul 2019 12:08



SolydXK needs you!
Development | Testing | Translations

User avatar
grizzler
Posts: 2114
Joined: 04 Mar 2013 15:45
Location: The Hague, NL

Re: Recent gnupg keyserver DOS attack

Postby grizzler » 06 Jul 2019 14:09

grizzler wrote:
04 Jul 2019 11:38
ilu wrote:grizzler: where is it?
You mean the akm script? I'm working on some additional features. Should be done today or tomorrow. I'll post an url when it's ready.
Always takes longer than expected... :(

There's a tarball here: https://duinsoft.nl/temp/akm.tar.xz

Put it somewhere on your machine, open a terminal on that directory and run:

Code: Select all

d=~/.config/akm && mkdir -p $d && tar -xJf akm.tar.xz -C $d
to unpack and 'install'. At that point, the executable akm is in the config directory (there's a read-me and info on testing there as well). You can run it there or move it to ~/bin (if that exists - create it if necessary) or /usr/local/bin.
Frank

SolydX EE 64 - tracking Debian Testing

danielx
Posts: 10
Joined: 29 Aug 2018 15:51

Re: Recent gnupg keyserver DOS attack

Postby danielx » 08 Jul 2019 10:40

Arjen Balfoort wrote:
04 Jul 2019 12:08
Ok, if it's just me...
posted here: viewtopic.php?f=2&t=7573&p=70470#p70470
blogged here: https://solydxk.com/news/recent-gnupg-k ... dos-attack
The blog article doesn't specify if the fix has to be applied also for root (/root/.gnupg/.gpg.conf) or if the root has to be excluded.
I ask because I checked on different machine, but the directory /root/.gnupg/ doesn't exist and I have other machine to check yet.

User avatar
grizzler
Posts: 2114
Joined: 04 Mar 2013 15:45
Location: The Hague, NL

Re: Recent gnupg keyserver DOS attack

Postby grizzler » 08 Jul 2019 12:41

As far as I'm aware /root/.gnupg/ isn't supposed to exist. You should never run gnupg as root. Ever. You may have to be root (through sudo) to use apt-get update, but that doesn'ŧ use a gnupg set up in /root.
Frank

SolydX EE 64 - tracking Debian Testing

User avatar
ilu
Posts: 2344
Joined: 09 Oct 2013 12:45

Re: Recent gnupg keyserver DOS attack

Postby ilu » 08 Jul 2019 12:45

You should not have a /root/.gnupg/gpg.conf.

User avatar
grizzler
Posts: 2114
Joined: 04 Mar 2013 15:45
Location: The Hague, NL

Re: Recent gnupg keyserver DOS attack

Postby grizzler » 09 Jul 2019 16:53

Frank

SolydX EE 64 - tracking Debian Testing


Return to “Bug Control”

Who is online

Users browsing this forum: No registered users and 3 guests