Move to LetsEncrypt (2017 edition)

[zerebubuth]

The *.openstreetmap.org certificate is due to expire on Tuesday, 2 May 2017. By this time, it seems that LetsEncrypt is supported by mainstream Java, and I'm not aware of any other platforms lacking support for it. I think the consensus is that we should move to it.

@tomhughes has already done some work prototyping this, and apparently @Firefishy has some thoughts on it too. Is there enough time to get this written, tested and rolled out with plenty of safety margin before the May deadline?

[tomhughes]

I was actually thinking it was March so I'm sure May should be ok ;-)

[tomhughes]

Here's the conversation @Firefishy and I had before Christmas:

02-12-2016 18:56:16 < Firefishy: StartSSL
02-12-2016 18:56:21 < Firefishy: we should switch ;-)
02-12-2016 18:56:39 < Firefishy: Expires Feb 2017
02-12-2016 18:57:20 > TomH: I have started working on it...
02-12-2016 18:57:55 < Firefishy: Cool. Be interested to hear how you intend to do it.
02-12-2016 18:58:04 < Firefishy: Or help if I can
02-12-2016 19:02:17 > TomH: Think I'm probably going to have each machine issue it's own cert
02-12-2016 19:03:32 < Firefishy: Own private key?
02-12-2016 19:08:23 < Firefishy: This looks interesting https://github.com/lukas2511/dehydrated
02-12-2016 19:09:37 > TomH: well certbot is perfectly good
02-12-2016 19:14:25 < Firefishy: Just thinking aloud... A shared private filesystem across all hosts, chef "discovers" instance without cert or needing a renewal... creates lock on shared fs, issues cert (hmmm how to route directly here multiple hosts)... saves cert to shared fs and next chef run on other hosts loads cert.
02-12-2016 19:14:52 < Firefishy: instance being a website.... on a host
02-12-2016 19:16:27 > TomH: so we just need a magic private filesystem shared everywhere?
02-12-2016 19:17:06 > TomH: and some wait to route the validation calls from the ACME server to the right place to get the validation tokens
02-12-2016 19:17:35 < Firefishy: or store the validation tokens on the shared filesystem with above tool.
02-12-2016 19:18:23 < Firefishy: https://community.letsencrypt.org/t/how-to-nginx-configuration-to-enable-acme-challenge-support-on-all-http-virtual-hosts/5622
02-12-2016 19:20:30 < Firefishy: I can sponsor a small cephfs cluster for a few months ;-)
02-12-2016 19:20:37 < lonvia: I've used https://github.com/hlandau/acme for waymarkedtrails. It's nicely flexible and painless.
02-12-2016 19:23:55 < Firefishy: At work we now no longer care about non-SNI enabled clients.
02-12-2016 19:24:23 < Firefishy: We wrote some non-SNI clients, but instead of hacking a solution, we replace the code with SNI enabled code.
02-12-2016 19:28:20 > TomH: seriously, why spend time building something so complex?
02-12-2016 19:31:20 > TomH: my plan is that a new version of apache_site can gather list of domains and then we build a script that runs certbot to issue a cert using webroot auth and a cronjob to do the renews
02-12-2016 19:33:07 > TomH: there are subtlies around making sure /.well-known/acme-challenge  is aliased to the place cerbot places the challengers
02-12-2016 19:33:39 > TomH: downside is have to have letsencrypt account token on each mahine (or let each machine create it's own account)
02-12-2016 19:34:11 > TomH: upside is not having to find a way to securely distribute keys and certs and to redirect challenges to the cert issuing machine
02-12-2016 19:35:48 < Firefishy: So for spike-01 to spike-05 there will be 5 different certs for same name?
02-12-2016 19:36:13 > TomH: yep
02-12-2016 19:36:36 < Firefishy: Shared filesystem wouldn't need redirecting challenges, just send them to shared fs.
02-12-2016 19:36:43 > TomH: that's the other downside, but it doesn't really matter
02-12-2016 19:36:51 > TomH: true, but we don't have one!
02-12-2016 19:37:08 < Firefishy: I can have us one by tomorrow ;-)
02-12-2016 19:37:20 < Firefishy: And then we build one ;-)
02-12-2016 19:37:55 > TomH: also how does the cert issuing machine know what certs to issue?
02-12-2016 19:38:09 > TomH: we can do it via search I guess but it adds extra delays
02-12-2016 19:38:11 < Firefishy: Yeah, that logic should move into chef
02-12-2016 19:38:33 > TomH: it means the cert won't issue immediately and you have to wait for a chef run on the other server
02-12-2016 19:39:04 > TomH: and you have an internet accessible distributed filesystem containing all your private keys, which is fecking scary
02-12-2016 19:39:38 < Firefishy: Sure, we'd need a "default" cert or a fallback for chef to use when there isn't an issued one.
02-12-2016 19:39:46 < Firefishy: sshfs? ;-)
02-12-2016 19:40:54 < Firefishy: distributed fs would likely need to only be available on internal lan / vpn.
02-12-2016 19:41:05 > TomH: well I'm not saying we wouldn't secure it somehow, just that you're only one failure away from disaster if there is some port that (with the right credentials) lets you access the keys
02-12-2016 19:41:15 > TomH: so now we have to get every machine on the vpn...
02-12-2016 19:44:08 < Firefishy: haha, ok. Let me think about it more.

[Firefishy]

Expires: Tue, 21 Feb 2017 00:49:58 UTC: *.osmfoundation.org osmfoundation.org openstreetmap.org blog.openstreetmap.org osm.org blog.osm.org blog.osmfoundation.org switch2osm.org stateofthemap.com opengeodata.org stateofthemap.org thinkup.openstreetmap.org thinkup.osm.org otrs.openstreetmap.org otrs.osm.org foundation.openstreetmap.org foundation.osm.org *.stateofthemap.com *.stateofthemap.org *.switch2osm.org switch2osm.com *.switch2osm.com openstreetmaps.org blog.openstreetmaps.org openstreetmap.com blog.openstreetmap.com *.opengeodata.org openstreetmap.net blog.openstreetmap.net

[tomhughes]

Yes that's the osmfoundation one and I guess it's that date I had in mind.

[tomhughes]

So to summarise the design I had been working on was basically that each machine would run certbot and get it's own certificate - for apache at least the apache_site resource would automatically arrange for an appropriate certificate to be requested.

The advantage of that design is that there's no need for machines to communicate what certificates they need to a central point, or to move the keys and certificates back to the machine that needs them.

There is an issue where multiple machines serve the same name however as letsencrypt might hit any of them for the validation token to whatever storage that is on needs to be visible to them all, which we don't currently have for the main web servers.

The other design (and what I use at home) is to have all machines redirect /.well-known/acme-challenge to a central machine that acts as the only machine to issue certificates. It then either has a fixed list of certificates to issue or we need some way for machines to tell it what they need. Plus you then need a way to get the resulting keys and certificates to the machines that need them.

[zerebubuth]

Would the "Chef Way" of distributing keys and certificates be via Chef's encrypted data bags? If so, is there any benefit to trying to use that mechanism to distribute them rather than a shared filesystem?

I agree that having only a central machine with the (plaintext) LetsEncrypt client tokens makes a lot of sense from the point of view of security, and all other things being equal, would prefer that.

[tomhughes]

Not really because a recipe can only read from a data bag - writing to it happens through the server API instead.

I guess you could have a script on the certificate master node that used knife to upload to a data bag after a certificate was obtained which might work.

Whether the data bag is encrypted is not really relevant, though it might be worth doing in this case in that it would control which nodes could retrieve the data.

[zerebubuth]

I guess you could have a script on the certificate master node that used knife to upload to a data bag after a certificate was obtained which might work.

Yes, that was what I was thinking: have the central .well-known/acme-challenge machine perform handle the LetsEncrypt protocol, make the private key and fetch the certs, then push them into Chef via the API (which I guess means knife?).

[tomhughes]

I have some sort of plan in mind now and am working on this, but one things we will need to decide is which machine we're going to use as the central host that generates and distributes the certificates...

[zerebubuth]

While I'd love to say "openshift", that's not ready yet. In the meantime, does it make sense to put it on ironbelly / grisu as a "central service"?

[tomhughes]

I agree that is kind of the obvious place, but it's also probably a bit more exposed (what with acting as an NFS server and things) then I would really like.

It may well be the best choice for now at least though.

[grinapo]

"Not to disturb the waters" (<- standard disclaimer), just mentioning that we have been suffering the startcom/wosign fuckup and had to move over to letsencrypt. My problem was that I had to make it automagic, but many servers doesn't use normal http (like email or xmpp) or tcp80 not available for the acme challenge, and a plenty of machines doesn't even have a public IP, so http-01-auth was out of question.

In the end I use certbot in manual mode with hooks, using dns-01 challenge by turning the knobs on powerdns using really simple scripts. The central keymanager host generates all the new keys and grab the certs, and pour them out using saltstack (which is the better variant on chef :-P) [or in some cases plain https which is alised by the IP to the specific certs of the requestor since some machines are behind various hideous stuffs like nats and fascist firewalls].

In your case this may not be interesting since you all have webservers and public IPs, but who knows what may help. At least certbot devs are nice fellas and quite responsive.

[tomhughes]

DNS challenges aren't going to work for us because we don't have real time control of our DNS records.

Also the certbot in Ubuntu 16.04 is quite old (like it's still called letsencrypt) and doesn't have hook support.

[grinapo]

Well it's python so could be even dropped into a user dir. I use the latest deb from debian/sid under debian/stable (apt pinned) since it doesn't pull in horribly much deps.

[tomhughes]

Out first two services (munin and piwik) are live with letsencrypt certificates. Now the real work of converting everything can start:

• blog.openstreetmap.org
• blogs.openstreetmap.org
• chef.openstreetmap.org
• dns.openstreetmap.org
• donate.openstreetmap.org
• forum.openstreetmap.org
• git.openstreetmap.org
• gps-tile.openstreetmap.org
• hardware.openstreetmap.org
• help.openstreetmap.org
• irc.openstreetmap.org
• lists.openstreetmap.org
• logstash.openstreetmap.org
• munin.openstreetmap.org
• nominatim.openstreetmap.org
• otrs.openstreetmap.org
• piwik.openstreetmap.org
• planet.openstreetmap.org
• stats.openstreetmap.org
• svn.openstreetmap.org
• taginfo.openstreetmap.org
• tile.openstreetmap.org
• trac.openstreetmap.org
• wiki.openstreetmap.org
• www.openstreetmap.org
• board.osmfoundation.org
• dwg.osmfoundation.org
• join.osmfoundation.org
• operations.osmfoundation.org
• wiki.osmfoundation.org
• 2007.stateofthemap.org
• 2008.stateofthemap.org
• 2009.stateofthemap.org
• 2010.stateofthemap.org
• 2011.stateofthemap.org
• 2012.stateofthemap.org
• stateofthemap.org
• switch2osm.org

[tomhughes]

Rollout has now been completed to all existing sites (and a few new ones).

[grinapo]

No action needed, just maybe you'll need such knowledge for future issues:
hotosm/osm-tasking-manager2#976