Options for personal/private SSL certs?

[paulca]

I tried let's encrypt for SSL certs and it works well.

The trouble is, when you are playing around with containerisation and splitting up all your "servers" into a dozen "services" with their own IP and hostname, configuring each and everyone of them with a custom lets encrypt cert is tedious.

That is not the battle.  The battle is that they only last 90 days. While umpteen ways exist to "auto-renew" etc...  they don't scale well to dozens of hosts.  Most don't even support the preferred alterative ... wildcard certs.

I have thus tried just signing my own certs and installing my own ca_root cert into the OS, which I can automate and template.  This though fails due to pretty recent changes in how browsers handle ca_root certs and intermediaries.  They ignore the OS and go straight to an external service.  You have to add your certificate at the application level to each and every browser and IDE which does this.  Getting a "self signed" cert to be accepted the whole way through a CI/CD build chain in docker is a f___king nightmare.

I looked up how much getting my own "intermediary" for my domain would cost and ... well... I think it's one of those, "If you have to ask, you can't afford it." things.

Next down the list then is a long lifed wildcard cert for my "lan" domain.  $66 a year!  They say.  Although, if you want a 5 year cert, you have to pay, up front for all 5 years.  That was $450 or $225 on sale!

That would only solve half the problem, although the main ones of "no public trust anchor" and "short renewals" are removed.  The installing the same certs across many hosts is more practically automated if the cert is not going to disappear in 90 days.

Has anyone been down this road outside of a large organisation on how they manage this?

In my day job, while this causes many, many long delays, SSL certs are usually handled by a support service via request (a CSR file + business justification) and approval.  CA certs are deliberately internal and self signed, but company managed.

Is $225 worth it to have 5 years of 'hassle free' reusable certs... and not be harassed with browser warnings and API access errors due to invalid SSL certs?

[berke]

Sounds like a steal, the Gandi entry-level wildcard cert is at $200.  As much as I hate this SSL scam I'd go for it if I were running dozens of hosts.
That being said, why do you say LE auto-renewals don't scale well?  You just need a cron to run their stupid command once in a while on a live server.

[audiotubes]

I created my own Root CA for this and all I had to do was to add my CA's cert as a Root CA in various browsers and the SSL chain for the box.

I make CSRs and sign them with the Root CA's cert, so no self-signed certs and I don't depend on an external Root. I also used stronger keys than you normally get from commercial root CAs and set a longer expiration. So far it works perfectly fine for the few webservers I have, email, etc.

If you did that and it doesn't work, it could simply be that you didn't add your Root CA's cert to the correct part of the browser key store. There are multiple contexts where you can add it on Windows, and in Firefox on various platforms.

[JohanH]

I'm in a large organization that has it's own root certificate in all company computers. AD, internal web servers and services etc. As long as you use internal DNS servers, this certificate works like an external certificate. But we still have to force this root certificate for our own services that we provide locally. This is mostly for java and similar client applications, because they are so stupid so they can't find certificates from the computer certificate store. So either the end user (mostly devs) has to copy or do tricks with certificates (npm/nuget, such stuff) or we have to create custom installations and bundle the root certificate with the app. It doesn't matter in this case if it's a self signed or a commercial certificate, we would have to do the same anyway.

Privately I set up Lets Encrypt on a server a few years ago. Never touched it since, it just works.

[Veteran68]

Yeah I'm not sure I see the issues with scaling LE.

I work in IT for a large organization as well, and our network team has largely moved from managing commercial SSL certs for internal servers to using LetsEncrypt with automation. We have literally thousands of servers. We have a "real" wildcard cert but limit its use, for security and recovery reasons, i.e. compromise would require touching every server, versus a server-specific cert.

I also use LetsEncrypt (using the certbot toolchain) on my private LAN, mostly with a wildcard cert. If wildcard doesn't work for something, it's pretty trivial to add additional domains to the renewal script -- I did this initially until I got the wildcard authentication against my private DNS server working.

Then I have certbot renewal hooks that fire off after a renewal to push the new cert(s) to all of my other servers, restart the necessary service daemons, etc. When I add a new server, it only takes a few minutes to clone+modify an existing renewal hook for it. It's pretty much been hands off for years now. I can't remember when I last had an issue due to an expired cert that didn't get updated on a server.

BTW certbot's default behavior is to renew 90-day LE certs every 60 days, as that is what LE themselves recommend.

Seems pretty scalable to me.

[nightfire]

This whole scenario depends on what type of deployment and users you have.
If you have only internal users that will access those services, where you also have control over the clients, an internal CA is in most cases the way to go in my opinion.
I am sysadmin and in my company we have the typical windows domain network, and some stuff honestly can be, lets say, interesting if not done right or not in the way M$ intends you to use their products...
Also some browsers like Firefox do not use the Windows certificate store by default, they need a GPO to do so:
https://admx.help/?Category=Firefox-ADMX&Policy=FirefoxADMX::MOZILLA_FIREFOX_LOCKED_CERT_SYSTEM_STORE

Also enrollment of the internal CA Certificates in the appropriate locations in the Windows Client certificate store can sometimes be a bit counter-intuitive, had also some dance with some RDP certs lately.

[Jackster]

LetsEncrypt does offer wildcard certs using DNS.

https://community.letsencrypt.org/t/acme-v2-production-environment-wildcards/55578

[paulca]

Yes Lets Encrypt does allow you to create wildcards.

Yes you can cron a renewal using the script.

Trouble is.  The platforms.  Take Proxmox for instance.  To change it's certificate just for it's web-interface, not it's inter-cluster coms cert, you have to either use it's LetEncrypt "ACME" API, the console or the webinterface.  The cert has to be registered and distributed by the cluster.  Aka a bespoke install.  In most cases convincing these types to use wildcards is fiddly or in the case of proxmox, impossible.   Thus you need 1 cert per node for the HTTPS and another cert for the cluster coms.  Several different console commands and web UI to update those.

Then there is apache and ngnix which have additional steps after LE completes.

Then, where the pain starts to build, you have "containerised" appliances like GitLab-CE, Docker-Registry, GitLabRunners.... all of which are docker containers built in docker containers running in docker containers.  If you want to place a self signed or LE cert into that setup, you have to basically rework all the docker files, docker-compose and kubernetes deployment scripts to "map" in the ca-cert for their "intra-service" communications to work.

So many of these platforms today not only expect you to be running in a cloud behind a "borrowed auto-gen cert", such as via AWS Certificate Manager, but will not work without a public CA trust anchor.  In the case of GitLab, trying to get all of the components to trust a self generated ca trust cert is an absolutely nightmare which basically have to start from step ONE of building the environment and customisations added at every step!

If it was a simple matter of running a cronjob to regenerate the cert per-hostname every 90 days and maybe a few copy/rsyncs and a few service restarts...  I'd be all over it.

The main issue is that I don't have 25 instances of a platform base OS.  I have 25 instances of a wide range of different services and almost ALL of them have some particular config or setup to accept/renew/update their certificate.  Some want the partial chain, some want the full chain, some want a crt, some want a pem.

Some of them want a cert they can use to SIGN CSRs dynamically (if you really want to use SSL within a dynamic container platofrm like K8S this is the preferred approach.  The vast majority of people just avoid it at all costs and setup HTTP inside the cluster and ban HTTPS to the edge nodes/load balancers.  This works for software development in enterprise, but when you are deploying 3rd party and open source "infrastructure" type stuff you can't control all of them that easily and .. as above.. they have a wide disparity of mechanisms for installing and updating certs.

Oh... and LE in my case can be done with DNS auth, however, that means that ALL of the hostnames I want to secure with LE have to be in public DNS.

tldr;  renewing the cert if only 10% the problem.

[paulca]

Talking about HTTPS in particular.  Incoming rant...

It used to be that it was only financial and highly sensitive connections which were secured with SSL.  Most of these certificates were full organisation identity trust chains.  Banks, credit card payment gateways etc. etc. 

The vast majority of the content on the internet was plain text HTTP.  HTTPS was expensive.  There is a very real CPU overhead and webhosts know that a website using HTTPS... if it becomes popular or gets DDOSed is going to "hurt" the server load significantly.  The trust chain payments (which are tied to profit and a little warranty) are basically a protection racket.

We briefly moved through a time where HTTPS was also used for a large number of websites with "domain verification only" certs to SSL encrypt the login page, maybe the password change page and maybe the "edit profile" page.  Maybe the whole shopping cart, but not the actual shop.

Today, every single host is HTTPS.  Point in case, this website is HTTPS.  Why?

I recently installed a web cache.  Squid.  The main purpose was to cover "apt update" etc.  When I run that on 20 hosts, I would like to cache as much as possible of it so as to "not be a dick" to the server providers.

For an experiment I opened it up to the rest of the network, including my browsers.  It was fine.  After a month I went to look for those juicy browsing stats of what I visited most often what used the most bandwidth, how fast were connections and how was lag....

I got nothing.  99% of traffic was all just "DIRECT TUNNEL HTTPS".

You can't cache HTTPS traffic.  So that alone probably multiplies the amount of energy consumed and the amount of carbon released as a result of a website like this one by a factor of 100!  Every request has to go through the full SSL handshake even to get the "HEAD" and thus defeats all but MIM caches.

There IS still a tiny iota of sanity out there.  Most... not all, but most opensource repositories and mirrors remain HTTP, so they can be cached.  Not all though and there is mounting pressure for ALL of that to go HTTPS to prevent "URL Hijacks" and malware injection.  The trouble with that is, if an enterprise runs an update on 1000 applications, all 1000 applications will have to make new SSL connections to at least verify their own cache.  Granted there are management layers you can put in place to ease that.  Self hosted mirrors for example.

When it comes to self-hosted, internal services, already on a private secured network, is HTTPS even necessary?  I mean, if you have "no single point of trust" or "minimum trust principle" as your main policy... you have made your own bed.  But for a home-lab, I find it irritating that everything will all but insult you for trying to turn HTTPS off!

/rant.

[paulca]

Answering my own question with my own words...

""
For an experiment I opened it up to the rest of the network, including my browsers.  It was fine.  After a month I went to look for those juicy browsing stats of what I visited most often what used the most bandwidth, how fast were connections and how was lag....

I got nothing.  99% of traffic was all just "DIRECT TUNNEL HTTPS".
""

Exactly.  It's not just "me" who can run a cache, so it's not just ME who can see that history.

That is why HTTPS has become the norm, because companies and criminals spying on them.

EDIT:  This limits you to only be spied on by those you hold SSL connections with.  In which case it is that entity 99% of the time which holds that cert.  It is also used to encrypt the data that company are mining from you, so you have no idea what it is.  Windows Telemetry for example is encrypted.

[DimitriP]

Today, every single host is HTTPS.  Point in case, this website is HTTPS.  Why?

For one thing, to make google search results "safer" which makes google happier,
and for another  most browsers have "require HTTPS" turned on by default.

[berke]

Today, every single host is HTTPS.  Point in case, this website is HTTPS.  Why?

For one thing, to make google search results "safer" which makes google happier,
and for another  most browsers have "require HTTPS" turned on by default.

Of course Google is happier, competitors can't passively index the Web by monitoring real traffic.

[Jackster]

Just proxy via Nginx then. The certbot will auto-update the cert every 90 days.
The Nginx to service can be SSL with your own cert that lasts however long you want it to last.

[Foxxz]

Just proxy via Nginx then. The certbot will auto-update the cert every 90 days.
The Nginx to service can be SSL with your own cert that lasts however long you want it to last.

This is the answer and how the pros do it. You can put all the domain names you need in the cert. Use the cert in nginx and reverse proxy back to your containers and backends.

certbot certonly -d www.domain1.com -d something.domain2.com -d gitlab.domain3.com

[paulca]

Just proxy via Nginx then. The certbot will auto-update the cert every 90 days.
The Nginx to service can be SSL with your own cert that lasts however long you want it to last.

This is the answer and how the pros do it. You can put all the domain names you need in the cert. Use the cert in nginx and reverse proxy back to your containers and backends.

certbot certonly -d www.domain1.com -d something.domain2.com -d gitlab.domain3.com

Finally we are getting somewhere

Interesting idea.  As long as that proxy trusts my random internal CA cert on the "inside", I can apply the public's anchored LE certs to it alone to satisfy the browser....  and MIM myself nicely.

By the way, I believe the browers now seeking remote services to validate certificates is in direct response to so many organisations running private ca_cert'd proxies like this.  The trouble is, they can MIM some pretty legally sensitive stuff inadvertantly, like someone's online banking or medical websites.  The browsers stepping in and refusing to serve pages for invalid certs would immediately bring the MIM to the attention of the user.  They have to manually add the "trojan cert" to the browser as a way of confirming they are ware of it.

[Foxxz]

Just proxy via Nginx then. The certbot will auto-update the cert every 90 days.
The Nginx to service can be SSL with your own cert that lasts however long you want it to last.

This is the answer and how the pros do it. You can put all the domain names you need in the cert. Use the cert in nginx and reverse proxy back to your containers and backends.

certbot certonly -d www.domain1.com -d something.domain2.com -d gitlab.domain3.com

Finally we are getting somewhere

Interesting idea.  As long as that proxy trusts my random internal CA cert on the "inside", I can apply the public's anchored LE certs to it alone to satisfy the browser....  and MIM myself nicely.

By the way, I believe the browers now seeking remote services to validate certificates is in direct response to so many organisations running private ca_cert'd proxies like this.  The trouble is, they can MIM some pretty legally sensitive stuff inadvertantly, like someone's online banking or medical websites.  The browsers stepping in and refusing to serve pages for invalid certs would immediately bring the MIM to the attention of the user.  They have to manually add the "trojan cert" to the browser as a way of confirming they are ware of it.

You can configure nginx to trust OS CAs or ignore invalid certs or whatever you want. If its all on the same system such as a docker container or a trusted network you can skip internal SSL altogether.

As for the browsers externally verifying certs it was more in response to certain countries undermining certificates in the same way you are describing organizational CAs

[exe]

Let's encrypt can be automated.

I used to use and now setting up kubernetes + cert-manager. Due to (insane for a single-node "cluster") complexity of the setup, I can't recommend this one as it takes quite some time investment. 

A more accessible option could be nginx + certbot or something (https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/).