Skip to main content
Flexible Top Header

I’ve upgraded my lab to 25.1 and have found that the Let’s Encrypt certs aren’t working as expected.  I don’t see errors in Firefox or Chrome but code to check the certs are failing against 25.1 with the error “ Certificate validation error: FQDN-25.1 unable to get local issuer certificate]” which lead me to test with curl.

I’m wondering if anyone else is seeing the same issue or is this something I’m doing wrong.

This is the error I see with “curl -v https://FQDN/”:

root@bam:~# curl -v https://FQDN/
*   Trying 10.0.10.26:443...
* Connected to FQDN (10.0.10.26) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

I don’t see the error on a 9.6.1 BAM using the same certificate file and key.

If I look at /data/server/conf/server.cert on the 25.1 BAM I just see the local cert.  The local and issuer certs are both in the file on the 9.6.1 BAM.

The BAM was upgraded from 9.6.1 to 25.1.  After the upgrade I had to load a renewed certificate and that’s when the issue started.  I spun up a new 9.6.1 BAM and loaded the new certificate on it to compare.

There were some issues in the beta/QA versions of 25.1 with the chain certs not properly being configured during an upgrade but I believe they were fixed.  In my lab I upgraded from 9.5.2 to the GA 25.1 release and the chain cert was carried over correctly.  On the 25.1 box you should have /data/server/conf/nginx.cert which should be a concatenation of /data/server/conf/server.cert (the end entity cert) and /data/server/conf/bundle (the chain/intermediate) - at least on my system.  I’ve not yet tried reconfiguring or applying a new cert on 25.1, this is just what resulted from my upgrade.

On your 9.6 systems are you applying a local cert file that contains both the end entity AND the chain/intermediate in one file?  Or did you apply distinct cert and intermediate?

I’d also use “openssl s_client -connect bamfqdn:443 -showcerts” rather than curl when troubleshooting just to get a clearer picture of what certs are being presented.

I just tested on 25.1 loading a new cert (well technically it was the same cert, but went through the Load Custom Cert method) - and it rewrote all mentioned files correctly and correctly presents the entire chain.

There were some issues in the beta/QA versions of 25.1 with the chain certs not properly being configured during an upgrade but I believe they were fixed.  In my lab I upgraded from 9.5.2 to the GA 25.1 release and the chain cert was carried over correctly.  On the 25.1 box you should have /data/server/conf/nginx.cert which should be a concatenation of /data/server/conf/server.cert (the end entity cert) and /data/server/conf/bundle (the chain/intermediate) - at least on my system.  I’ve not yet tried reconfiguring or applying a new cert on 25.1, this is just what resulted from my upgrade.

On your 9.6 systems are you applying a local cert file that contains both the end entity AND the chain/intermediate in one file?  Or did you apply distinct cert and intermediate?

The original cert loaded during the 9.6.1->25.1 upgrade worked, it was only loading the new cert on the 19th when the old one expired.

The file has both certs together.

I’d also use “openssl s_client -connect bamfqdn:443 -showcerts” rather than curl when troubleshooting just to get a clearer picture of what certs are being presented.

I did that as well and verified the issuer cert wasn’t being presented on 25.1.  It’s just a lot more verbose output so I didn’t include it.

I updated nginx.cert to include the issuer cert and restarted Proteus and it’s presenting both as expected.  So I need to investigate more to see why it’s not including the issuer cert.

You submitted 3 files (key, cert, intermediate/issuer) while loading the new cert?

You submitted 3 files (key, cert, intermediate/issuer) while loading the new cert?

Two, cert (issuer cert is included) and key.  If I try to load the issuer cert separate from the host cert with the host cert and key I get an error and see this in the logs:

2025-Aug-21 13:51:58.209 ERROR  jetty-thread-130]  com.bluecatnetworks.proteus.api.service.ExceptionResponse] A NullPointerException was encountered while servicing an API request
java.lang.NullPointerException: Cannot invoke "java.security.cert.Certificate.getEncoded()" because "certificate" is null
        at com.bluecatnetworks.tools.keystore.KeyStoreUtils.pemFormatCertificate(KeyStoreUtils.java:633) ~~shared-keystore-25.1.0-10

 

Separate your issuer cert(s) and your end-entity cert into separate files.  Upload the entity cert as Domain Signed Certificate and your issuer cert(s) as Intermediate Bundle Certificate.  I verified if you bundle the issuer with the end entity and upload that as Domain Signed that the BAM behaves as you describe (issuer is not presented).  Uploading the issuer (aka chain, aka intermediate - all the same thing) separate from the Domain Signed (“end entity”) cert should get you the appropriate behavior.  I’ve never tried bundling them together like you did under 9.x so I don’t know what the previous behavior was (I assume you did that’s why you were continuing to do it that way in 25.1) - but it definitely doesn’t seem to work if you do that in 25.1.

 

 

Oh, and drag/drop or upload all 3 files BEFORE clicking Update web access settings.  They need to be submitted all together.

As I mentioned in my previous post I did try splitting the certs out and I did upload them all together.  When I do I get the error “Unable to update Web access settings: An unexpected error was encountered while servicing request” and server.log shows the error I included above.

I’m sure I’m doing something wrong but can’t see what it is.

Sorry, seems a CARE case is in order.

That was going to be the next step.

Reply


Terms & ConditionsCookie settings