TLS Certificate Management
How APIs solved the management of client-side TLS Certificates used for MTLS authentication
Problem MTLS-authenticated APIs are difficult to manage, especially when the API provider also supplies the client-side TLS certificates. The overhead of managing and maintaining certificates becomes too high.
Solution By leveraging DigiCert, we built a user interface and an API for users to self-manage their client-side TLS certificates
What is MTLS?
Introduction
If your systems and APIs require your users to be MTLS-authenticated, how can you provide a seamless way for your users to get a TLS certificate and start using your MTLS-authenticated APIs and systems?
Recently, a large customer of ours had this very problem to solve. Like many businesses that have public and partner APIs, they employ MTLS Authentication to help secure connections to some of their high-value APIs. Our customer provided the certificate-authority (CA) and were therefore providing privately-signed certificates for both the client-side (their API users) and the server-side. The API users would send a Certificate Signing Request (CSR) to our customer, who would then email the user their client-side TLS certificate that was privately signed by the CA.
The problem was, privately-signed certificates only live for 3 years and with a growing user-base there is an increasing cost (manual effort) in creating and replacing client-side TLS certificates.
It was soon costing our customer hundreds of thousands of dollars to manage this problem
Our Solution
Given our experience with APIs and TLS certificates we offered to build a solution whereby their API users can self-manage their TLS certificates, and any certificate changes would be automatically picked up by the APIs. This would save the company hundreds of thousands of dollars each year, provide a better user experience and significantly reduce the time it takes for users to access MTLS-authenticated APIs and systems.
We built 2 interfaces for users to self-manage their own certificates:
TLS Certificate Management API - an API that will provide the function to retrieve, renew, revoke and create certificates.
TLS Certificate Management UI - a user interface providing the same features as the API.
The key to the solution was DigiCert's API, which allowed us to automate the certificate store
The Solution Detail
The solution was made up of the following technology components;
DigiCert API - the certificate authority where all the certificates are managed and stored.
Identity system - the existing identity system for users with the introduction of new roles to authenticate users through the API or UI.
Dell Boomi processes - for the backend logic verifying the users' JWT token, integrating with the certificate store (for transformation), updating the certificate record table and providing certificate expiry notifications.
Azure Table - as a source-of-truth for relating certificate identities to user identities.
Azure Queue - as a message queue that was polled by the company's notification system to notify users of upcoming certificates that are soon to expire.
Edge-network-device MTLS authentication - shifting from individual system-based MTLS-authentication to network-based MTLS through the load-balancers and implementing OCSP for real-time certificate validity checks.
API at the Forefront
The TLS Certificate Management API was the key part to get right. Taking an API-first approach helped to ensure the users' needs would be addressed, and would also ensure the user interface had the same features in lockstep with our API.
The main user feature we wanted to provide was the ability of users to create and renew their TLS certificates, as this solved the bigger problem of manually managing client-side TLS certificates. Also, users needed a way to retrieve their TLS certificates related to their user account and also revoke TLS certificate they may not need anymore. For the first release of the TLS Certificate Management API and UI, we provided these 7 features;
Create new certificates - by providing a valid CSR, a user can create a new TLS certificate signed by a number of CAs available to them.
Download certificate - the user can download their TLS certificates in multiple formats.
Get a list of certificate orders - a user can retrieve a list of all their TLS certificates.
Get details on a certificate order - a user can retrieve details on a particular TLS certificate.
Reissue a certificate - allows a user to duplicate an existing valid certificate with a new CSR (and private key). This allows a user to differentiate client-based access to the MTLS-authenticated system using duplicated certificates with different serial numbers.
Renew a certificate - within 90-days of expiry the TLS certificate can be renewed, allowing the user to stay connected to MTLS-authenticated APIs and systems.
Boomi Process Development and DigiCert Integration
It’s always good practice to push as much logic as you can to a back-end, or an integration layer, so it can do the heavy lifting of transformation and validation, leaving the API lightweight. In this case we used our customer’s pre-existing Integration platform, Dell Boomi. Although to a seasoned developer it would be easier to create a dedicated backend application for this work, the Boomi platform provided great out-of-the-box integrations to Azure (for APIM, Table, Queue, Key Vault and other services we needed) and will be easier for non-developers to maintain going forward.
We used Boomi to integrate directly with Digicert’s CertCentral API (specifically their Services API). DigiCert provides a wealth of features and options for certificate-management, and they also provide all this through a rich set of APIs, making it the perfect choice for creating an automated client-side TLS certificate solution.
Most of the interaction we needed was only with a couple of endpoints;
https://www.digicert.com/services/v2/order- for creating TLS certificates, listing certificate orders and their details, renewing, reissuing and revoking TLS certificates.https://www.digicert.com/services/v2/certificate/- for downloading TLS certificates.
OCSP Implementation
A key of automating the self-serve to TLS certificates for MTLS authentication is ensuring that any TLS certificate changes (new, reissued, renewed, or revoked certificates) are picked-up by the MTLS-authenticated systems. By pushing the MTLS-authentication to the network edge (prior to the protected systems) we were able to use DigiCert’s OCSP responders to check the validity of certificates as TLS connections were established.
There were initially some concerns around additional latency and service availability. Additional latency in establishing TLS connections would slow response time for users, therefore it was important to test this against the existing MTLS authentication method using locally checked certificates fingerprints. Trials showed no more than 10ms of response time with no discernable difference to the current local certificate validation checks. Service availability of OCSP was also an important consideration, the Digicert OCSP responder availability target was 99.99% and in fact it had a much higher actual availability. As a fallback in case of OCSP responder unavailability, a certificate revocation list (CRL) was implemented locally on the network edge devices as a fallback mechanism.
After implementing OCSP at the network edge we now had an automated way of ensuring TLS certificate validity checks.
User Interface
The user interface was created as a single-page-application (SPA) in Vue.js on the customer’s existing user portal, which also took care of authentication. We wanted to make sure the UI was as simple to use as possible, and also integrated completely with all the features of the TLS Certificate Management API. Essentially the UI was just a graphical mask over the API experience, which is exactly what we were aiming for so that our development was API-driven.
We only needed to make one change to the API and that was how it handled authentication from the UI, which was already in a logged-in state (as opposed to direct API-use by a client, which required them to authenticate). We solved this through the API by the UI passing on the session-id cookie to the API in a header, which when provided the API would use that to authenticate the user to the Identity Service.
Outcome
We had a hard deadline to get the solution launched - before the majority of client-side TLS certificates hit their renewal period, which was June 2023. We launched the TLS Certificate Management application (the UI and the API) mid-April, giving us plenty of time to ensure the users were familiar with the new self-serve capability and to iron out any bugs we didn’t catch during development and testing.
Overall the launch was a huge success, and over the months we’ve seen all users adapt quickly to using the TLS Certificate Management application for managing their certificates, and likewise a significant drop in manual requests. This realised the major benefit of providing the solution in removing the manual overhead our customer was providing in managing the client-side TLS certificates and saving our customer hundreds of thousands of dollars each year.
