Field Report – PKI Spotlight Changes Everything

It’s been nearly 20+ years that I have been focused on PKI and working with customers around the world. In that time, new versions of Windows Server OS have brought various levels of new features, though not much lately. In all of that time, there has rarely, if ever been a moment that has been so earth-shattering in support and working with PKI as I had last week with one of our customers. This is a story about that experience.

One of our large energy customers had reached out as we had helped them nearly a year ago with setting up Azure AD Multi-Factor Authentication with the Network Policy Server extension. This required creating a certificate template and configuring the extension to use the certificate based on a thumbprint of the cert.

A year later, inexplicably, the MFA certificate on one of their test systems was “automatically” renewed. The odd thing was that the template was not marked for Autoenrollment, at least according to the customer. The reason this was an issue was that when the certificate is issued or renewed, someone has to manually update the Azure system with the thumbprint of the certificate. This is not done automatically, so having the MFA certificate renewed without manual intervention meant there would be an outage on their production system when, and if, they automatically renewed the certificate.

So we set out to review why the system seemingly auto-enrolled for a new certificate. We looked at all of the usual places, template settings, CAPI logs, and the CA database request attributes. All things pointed to the fact that this was not manually performed, but something else had automatically renewed the certificate. But there was no smoking gun. My instinct told me that it must be the template, it must have been set to autoenrollment.

The customer did indicate that when they saw this issue occurred that they immediately removed Read and Enroll rights for the NPS servers so that they couldn’t perform any further renewals. In my mind, the only thing that seemed to make sense would be that the template had been enabled for auto-enrollment and that the customer also, inadvertently, removed read, enroll, and auto-enrollment. That would have been the “best guess” and based on years of experience. But it was just a hunch.

It was then, that we remembered that we had a brand new tool to help with scenarios like this. This customer had recently deployed PKI Spotlight! One of the features of PKI Spotlight is the central collection and storage of events across the PKI – including changes to template properties in AD.

We quickly pulled up PKI Spotlight, jumped to Events, and filtered for events in DS Certificate Templates. There in front of us was every change to templates since PKI Spotlight was deployed. We could see the customer PKI Admin modify the template where we could see the template version change, and we could also see the template ACLs change – and indeed only Read and Enroll permissions had been granted and removed by the PKI Administrator. There was no ACL for auto-enrollment. So we knew for sure it wasn’t a user error. We had definitive proof that some other process had caused this renewal. We also knew it wasn’t the standard auto-enrollment mechanism and something within NPS or another mechanism is at play.

We are now working to reproduce the issue and elevate it to Microsoft as necessary if we can determine this is a bug in NPS. Since we don’t have a strong root cause at this point, we decided to remove the template from the CA until we can properly determine how and when the renewal will occur and through which mechanism so we can properly coordinate the MFA thumbprint update.

Without PKI Spotlight and its monitoring abilities, we would have been forced to make assumptions about activities within the PKI and potentially allow a company-wide impact to occur by assuming the problem wouldn’t occur again. So a seemingly simple process of seeing the event occur enabled us to PREVENT an outage across the company. It is truly a brand new day in PKI!

You can see PKI Spotlight in action on our Minding the Keystore Youtube channel.