Handling X509KeyStorageFlags in applications

Hello everyone!

While participating on StackOverflow.com, I’m observing common in-app certificate handling misuses in .NET applications and I want to share some thoughts on this. Today I would like to speak about handling X509Certificate2 object creation inside the application code, common problems in handling private key material, potential issues and how to overcome them.

Problem Description

The not-so-recent “TLS Everywhere” enforcement in Internet PKI forced software developers to use digital certificates in their applications more frequently. Many applications are no longer monolithic, they use external services to process the data and may use certificates for internal use. Common use cases covered by this blog post include:

• Application uses client certificate to authenticate itself at remote API;
• Application uses certificate to digitally sign or encrypt some content.

However, PKI is still hard and the number of PKI misuses in software increased accordingly and one of [many] common issues is certificate and private key handling inside .NET applications. I found that many examples of certificate usage inside applications are broken or flawed in one or another ways. Threads on public forums use copy/pastes from other boards with a little care of what exactly these flags do behind the scene.

Most problems with these use cases occur in web apps when certificate lookup/instantiation fails. For example, certificate reading works on a developer machine, but fails when the application is deployed to production. Few common threads on StackOverflow related to this subject:

• c# – X509Certificate Constructor Exception – Stack Overflow
• c# – HttpClient request with x509 Certificate works on Debug but fails in Production (403 forbidden) – Stack Overflow
• c# – ASP.NET – The specified network password is not correct – Stack Overflow
• Asp.Net Core Web Api With Angular Template Unhandled exception. System.InvalidOperationException: Type ” is not supported – Stack Overflow
• c# – X509 certificate not loading private key file on server – Stack Overflow
• c# – System.InvalidOperationException: Timeouts are not supported on this stream. at System.IO.Stream.get_ReadTimeout() – Stack Overflow

Many reasons may lead to these issues. And even if application looks working, security flaws are introduced in the application when private key material is unintentionally shared with other processes or simply leaked from the system.

Application Certificate Deployment Models

There are two common deployment models for application certificates:

• Persistent — certificate is installed in certificate store which is independent to application.

In persistent deployment model, a certificate is installed in certificate store (such as Windows Certificate Store) using out-of-band process. The installed certificate is shared with any other application that runs under same security account.

The benefits of this deployment model include simplified certificate management, it can be easily installed and managed using external mechanism, such as certificate autoenrollment and certificate lifecycle managers (CLM). CLM makes it easy to manage certificates in certificate store without having to update or re-deploy the application each time the certificate is updated.

This deployment model is the only model that supports keys stored on hardware devices, TPM or HSM (Hardware Security Module).

However, simplified management has its own downsides. There is no certificate isolation/protection between processes. Often, when application is removed, the certificate is not removed and remains on a system. Even if certificate is removed from persistent store (windows certificate store), this doesn’t necessarily removes the private key: The case of accidentally deleted user certificates. Another process under same security account or computer administrator can easily access the certificate and its private key material.

An important note: of course, the private key can be deleted when certificate is deleted from Windows Certificate Store, but it requires some sophisticated techniques and I never seen that anyone uses it.

These downsides may compromise application security. If the certificate must be protected from unauthorized use by anyone except target application, then this deployment model SHALL NOT be used. How certificate and its private key is accessed in the code? Here is a quick example:

class MyClass {
    public static X509Certificate2 GetCertificateFromStore(StoreLocation storeLocation, String searchPattern) {
        using (var store = new X509Store(StoreName.My, storeLocation) {
            store.Open(OpenFlags.ReadOnly);
            X509Certificate2Collection certs = store.Certificates.Find(findType, searchPattern, true);
            if (certs.Count > 0) {
                return certs[0];
            }

            throw new CryptographicException("Requested certificate was not found");
        }
    }
}

The code opens certificate store, looks for certificate and then use it. No secret input is provided, so any application can use same code to access certificate installed in permanent store. However, this code may fail sometimes when storeLocation parameter points to LocalMachine. It works on developer machine and fail when application is deployed to server. This is because of different security context. Unfortunately, it is quite often when developers use highest privileges on a development machine for little to no reasons. Administrators can read and write certificates from local machine store. Standard users do not have access to certificates installed in local machine store. And applications deployed to server do not often have access to certificates installed Local Machine store. First rule of thumb: do not use Local Machine store unless the application runs under local system security context. And in 99% of applications aren’t and shall not.

• In-app — certificate is embedded in the application itself or stored as external protected file. This includes PFX files, key chains, Azure Key Vault and many others.

This certificate deployment model has opposite features than Persistent model. In-app model provides better certificate and private key security: the private key may not be shared with other applications that run under same security account and may prevent computer administrator from accessing private key material. The idea behind this is that application loads the certificate into memory only when necessary and then unloads the certificate and private key. Certificate itself is protected with some secret (simply, PFX password) which is known only to authorized applications. How the certificate and private key is accessed in this case? Here is a quick example:

class MyClass {
    public static X509Certificate2 GetCertificateFromPfx(Byte[] pfxBytes, String password) {
        return new X509Certificate2(pfxBytes, password);
    }
}

There is a parameter called password which is known only to authorized applications and which provides private key isolation between processes. Applications that do not know the password can’t access the certificate and private key. Although the code looks secure, in fact it is not in the current way. The problem is with default behavior of the X509Certificate2 constructors that import certificate from PFX. Using default constructors, such as X509Certificate2(String, String), X509Certificate2(String, SecureString), X509Certificate2(Byte[], String), X509Certificate2(Byte[], SecureString), private key is silently copied and persisted on a system:

PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"
PS C:\> $c = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $pfxBytes,$password
PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"


    Directory: C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a--s       20.06.2021.     18:55       1603 487b53ff0ac839b14243c702864666b4_e63ebd9c-34b8-4edd-89c0-303a056a6d8e


PS C:\>

See Key Storage and Retrieval article to find paths to directories where Windows store private key files.

This example reveals the fact that private key remains on a system and literally equals to persistent model. It is pretty inevident behavior which introduces security flaws with normal working code. Any simple application/script can use FileSystemWatcher to detect key file creation and immediately create a copy of the key file. Unfortunately many developers simply miss this, because the Microsoft documentation isn’t very clear and require a solid background knowledge on how cryptography works. Didn’t I say the PKI is hard?

To avoid this non-desirable behavior, a constructor with three parameters must be used. Third parameter is of type of X509KeyStorageFlags enumeration. Let’s explore options with a brief explanation:

• DefaultKeySet — copies private key to default store. It can be specified in PFX properties (user key set or machine key set). If none specified in PFX, user key set is used. This is something you don’t want to use, because it persists the key on a disc. And if PFX specifies Machine Key Set and application doesn’t have local administrator/system permissions, the call will fail with exception.
• UserKeySet — forces private key copy to user profile regardless what is specified in PFX properties.
• MachineKeySet — forces private key copy to local system profile. If application doesn’t have local administrator/system permissions, the call will immediately fail.
• Exportable — copies private key to default key set (see rules above) and makes private key exportable. This makes zero sense, because application doesn’t need to have an access to private key material. If the application require this, the application is simply wrong. Instead, the application must deal with handles provided by operating system. Nothing else.
• UserProtected — this flag enables private key strong protection. This doesn’t work with any non-interactive applications, because raise UI popups and require user input. And in web/background applications the popup is raised in background sessions you cannot access.
• PersistKeySet — is similar to DefaultKeySet with the exception that private key permanently persists on a disk even if you explicitly release all handles.
• EphemeralKeySet — the only flag that doesn’t copy private key material on a disk and the only that makes sense, because it does exactly what we need — temporarily load the key from PFX and unload when finished.

Obviously, if no parameter of X509KeyStorageFlags enumeration is passed, a DefaultKeySet value is used, which indeed copies the private key to file system. Before .NET Framework 4.7.2 and .NET Core 2.0 any enumeration value or combination persisted the key on a disk and behave similar like persistent deployment model. Starting with aforementioned framework versions, there is a new enumeration value called EphemeralKeySet. If you use it, the private key file is not created on a disk:

PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"
PS C:\> $c = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $pfxBytes,$password,"ephemeralkeyset"
PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"
PS C:\>

One could argue that starting with .NET Framework 4.5.2 you can call X509Certificate2.Dispose() method to delete temporary key file from disk, however for short period of time the file sits on a disk. And any simple application/script can use FileSystemWatcher to detect key file creation and immediately create a copy of the key file. And call X509Certificate2.Dispose() every time when you finish to deal with the certificate.

This leads to an evident solution: you SHALL use EphemeralKeySet enumeration value when load certificate with private key from PFX or Azure Key Vault, for example. If you use any other flag, then you are doing something wrong.

Summary

In this post, I outlined common certificate usage scenarios in applications, common certificate deployment options, their properties, benefits and problems. In summary, I want to provide common best practices when dealing with certificates inside applications:

• Use persistent certificate deployment model if you store certificate and private keys on a hardware device, such as TPM or HSM.
• Be sure if you understand the certificate and private key sharing across processes in persistent certificate deployment model.
• Do not use LocalMachine certificate store in persistent certificate deployment model. Use only CurrentUser store.
• Use only three-parameter constructor for X509Certificate2 class and use only EphemeralKeySet in the third parameter in in-app certificate deployment model.
• Always call X509Certificate2.Dispose() method when finishing the certificate usage. No exceptions.

If you do not follow these simple rules — you misuse the PKI.

Happy coding!