Friday, 24 October 2008

Hardening SSL Cipher Strength and SSL Protocol Support on ISA Servers

There are many weird and wonderful ways that you can further harden your ISA Server installation over and above the level provided by the default install. This subject is covered in some detail in the Microsoft articles titled ISA Server 2006 Security Guide and ISA Server 2004 Security Hardening Guide. However, an advisory that seems to come up on a majority of security/penetration reports that I have seen when testing ISA Server deployments, is that of recommended minimum negotiable SSL cipher strengths and gratuitous SSL protocol support.

Microsoft have produced a very detailed knowledge base article called How to Restrict the Use of Certain Cryptographic Algorithms and Protocols in Schannel.dll (KB245030) that covers the overall concept of SSL security restrictions, but it is quite complicated in places, mainly due to the technical depth and detail provided. The aim of this blog post is to cover what I believe to be the two main mitigation areas that need to be employed; namely improving the SSL cipher strength to only support 128bit or above cipher suites, and ensuring that only recommended SSL protocols be used. Both of these changes eliminate any of the concerns with using cipher strengths below 128bit, or weaker SSL protocols, which are often flagged during security testing as providing an insufficient level of security for most environments. They also fix an ISA Server specific issue, discussed later.

It is worth noting that, if required, ISA Server can be configured to enforce 128bit encryption for web publishing rules which utilise HTTPS. An example of this is shown below:

Unfortunately, this does not prevent weaker SSL ciphers strengths from being negotiated with the operating system, which often leads to false positives during security testing. An understanding of why this occurs is provided in the following knowledge base article ISA Server 2006 and ISA Server 2004 do not reject weakly encrypted authentication requests for access to an SSL Web site after you configure ISA Server to require 128-bit encryption (KB937293)

What this option does do, however, is ensure that ISA will not pass any traffic that is not encrypted to this required 128bit level and consequently deny user access. In addition, this option does not provide any way to enforce the use of specific SSL protocols. It also doesn't enforce the minimum cipher strength if people forget to tick the box! :)

Based upon my understanding, ISA Server hands all SSL processing to the operating system; Schannel.dll to be precise. Therefore, if the operating system has been left at the default state, it will be possible to negotiate SSL connections using inappropriately weak cipher strengths.

With this view in mind, I decided to ensure that (where possible) all my ISA Server deployments would include a higher level of SSL cipher strength and limit SSL protocols (compared to the default state) to provide a more secure (hardened) platform when using ISA Server to host SSL connections. In addition, I wanted to ensure that security and penetration testing of implemented solutions wouldn't highlight potential false positives from SSL negotiation, and also ensure that if the customer was unaware of the 'Require 128bit encryption...' option you could still rely on the operating system to provide a level of protection to prevent the use of weak SSL security.

Using the KB article, I decided that I would prevent the operating system from accepting cipher suites below 128bit, therefore I was able to define the following standards:

  • DES 56/56: DISABLED
  • RC2 40/128: DISABLED
  • RC2 56/128: DISABLED
  • RC2 128/128: ENABLED
  • RC4 40/128: DISABLED
  • RC4 56/128: DISABLED
  • RC4 64/128: DISABLED
  • RC4 128/128: ENABLED
  • Triple DES 168/168: ENABLED
I also decided to standardise on the use of SSL 3.0 and TLS 1.0 only, therefore I was able to define the following standards:

  • PCT 1.0 Client: DISABLED
  • PCT 1.0 Server: DISABLED
  • SSL 2.0 Client: DISABLED
  • SSL 2.0 Server: DISABLED
  • SSL 3.0 Client: ENABLED
  • SSL 3.0 Server: ENABLED
  • TLS 1.0 Client: ENABLED
  • TLS 1.0 Server: ENABLED

To this end I was then able to create the following registry files:


Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\DES 56/56]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 128/128]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 40/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 56/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 128/128]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 64/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\Triple DES 168/168]"Enabled"=dword:ffffffff


Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Client]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Client]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]"Enabled"=dword:ffffffff

If required, copies of these registry files can be found here and here, respectively.

Once you have applied both of these registry files, the operating system will then meet the required cipher strengths and protocol support defined above.

Please Note: A reboot is necessary to apply the above registry changes and allow Schannel.dll to enforce the new standards.

Although I have aimed this blog post at ISA Server, the SSL hardening elements defined in this article are equally applicable to any system that supports or terminates SSL/TLS connections. Therefore, the registry files could just as equally be applied to a secure Internet facing web server to ensure suitable strong SSL cipher suites and protocols are employed.

As with all security measures and hardening, there is risk that these changes may impact functionality. For example if you have clients that are not able to support 128bit encryption or not configured to utilise SSL 3.0/TLS 1.0, connections from these clients will undoubtedly fail. A good example of this is Internet Explorer, which should ideally be configured as shown below:

It is also worth noting that these changes are not required on Windows Server 2008 platforms as the secure-by-default configuration has higher entry-level restrictions on cipher strength and protocol support. However, as the current version of ISA Server is not supported on Windows Server 2008, therefore the above standards are still very applicble to a vast majority of ISA Server deployments which utilise secure web publishing.

The end result is that you will have a more secure deployment of ISA Server and will also minimise (if not eradicate) the number of SSL related risks which commonly appear on security/penetration test reports for HTTPS based services. Both of which are pretty desirable in my opinion ;)

UPDATE 08/11/08

In addition to the use of registry files, I have found a great tool written by Marcello Gorlani called CipherControl. The tool provides a GUI utility for changing SCHANNEL parameters which is very useful!

I also wanted to add that once the appropriate Cipher strengths have been amended (using CipherControl or the above registry method) they can then be audited using Serversniff or SSL Digger to confirm the required configuration.