Friday, 24 October 2008

Hardening SSL Cipher Strength and SSL Protocol Support on ISA Servers

There are many weird and wonderful ways that you can further harden your ISA Server installation over and above the level provided by the default install. This subject is covered in some detail in the Microsoft articles titled ISA Server 2006 Security Guide and ISA Server 2004 Security Hardening Guide. However, an advisory that seems to come up on a majority of security/penetration reports that I have seen when testing ISA Server deployments, is that of recommended minimum negotiable SSL cipher strengths and gratuitous SSL protocol support.

Microsoft have produced a very detailed knowledge base article called How to Restrict the Use of Certain Cryptographic Algorithms and Protocols in Schannel.dll (KB245030) that covers the overall concept of SSL security restrictions, but it is quite complicated in places, mainly due to the technical depth and detail provided. The aim of this blog post is to cover what I believe to be the two main mitigation areas that need to be employed; namely improving the SSL cipher strength to only support 128bit or above cipher suites, and ensuring that only recommended SSL protocols be used. Both of these changes eliminate any of the concerns with using cipher strengths below 128bit, or weaker SSL protocols, which are often flagged during security testing as providing an insufficient level of security for most environments. They also fix an ISA Server specific issue, discussed later.

It is worth noting that, if required, ISA Server can be configured to enforce 128bit encryption for web publishing rules which utilise HTTPS. An example of this is shown below:

Unfortunately, this does not prevent weaker SSL ciphers strengths from being negotiated with the operating system, which often leads to false positives during security testing. An understanding of why this occurs is provided in the following knowledge base article ISA Server 2006 and ISA Server 2004 do not reject weakly encrypted authentication requests for access to an SSL Web site after you configure ISA Server to require 128-bit encryption (KB937293)

What this option does do, however, is ensure that ISA will not pass any traffic that is not encrypted to this required 128bit level and consequently deny user access. In addition, this option does not provide any way to enforce the use of specific SSL protocols. It also doesn't enforce the minimum cipher strength if people forget to tick the box! :)

Based upon my understanding, ISA Server hands all SSL processing to the operating system; Schannel.dll to be precise. Therefore, if the operating system has been left at the default state, it will be possible to negotiate SSL connections using inappropriately weak cipher strengths.

With this view in mind, I decided to ensure that (where possible) all my ISA Server deployments would include a higher level of SSL cipher strength and limit SSL protocols (compared to the default state) to provide a more secure (hardened) platform when using ISA Server to host SSL connections. In addition, I wanted to ensure that security and penetration testing of implemented solutions wouldn't highlight potential false positives from SSL negotiation, and also ensure that if the customer was unaware of the 'Require 128bit encryption...' option you could still rely on the operating system to provide a level of protection to prevent the use of weak SSL security.

Using the KB article, I decided that I would prevent the operating system from accepting cipher suites below 128bit, therefore I was able to define the following standards:

  • DES 56/56: DISABLED
  • RC2 40/128: DISABLED
  • RC2 56/128: DISABLED
  • RC2 128/128: ENABLED
  • RC4 40/128: DISABLED
  • RC4 56/128: DISABLED
  • RC4 64/128: DISABLED
  • RC4 128/128: ENABLED
  • Triple DES 168/168: ENABLED
I also decided to standardise on the use of SSL 3.0 and TLS 1.0 only, therefore I was able to define the following standards:

  • PCT 1.0 Client: DISABLED
  • PCT 1.0 Server: DISABLED
  • SSL 2.0 Client: DISABLED
  • SSL 2.0 Server: DISABLED
  • SSL 3.0 Client: ENABLED
  • SSL 3.0 Server: ENABLED
  • TLS 1.0 Client: ENABLED
  • TLS 1.0 Server: ENABLED

To this end I was then able to create the following registry files:


Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\DES 56/56]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 128/128]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 40/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 56/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 128/128]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 64/128]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\Triple DES 168/168]"Enabled"=dword:ffffffff


Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Client]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Client]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client]"Enabled"=dword:ffffffff
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]"Enabled"=dword:ffffffff

If required, copies of these registry files can be found here and here, respectively.

Once you have applied both of these registry files, the operating system will then meet the required cipher strengths and protocol support defined above.

Please Note: A reboot is necessary to apply the above registry changes and allow Schannel.dll to enforce the new standards.

Although I have aimed this blog post at ISA Server, the SSL hardening elements defined in this article are equally applicable to any system that supports or terminates SSL/TLS connections. Therefore, the registry files could just as equally be applied to a secure Internet facing web server to ensure suitable strong SSL cipher suites and protocols are employed.

As with all security measures and hardening, there is risk that these changes may impact functionality. For example if you have clients that are not able to support 128bit encryption or not configured to utilise SSL 3.0/TLS 1.0, connections from these clients will undoubtedly fail. A good example of this is Internet Explorer, which should ideally be configured as shown below:

It is also worth noting that these changes are not required on Windows Server 2008 platforms as the secure-by-default configuration has higher entry-level restrictions on cipher strength and protocol support. However, as the current version of ISA Server is not supported on Windows Server 2008, therefore the above standards are still very applicble to a vast majority of ISA Server deployments which utilise secure web publishing.

The end result is that you will have a more secure deployment of ISA Server and will also minimise (if not eradicate) the number of SSL related risks which commonly appear on security/penetration test reports for HTTPS based services. Both of which are pretty desirable in my opinion ;)

UPDATE 08/11/08

In addition to the use of registry files, I have found a great tool written by Marcello Gorlani called CipherControl. The tool provides a GUI utility for changing SCHANNEL parameters which is very useful!

I also wanted to add that once the appropriate Cipher strengths have been amended (using CipherControl or the above registry method) they can then be audited using Serversniff or SSL Digger to confirm the required configuration.


  1. Hi Jason,
    Very, very, very good article indeed.
    Definetely clears the whole situation.
    In the doc for EAL certification, as part of Installation Procedures of ISA, page 10 and 11 or page 14 and 15, they apply some registry settings to "enforce 128 bit encryption for Forms-based authentication" right after they install ISA.
    But few people read that doc anyway. And you've covered a larger area and detailed the process too.

  2. Hi Adrian,

    Thanks :)

    Great link, I have seen that doc in the past but never linked it to the MS KB article.



  3. Thank you sincerely for such an excellent post on this topic.

    Throw up a tip jar and I'm buying you a nice lunch. I think you just saved me two days of work.

  4. Does IIS 7 required the same patch?

  5. >>Anonymous

    No, Windows Server 2008 (upon which IIS7 runs) has much better default SSL ciper settings.

    However, ISA Server 2006 is only supported on Windows Server 2003.



  6. Thanks Jason. Saved me a lot of work. Greetings from the Netherlands.

  7. Can you help me to set Windows 2008 R2 with TMG to only set up null encrypted SSL connections???

    1. Why would you want to do that? Probably look here: to enable those specific ciphers. It seems an odd thing to do, so not even sure it is possible (never tried it).

  8. Your blog is Awesome blog post nice quality .UKfree vpn A good VPN provider will offer servers in a large range of different countries.

  9. Thanks for the valuable feedback. I think that strategy is sound and can be easily replicable.Great posts. I love this article. My security services 1300 788 828 is security monitoring company that provides cheap venue security, business security, Alarm systems, event security, building construction site security, crowd control services and helps to hire friendly personal security officer, security technician, private bodyguard in Sydney, Brisbane, Canberra and The Gold Coast Australia.

  10. I think mimicking popular posts on other blogs is one of the best ways to get a good idea which will be popular.Such a lovely blog you have shared here with us. Really nice.
    Cctv installation

  11. I and my friends watch the football game clips at YouTube for all time, because they have in nice quality.
    Photo retouching services

  12. Hurrah, that’s what I was exploring for, what a stuff! existing here at this blog, thanks admin of this web site.
    Web marketing services in Toronto

  13. Hi, can any body assist me how to get this video tutorial from this web site, I have watched and listen it at this time but would like to down load it.
    Cctv camera Sydney