Tuesday, 8 September 2009

Group Policy Processing Errors on ISA Server and Fun with Large ICMP Packets

I recently came across an interesting problem on an ISA Server deployment which I thought might be useful to share.

In this particular deployment, the ISA Server array members were located in a DMZ behind an existing back firewall. Consequently, the back firewall separated ISA Server from internal infrastructure like Active Directory Domain Controllers, Exchange servers etc.

Please Note: Placement of ISA Server in an existing DMZ is a far from ideal deployment scenario, but unfortunately common when using ISA Server as a reverse proxy solution. In an ideal scenario, the ISA Server array members would be placed in parallel to the existing back firewall, or possibly dual-homed directly between the existing DMZ and the internal network. However, that is a discussion for another day ;)

In order to use the Kerberos Constrained Delegation (KCD) feature of ISA Server, it was necessary for these array members to be members of the internal Active Directory domain; the same domain as the published Exchange servers in fact. You can read more here on KCD here and here.

To support the necessary communications for Active Directory domain membership, the back firewall was configured to permit the necessary protocols from the array members to the Active Directory domain controllers. See my previous blog article here for details on the protocols required. 

I tested my web publishing rules and everything was working as expected from an ISA Server perspective. Time to go home? No, not quite :(

However, what did happen was that I started receiving Userenv application event log errors, specifically:

Event ID: 1054
Source: Userenv
Type: Error
Description:
Windows cannot obtain the domain controller name for your computer network. (The specified domain either does not exist or could not be contacted). Group Policy processing aborted.

At first I assumed that the back firewall had been configured incorrectly, but after validating the configuration using Ping and PortQry, I realised that this was not the case.

After a bit of investigation I came across the following Microsoft KB article which describes how to enable debug logging for Userenv:

How to enable user environment debug logging in retail builds of Windows (KB221833)

From looking at the userenv.log file generated using the above KB article, I could see the following errors when using the gpupdate /force command:

PingComputer: Second send failed with 11010

So, this started me thinking about Ping and the use of ICMP between the array members and the Active Directory Domain Controllers. If you refer back to my previous blog article here you will see that PING (ICMP Request -Type 8) is included within the required protocol list. Therefore, ICMP should not be blocked by firewall policy processing on the back firewall.

I confirmed this was not the case using a simple Ping command (which uses a default packet size of 32 bytes) from the array members to one of the Domain Controllers:

Ping <DC IP Address>

Result: Reply from <DC IP Address>: bytes=32 …

Therefore, we had successful ping replies.

More investigation led me to the follow Microsoft KB article:

How to troubleshoot Group Policy object processing failures that occur across multiple forests (KB910206)

Looking at Scenario 2 in this KB article seemed to pinpoint my exact symptoms. In particular, the following text intrigued me:

Group Policy processing stops if large fragmented ICMP packets are not allowed on the network because of slow link detection. When this occurs, Microsoft Windows XP does not detect a fast link or a slow link. Instead Windows XP fails out of Group Policy processing. Only a generic USERENV 1054 event is logged in the Application event log. This problem affects both computer and user policies.

This got me thinking that maybe the back firewall wasn’t blocking ICMP traffic, but was in fact blocking (or dropping) large or fragmented ICMP packets. A quick test using the –l Ping switch with a packet size of 2048 bytes (the default value used by Group Policy processing) confirmed that ICMP was in fact being blocked when using 2048 byte packet sizes:

Ping <DC IP Address> –l 2048

Result: Request timed out.

Lowering the packet size to 1024 made no difference, still blocked:

Ping <DC IP Address> –l 1024

Result: Request timed out.

Finally, lowering the packet size to 500 bytes resulted in successful replies:

Ping <DC IP Address> –l 500

Result: Reply from <DC IP Address>: bytes=500 …

The only real conclusion I could draw from this process of elimination was that the back firewall was configured to block ICMP packets above a certain size or fragmented ICMP packets; most likely packets above 512k. I assumed this was a security feature or some form of DOS protection on the back firewall, and a reasonable measure.

Disabling slow link detection as described in the above KB article (KB910206) didn’t appear to help though and I was still receiving event log errors. I don’t give up easily though!

So, I looked for a solution to reduce the actual ICMP packet sizes used in Group Policy processing. A bit more investigation resulted in the following Microsoft KB article:

Group Policies may not apply because of network ICMP policies (KB816045)

After creating the PingBufferSize registry key and setting this to a value of 500, I used the gpupdate /update command again, and this time I received a successful application event log entry of:

Event ID: 1704
Source: SceCli
Type: Information
Description:
Security policy in the Group policy objects has been applied successfully.

From that point on, the Userenv errors stopped occurring and Group Policy was applied successfully…case closed I think ;)

The only thing left to do was to apply the registry change to all array members to ensure consistent Group Policy process across the entire array.

As normal, the problem was not actually caused by ISA Server, but by the surrounding network infrastructure. It might have been nice if the back firewall administrator had told me about this ICMP limitation from day one, but they likely didn't even know :)

Hope this helps someone else with a similar problem…

How to Clear the Change Tracking Log in ISA Server 2006 SP1

For reasons I won’t go into, I recently wanted to clear the Change Tracking log and remove all previous entries from the list. There appeared to be no obvious way to achieve this in the GUI(probably for good reason) but I wanted to share with the community how it can be achieved anyhow…

So, below we can see our example Change Tracking log full of entries we no longer need. This can be seen by looking at the Change Tracking tab under the Monitoring node:

changetracking1

Clearing the log then involves two simple steps…

Step 1: Modify Existing Change Tracking Limit

Below is a screenshot of the default Change Tracking configuration:

changetracking 

This needs to be reconfigured as follows:

For Standard Edition

  • In the console tree of ISA Server Management, expand Microsoft Internet Security and Acceleration Server 2006, expand the <Server Name> node, and then click Monitoring.
  • Select the Change Tracking tab.
  • From the Tasks tab, click Configure Change Tracking.
  • Change the default entries limit set from 1000 (the default) to 0. If you have a value different to 1000, change this to 0 anyhow.
  • Click OK to close the Change Tracking window.
  • Click the Apply button in the details pane to save the changes and update the configuration.

For Enterprise Edition

  • In the console tree of ISA Server Management, expand Microsoft Internet Security and Acceleration Server 2006. Right click the Enterprise node and select Properties.
  • Select the Change Tracking tab.
  • Change the default entries limit set from 1000 (the default) to 0. If you have a value different to 1000, change this to 0 anyway.
  • Click OK to close the Enterprise Properties window.
  • Click the Apply button in the details pane to save the changes and update the configuration.
Please Note: If you are using Enterprise Edition, you will need ISA Server Enterprise Administrator permissions to make the above changes.

We should then have:

changetracking2

Step 2: Restore Original Change Tracking Limit

In order to restore the original Change Tracking limit, we simply reverse the procedure defined above:

For Standard Edition

  • In the console tree of ISA Server Management, expand Microsoft Internet Security and Acceleration Server 2006, expand the <Server Name> node, and then click Monitoring.
  • Select the Change Tracking tab.
  • From the Tasks tab, click Configure Change Tracking.
  • Change the default entries limit set from 0 to 1000 (the default). If you previously had a value different to 1000, restore the original value.
  • Click OK to close the Change Tracking window.
  • Click the Apply button in the details pane to save the changes and update the configuration.

For Enterprise Edition

  • In the console tree of ISA Server Management, expand Microsoft Internet Security and Acceleration Server 2006. Right click the Enterprise node and select Properties.
  • Select the Change Tracking tab.
  • Change the default entries limit set from 0 to 1000 (the default). If you previously had a value different to 1000, restore the original value.
  • Click OK to close the Enterprise Properties window.
  • Click the Apply button in the details pane to save the changes and update the configuration.

We should then be back to the default Change Tracking configuration:

changetracking

If we now look at the Change Tracking tab under the Monitoring node we should see the following:

changetracking4

So, there we go, a nice blank change tracking log in two easy steps – simple! ;)

Please Note: For Enterprise edition, the above procedure assumes Change Tracking is enabled at the ISA Server enterprise level and will consequently clear all array-level Change Tracking logs at the same time. If you have change tracking enabled at the array-level only, and want to clear a single array Change Tracking log, this can be done by applying the same concept to the array object as opposed to the entire ISA Server enterprise.

If you prefer a command line approach as opposed to the GUI, it is also possible to clear the log using scripts as detailed here:

Change Tracking - Log Management via Scripts

It is also interesting to note that the upcoming new release of ISA Server, named Threat Management Gateway (TMG), will support an improved Change Tracking feature. Part of this update will include a dedicated API with a ClearLog function. Read more here:

Change Tracking in TMG

Hope this is useful…