Monday, 20 April 2009

ISA Server 2006/2004 Configuration Storage Server Frequently Asked Questions

Most of my ISA Server deployments seem to be based around Enterprise Edition nowadays, which is a good sign that more Enterprise customers are starting to see the benefits of ISA Server as an advanced application-layer firewall. As I use Enterprise Edition regularly, it is easy to forget that many of the concepts that I take for granted are actually quite new to people who are more in tune with ISA Server Standard Edition. The ISA Sever Enterprise Edition integrated NLB feature is a good example here and why I wrote the NLB resource guide provided here, as many people seemed to struggle with this particular area. Another area that has been raising a few questions on the forums lately is that of Configuration Storage server design and general concepts. A lot of the information available from Microsoft is provided as part of the ISA Server 2004 product documentation set, and hence is often missed by people looking at the ISA Server 2006 documentation. Very little has changed (if anything) in terms of Configuration Storage servers between the two versions, so all of this existing documentation is still hugely valid, yet often overlooked.

Please Note: Threat Management Gateway (TMG), the latest incarnation of ISA Server, is only just around the corner now. However, I think ISA Server 2006 will be around for some time yet (especially as it is the latest version of ISA Server to be Common Criteria certified) and hence a good understanding of the key concepts for this version will still be of value to many people.

As with the NLB Resource Guide, I have created this article in a ‘FAQ style’ as I think this is a good readability format (tell me if it’s not!) but which also allows me to answer forums posts with “…go read Question #” answers, where applicable.

Anyhow, let’s get started!

Question 1: What is a Configuration Storage server?

A Configuration Storage server is used to store the configuration data for the ISA enterprise, which includes array data for all arrays within the ISA enterprise. The Configuration Storage server uses Active Directory Application Mode (ADAM) as the technology of choice for this data storage.

There can be multiple Configuration Storage servers in the enterprise, each holding an exact replica of the enterprise configuration, which also includes all array data.

Array members communicate with the Configuration Storage server to get up-to-date configuration information. Each ISA Server array member has a local copy of its configuration that is a replica of the configuration stored on the CS role – this replica is stored locally in the registry as a local ‘cached copy’.

Each array points to a specific primary Configuration Storage server from where it gets the latest configuration. You can also specify an alternate Configuration Storage server, which is used when case the primary Configuration Storage server in unavailable for any reason.

It should be noted that the Configuration Storage server doesn't provide any other services apart from storage (and replication) of the enterprise configuration data. It is sometimes thought that the Configuration Storage server provides some form of authentication service (similar to Microsoft IAS) but this is not the case. It’s an important role in the architecture, but ultimately, quite a simple one! :)

Please Note: It is not possible ‘connect’ different ISA enterprises in the same way that you can with Active Directory forest trusts. Consequently, the enterprise boundary needs to be considered to ensure that the entire system can be managed from a single management console; this is one of the primary reasons for choosing Enterprise Edition after all. In reality, it is sometimes actually beneficial to have separate enterprises. Due to their inherent isolation, they obviously present a good security boundary, so can be useful when organisation wish to completely separate the administration model or locate the array members in a highly-secure environment – for example, maybe the solution will be deployed into a hosted environment and managed by a third party…

Question 2: How many Configuration Storage servers should I install?

There are many things to consider when planning how many Configuration Storage servers are required, like:

  • How many do I need?
  • Where should they be located?
  • How should they be configured?

These include elements like network speed, need for reliability, and the number of ISA Server array members that will connected to the Configuration Storage server. As a single Configuration Storage server is able to support approximately 40 array members, this is not normally a major design driver in all but the largest of deployments.

In a highly-connected network, the number of Configuration Storage servers is normally determined by the number of arrays and the required level of fault tolerance for each array. Just like a domain controller, the Configuration Storage server should ideally be located in close proximity to the array members to minimise network traffic and potential connectivity issues.

In order to implement Enterprise Edition, you need at least one Configuration Storage server; however in order to provide redundancy for this vital role, it is recommended to use a minimum of two servers. This ensures that in the event of failure of the primary Configuration Storage server, it will still be possible to manage the enterprise using an alternate Configuration Storage server.

Please Note: The use of two, or more, Configuration Storage servers assumes that they are domain joined (this is covered in more detail in Question 14).

If you only plan to deploy a single array within the enterprise, it would not normally be necessary to deploy more than two Configuration Storage servers, as it is only possible to define primary and alternate Configuration Storage servers (e.g. two in total) for a single array. The exception to this would be some form of warm-standby Configuration Storage server (or servers) that could be located at a secondary location in the event of a disaster recovery scenario.   

Question 3: Where should the Configuration Storage server be installed?

This question is often debated, but the best recommendation is to install the Configuration Storage server role on dedicated member servers.

Yes, you can co-locate the role onto existing servers, or even install it on the actual array members themselves, but you should only really do this if you are happy to accept the associated risks of co-location (this is covered in more detail in Question 5) or you cannot use dedicated servers due to budget/cost constraints. 

In high-security deployments I have sometimes installed the Configuration Storage server on dedicated servers in their own ISA Server protected network (an ISA perimeter network, or DMZ, if you will). This ensures that all connectivity to the Configuration Storage server is protected by the ISA Server array itself (even from “trusted” internal networks) and provides a good threat mitigation model for such vital servers.

Deployment considerations are also discussed in the topic Configuration Storage Server Deployment Guidelines in this document.

Question 4: So, what does a typical Configuration Storage design look like then?

Well, there is never anything “typical” in the world of IT, but lets try and pick a suitable multi-site example environment.

Consider an organisation with three data centres located globally, let’s say London, Miami and Singapore. All three data centres are connected by fast private network links using MPLS. Let’s assume ISA Server will be deployed at all three data centres as an edge application-layer firewall. A likely design for this environment is shown below:

image 

Key features of this design include:

  • A single, global ISA enterprise is used across all locations. This enterprise contains three arrays; one for each data centre location.
  • Dedicated Configuration Storage servers are used in each data centre location.
  • Configuration Storage servers are located local to the array members themselves.
  • Both primary and alternate Configuration Storage servers are located at each data centre to provide fault tolerance in the event of Configuration Storage server failure.

Assuming that London is the primary data centre and hardware cost constraints limited the solution to single dedicated servers in the other data centres, it would also be feasible to configure the Miami and Singapore arrays to define the alternate Configuration Storage servers as LONCSS1 and LONCSS2, respectively. This provides an improved level of fault tolerance for Miami and Singapore through the use of failover to remotely located Configuration Storage servers. Consequently, the design could look like this:

image

The first model provides a better availability model, but this ‘diluted’ model is still viable and would save the cost of two additional servers. However, there is obviously a stronger dependency on the intra-data centre network connectivity with this model, which may need consideration…

Question 5: What issues do I need to consider if I install the Configuration Storage server role on array members when using NLB?

Installation of the Configuration Storage server on array members is a supported configuration, but often not recommended, as discussed in Question 4. When using this deployment model with multi-homed array members and NLB, it is recommended to make changes to ensure correct and stable operation of the Configuration Storage server. These changes are discussed within Question 9 of my NLB FAQ which can be found here. A quick synopsis is provided below:

Yes, if you are using Windows based authentication for intra-array credentials.
In the event that the CSS role has been installed on the actual array members (although this is not recommended as best practice) it is necessary to make several changes before NLB can be enabled on any of the array networks. These changes are necessary due to the use of Kerberos based authentication between array members and the CSS role which necessitates the use of correct Service Principal Names (SPNs). If these SPN changes are not made, a request to connect from an array member to the Configuration Storage server may fail.
The required configuration to regain normal operation is provided in the Multiple Network Adapters and NLB section of this document Network Load Balancing Integration Concepts for Microsoft Internet Security and Acceleration (ISA) Server 2006

Question 6: What happens when the primary Configuration Storage server is not available?

By default, each array member polls its current Configuration Storage server for configuration changes every 15 seconds (the default polling time). If the primary Configuration Storage server is unavailable for 30 minutes (the default fallback delay), the array member switches to the alternate Configuration Storage server (assuming one has been defined).

After using the alternate Configuration Storage server for six hours (the default primary testing delay), the array member periodically tests for connectivity with the primary Configuration Storage server. After the primary Configuration Storage server is continuously available for 10 minutes (the default primary stabilisation time), the array member then switches back to the primary.

This entire process is completely automated and transparent, hence you will normally only be aware of it happening if you attempt to use the ISA Management console connecting to the failed Configuration Storage server or look at the alerts tab regularly (of course you do!). For those with SCOM/MOM, you should also receive alerts via the SCOM/MOM console if you are using the ISA Server management pack; assuming you are not using ISA Server 2006 Service Pack 1 that is, because this broke the SCOM management pack…poor show guys :(

Question 7: What happens when both the primary and alternate Configuration Storage servers are not available?

If both the primary and the alternate Configuration Storage servers fail, the ISA Server Management console will not provide access to any ISA Server functionality, because it requires a connection to a working Configuration Storage server. At the same time, the ISA Server array members will continue to provide firewall, VPN, and proxy services based on the last known configuration that was received from the Configuration Storage server. However, you will not be able to monitor or change any configuration until at least one of the Configuration Storage servers is restored, or until you connect the computer running ISA Server services and the ISA Management console to a different (working) Configuration Storage server in the enterprise.

The key takeaway here is that everything carries on working even when all Configuration Storage servers are unavailable; it’s just that the firewall nodes become temporarily unmanageable from the enterprise. Therefore, they are still diligently protecting your network, if not manageable, which is a good thing ;) 

Question 8: Is it possible to modify the default timeout values for Configuration Storage server failover, fallback and stabilisation?

Yes, this is possible, but not via the ISA Management console. In order to amend the default values, it is necessary to apply changes using a custom VBS script provided by Microsoft.

It is possible to change the following parameters via the script, with the key parameters defined below:

  • FallbackDelay: This property gets or sets the time (in minutes) after which an array member will start using the other Configuration Storage server when the current Configuration Storage server is unavailable. Its default value is 30 minutes, and its range of permissible values is from 1 through 99,999.
  • PrimaryTestingDelay: This property gets or sets the time (in minutes) after which an array member will start testing for connectivity with the primary Configuration Storage server when using the alternate Configuration Storage server. Its default value is 360 minutes (6 hours), and its range of permissible values is from 1 through 99,999.
  • PrimaryStabilizationDelay: This property gets or sets the time (in minutes) during which the primary Configuration Storage server must be continuously available before an array member will reconnect to it. Its default value is 10 minutes, and its range of permissible values is from 1 through 99,999.

Additional information (and the actual script) can be found in the following Microsoft article:

Setting Configuration Storage Server Delay Times

Please Note: Although this article references ISA Server 2004, it is still appropriate for ISA Server 2006.

I always struggle to find this document (yeah I’m useless with bookmarks!) and many people have never even seen it from what I can tell, so I think it is useful to mention it here. 

Question 9: How do I secure (harden) my Configuration Storage server?

Depending upon your chosen Configuration Storage server deployment method, your mileage may vary here. For obvious reasons, it is easiest to secure the server when it is deployed on a dedicated member server that provides no other role or service. The easiest (and most Microsoft supportable) approach is to use the Security Configuration Wizard available as part of Windows Server 2003 Service Pack 1. With an SCW update specifically for ISA Server 2006 applied (from here) the SCW tool is able to understand and define a specific server role based upon the Configuration Storage server function, and consequently configure the dependent services and Windows firewall configuration as necessary for that role.

If the Configuration Storage server is co-located on an existing member server which provide additional server roles, or on the actual array members themselves, then this will need to be factored into the SCW application process.  

Before getting SCW whipped up, it is also important to consider the placement of the server within the network. For example, a Configuration Storage server that is located within an ISA protected network (a dedicated ISA perimeter network maybe) is already isolated against many attacks, subsequently the benefits provided by SCW (or a similar host hardening process) are only going to provide marginal benefit. I not saying you shouldn’t do it as part of an overall defence in depth strategy, but it’s security exposure is already greatly reduced anyhow. However, if the server is placed on an ‘internal’ network that could be potentially dangerous (like a University campus network for example!) then hardening measures will be much more important.

A good starting place when looking at security measures for the Configuration Storage server role is the ISA Server 2006 Security Guide available from here.

Please Note: If there is enough interest, I can probably do a walkthrough article on using SCW for a Configuration Storage server, but it is relatively straightforward to be honest, assuming you know how to drive the SCW tool itself.

Question 10: How do I minimise replication traffic between Configuration Storage servers in the same ISA Server enterprise?

In a highly-connected environment, replication between Configuration Storage servers is often negligible and often best left unchanged. By default, all Configuration servers are installed into as single site; namely the default ADAM site called Default-First-Site-Name. Consequently, there is an assumption that all replication will occur over well connected networks within minimal need to consider bandwidth utilisation and hence aim to optimise traffic flow biased for speed.

However, in certain scenarios, it may be useful to consider the use of ADAM sites and an inter-site replication model (rather than the default intra-site model) for ISA enterprise data. This can be accomplished by creating custom ADAM sites and assigning Configuration Storage servers to individual sites in order to distinguish between different parts of the network. To achieve this, you can use the Adam Sites Tool (AdamSites.exe) available from here to create and configure ADAM sites.

At a high level, this process involves:

  • Use the AdamSites.exe tool to create a site.
  • Use the AdamSites.exe tool to create a site link to connect this site to other sites in the ADAM deployment.
  • Use the AdamSites.exe tool to move the Configuration Storage server to the new site.

So, considering our example detailed in Question 4, this would likely involve a site topology as follows:

image 

The ADAM Sites tool and process is not documented as well as it could be IMHO, and would benefit from a good working example. Hence, I have provided a basic walkthrough (using the example environment above) in my blog article available here.

Please Note: The design of ADAM sites and Configuration Storage servers is analogous to Active Directory sites and domain controllers. Hence a similar design and approach can often be considered if you have a good working knowledge of Active Directory.

Question 11: Why does the ISA Management Console fail to automatically reconnect to the alternate Configuration Storage server?

Unlike the firewall service, the ISA Server Management console is configured to reference a specific Configuration Storage server. Hence in the event that this server is not available, it is necessary to manually reconfigure the console to use an alternate Configuration Storage server. 

Question 12: What hardware specification is required for the Configuration Storage server?

In general, the Configuration Storage server does not require a particularly powerful server. An entry level server with a single core 2GHz+ processor and 1 GB RAM is normally sufficient to support approximately 40 array members; hence this often fulfils most requirements.

In my experience, there are no real disk subsystem requirements apart from the sensible use of data redundancy with something like RAID1.

Virtualise me? :) The Configuration Storage server is also an ideal candidate for virtualisation if you run VMware ESX or Microsoft Hyper-V virtualised environments. It’s virtual resources overhead is normally pretty low too.

Question 13: What should I do if my Configuration Storage server is stolen?

The Configuration Storage server and each array member contain encrypted, confidential information about the other array members and about the ISA Server enterprise. The array member also has a private/pubic key pair that is used to decrypt the information. For this reason, if a Configuration Storage server (or an array member for that matter) confidential information about all the array members is potentially at risk. The key items to consider include:

  • Modify all confidential information on all the other array members. Confidential information includes user credential passwords (for example, used for logging on to a computer running SQL Server), Remote Authentication Dial-In User Service (RADIUS) shared secrets, or preshared Internet Protocol security (IPSec) keys.
  • Revoke any certificates that were installed on the stolen server. These certificates can be certificates that were installed on array members for Web publishing or site-to-site VPN authentication etc.
  • In addition, when a Configuration Storage server is stolen and there is a replica Configuration Storage server configured, the stolen Configuration Storage server should be removed from the ADAM server configuration as a replica set using the RepAdmin tool.

Question 14: Should my Configuration Storage server be domain joined?

This is another often debated question and closely linked to a similar question of “Should my ISA Server array members be domain joined?”. Tom Shinder does a great job of covering the key reasons for domain joining ISA Server here which includes consideration of the Configuration Storage server role too. To summarise the key issues specifically related to Configuration Storage servers:

  • When deployed in workgroup mode, it is only possible to implement a single Configuration Storage. This architecture obviously provides a poor level for fault tolerance. The Configuration Storage servers provide a vital role in the ISA Server Enterprise architecture and subsequently need to be designed to provide fault tolerance in the event of server failure.
  • When deployed in workgroup mode, it requires the use of local accounts which are unmanaged; often with non-expiring passwords :(
  • When deployed in workgroup mode, server certificates are required on Configuration Storage servers and an associated PKI trust model needs to be in place. This is often much harder (for most ISA admins) to achieve than it sounds ;)

Please Note: Workgroup mode is the name given to a non-domain joined deployment.

Consequently, I would recommend that all Configuration Storage servers are domain joined at all times and this is my personally preferred deployment model.

Question 15: What should I consider if my Configuration Storage server is installed on an Active Directory Domain Controller?

Although not ideal, it is possible in certain scenarios to deploy a Configuration Storage server on a Domain Controller role.  Although installation of ISA Server array members on a Domain Controller is not supported, installation of the Configuration Storage server role is supported. In this scenario, it is necessary to consider the following aspects:

  • Configure the Configuration Storage server service (ISASTGCTRL) to run under the context of a dedicated service account which has user level permissions.
  • Amend Domain Controller object permissions to allow SCP registration. This greatly reduces event log noise (Event ID: 2537) from failed SCP registrations.
  • Amend Domain Controller objects to provide Generate Security Audits permission. This again greatly reduces event log noise (Event ID: 2521) of failed auditing alerts.

These areas are covered in the excellent blog entry article here.

Question 16: What do I do if neither of my primary and alternate Configuration Storage servers are available and they are going to be offline for some time?

When faced with this scenario, it is often necessary to deploy a new configuration server and restore the entire enterprise configuration. However, as the array members cannot connect to a Configuration Storage server, it is not possible to update the array properties to define the replacement Configuration Storage server (a nice catch 22!). In order to solve this problem, it is possible to use a custom VBS script which is provided by Microsoft on the ISA Server CD. This script is called ChangeStorageServer.vbs and more information is available in the Configuration Storage Servers are Not Accessible section of this document.

Question 17: Where can I find more information on Configuration Storage server related deployment guidelines, troubleshooting and security?

The following handy documents (in addition to this FAQ, of course!) should help…

Deployment Guidelines for ISA Server 2004 Enterprise Edition

Troubleshooting Configuration Storage Servers in ISA Server 2004 Enterprise Edition

ISA Server 2006 Security Guide

Transferring Configuration Storage Server FSMO Roles

Right, I think that is a good array (excuse the pun!) of questions for now which hopefully should answer most of the common queries for people moving to ISA Server 2004/2006 Enterprise Edition for the first time.

7 comments:

  1. Excellent! We are doing an initial deployment of an ISA Array with two members and for various reasons, we are forced to have our config storage on one array member. Is it possible to have the config storage server installed on each array member, purely for fault tolerance.

    ReplyDelete
  2. >> Anonymous

    Although not ideal, installing the CSS on array members is viable. If you have two or more array members, I would always install the CSS role onto at least two of them.

    So, to answer your question; Yes it is possible. Furthermore, if you have to put the CSS co-located on array members I would still recommend having two CSS roles for fault tolerance.

    ReplyDelete
  3. I'm trying to upgrade my isa 2006 ent server's hardware so brought a new server pc. I configured my isa 2006 in a AD Domain Machine. That why I took a system state of older PC. After restoring the system state I am unable to start my management console of ISA and its failed to connect with storage server so now I'm in a deef trouble. My os is server 2003 sp 2. Can u help me?

    ReplyDelete
  4. I really learned a lot from these frequently asked questions that you posted. Thanks a lot for sharing.

    document storage

    ReplyDelete
  5. This is just what I needed. Thank you very much.
    usa vpn

    ReplyDelete
  6. Great post. Thanks you for sharing this.

    ReplyDelete