BizTalk and configuration

In many integration scenarios you need some configuration values to make your integration process work smoothly and in an agile way. For instance you may not want to recompile your code just because some endpoint need an increased timeout value.

So what are the options to get these configuration values into your process? (i.e. orchestration)

1 - Configuration file

Normally then coding in .Net you put configuration in a config file. For the .exe processes it's the app.config file that's used for this. For web services and WCF services hosted in IIS you got the web.config file.

Biztalk host instances, that run your orchestration processes, are nothing more than windows services, that in turn are exe processes that run for a long time and without user interaction.
So for each Biztalk installation on a server you got a configuration file that is loaded into memory by each instances of your Biztalk hosts. This file is located in the biztalk installation directory, usually: C:\Program Files\Microsoft BizTalk Server XXXX\BTSNTSvc.exe.config
on each server where you have biztalk installed!

You could add keys to this file and pick them up in your orchestrations, but it's RISKY!
Ther're several reasons for this:

1 - You need to change the config file on every server that runs instances of your biztalk hosts.
2 - One single syntax error in this file and ALL your host instances on that server goes DOWN.
3 - If you upgrade your platform and do a new installation you need to reenter these configuration values.

So let's go ahead and look for other options!

2 - Windows Registry

The Windows Registry is a hierarchical database that stores configuration settings and options. You access the registry via the start menu just typing 'regedit' from the 'run' command window (XP, server 2003) or 'search' command text window (windows 7, server 2008). The structure looks like folders (called: keys) and you can create your own keys, subkeys and values.
You access your keys / values from .Net code. From within an orchestration you can call a 'Helper' .Net assembly and pick up the values at runtime.

As with the configuration file option you need to change the values on EACH server then you want to make a change. But in case of a syntax error it won't cause your biztalk runtime to go down. Only the related application / orchestration using that particular value will be affected.
The values are updated in runtime as opposed to the configuration file case where it needs to be reloaded in memory (i.e host instances need to be recycled).

Due to the cumbersome structure of regedit and to the fact that you can only put simple key / value pairs, I think one should try to limit the number of configuration values in the registry.

If you need complex configuration (i.e composed keys and a generic store for configuration values) you should use a relational database. That leads us to the third configuration option:

3 - SQL server database

Sooner or later I guess? that most BizTalk installations end up with a custom configuration database. From my own experience I can see that it's often used to keep temporary data (for instance data that at some point will be batched and delivered), state data and of course configuration.
With a relational database you can store any information. And it may be the only choice if you have relational data and want to store it in a reasonble way.
To write / retrieve data from the database just use the sql adapter or the newer WCFSQL-adapter.
The drawback of using a relational database is SPEED. The data has to be picked up from disk and if you use an adapter (ie not doing direct ODBC calls) you will in addition have persistence points before each configuration send request.

There's also a build in feature in BizTalk called "cross references". It consists of some relational tables in BizTalk mgmt database. Typically you would use these tables to do lookups in mappings but nothing prevents you from putting other kind of configuration here. The value lookups in mappings use some caching feature to improve performance while the id lookups are not cached. The length of the value field in the cross ref tables is limited to 50 chars.

So is there a faster way?

4 - Sigle sign on (SSO) store

The SSO store is already present in every BizTalk installation and is used by the adapters to store configuration so it's quite a natural place to keep custom configuration. You can create your own SSO applications in which you have key / value pairs. This can be done via a MMC snap in. To retrive the configuration you need to write some code.
(More details about this option can be found in the "Professional biztalk server 2006" book by Daren Jefford, Kevin B Smith and Ewan Fairweather (page 594- 595))
The configuration stored in SSO is secured and present in memory on the biztalk servers.
However you're still limited to key value pair kind of storage and in some situations it's not enough.

So is there any option that's both fast and relational?

5 - BRE (Business rule engine)

The BRE is typically used to evaluate XML data at runtime. But you can, with a bit of coding, use long-term facts as a distributed configuration store in your BizTalk server park!
You can for instance have your configuration tables in some SQL database and at runtime (first execution cycle of the policy) put these tables as typed data tables in memory on your servers. You can then implement logic to update your cache at regular intervals and / or then data has been changed / added.
With the BRE you could create a relational, distributed configuration store in memory that is quite dynamic. (you will not need to restart your host instances but instead wait for the refresh cycle to get new configuration up into memory)
There's a little coding to achive this. If you want to dig deaper the implementation is described in the "Professional biztalk server 2006" book by Daren Jefford, Kevin B Smith and Ewan Fairweather (page 354- 355).

Conclusion:

There're many different technologies that can be used to store configuration data for use in a Biztalk solution / installation.
The choice of technology to use is highly dependant of your needs (performance, complexity of configuration data, ease of use...)
My own opinion is that one should avoid to use the biztalk host configuration file. I also think that the extra effort during the build up of a new biztalk installation to outline the strategy and policy to use for configuration data is well worth.