BizTalk and configuration

In many integration scenarios you need some configuration values to make your integration process work smoothly and in an agile way. For instance you may not want to recompile your code just because some endpoint need an increased timeout value.

So what are the options to get these configuration values into your process? (i.e. orchestration)

1 - Configuration file

Normally then coding in .Net you put configuration in a config file. For the .exe processes it's the app.config file that's used for this. For web services and WCF services hosted in IIS you got the web.config file.

Biztalk host instances, that run your orchestration processes, are nothing more than windows services, that in turn are exe processes that run for a long time and without user interaction.
So for each Biztalk installation on a server you got a configuration file that is loaded into memory by each instances of your Biztalk hosts. This file is located in the biztalk installation directory, usually: C:\Program Files\Microsoft BizTalk Server XXXX\BTSNTSvc.exe.config
on each server where you have biztalk installed!

You could add keys to this file and pick them up in your orchestrations, but it's RISKY!
Ther're several reasons for this:

1 - You need to change the config file on every server that runs instances of your biztalk hosts.
2 - One single syntax error in this file and ALL your host instances on that server goes DOWN.
3 - If you upgrade your platform and do a new installation you need to reenter these configuration values.

So let's go ahead and look for other options!

2 - Windows Registry

The Windows Registry is a hierarchical database that stores configuration settings and options. You access the registry via the start menu just typing 'regedit' from the 'run' command window (XP, server 2003) or 'search' command text window (windows 7, server 2008). The structure looks like folders (called: keys) and you can create your own keys, subkeys and values.
You access your keys / values from .Net code. From within an orchestration you can call a 'Helper' .Net assembly and pick up the values at runtime.

As with the configuration file option you need to change the values on EACH server then you want to make a change. But in case of a syntax error it won't cause your biztalk runtime to go down. Only the related application / orchestration using that particular value will be affected.
The values are updated in runtime as opposed to the configuration file case where it needs to be reloaded in memory (i.e host instances need to be recycled).

Due to the cumbersome structure of regedit and to the fact that you can only put simple key / value pairs, I think one should try to limit the number of configuration values in the registry.

If you need complex configuration (i.e composed keys and a generic store for configuration values) you should use a relational database. That leads us to the third configuration option:

3 - SQL server database

Sooner or later I guess? that most BizTalk installations end up with a custom configuration database. From my own experience I can see that it's often used to keep temporary data (for instance data that at some point will be batched and delivered), state data and of course configuration.
With a relational database you can store any information. And it may be the only choice if you have relational data and want to store it in a reasonble way.
To write / retrieve data from the database just use the sql adapter or the newer WCFSQL-adapter.
The drawback of using a relational database is SPEED. The data has to be picked up from disk and if you use an adapter (ie not doing direct ODBC calls) you will in addition have persistence points before each configuration send request.

There's also a build in feature in BizTalk called "cross references". It consists of some relational tables in BizTalk mgmt database. Typically you would use these tables to do lookups in mappings but nothing prevents you from putting other kind of configuration here. The value lookups in mappings use some caching feature to improve performance while the id lookups are not cached. The length of the value field in the cross ref tables is limited to 50 chars.

So is there a faster way?

4 - Sigle sign on (SSO) store

The SSO store is already present in every BizTalk installation and is used by the adapters to store configuration so it's quite a natural place to keep custom configuration. You can create your own SSO applications in which you have key / value pairs. This can be done via a MMC snap in. To retrive the configuration you need to write some code.
(More details about this option can be found in the "Professional biztalk server 2006" book by Daren Jefford, Kevin B Smith and Ewan Fairweather (page 594- 595))
The configuration stored in SSO is secured and present in memory on the biztalk servers.
However you're still limited to key value pair kind of storage and in some situations it's not enough.

So is there any option that's both fast and relational?

5 - BRE (Business rule engine)

The BRE is typically used to evaluate XML data at runtime. But you can, with a bit of coding, use long-term facts as a distributed configuration store in your BizTalk server park!
You can for instance have your configuration tables in some SQL database and at runtime (first execution cycle of the policy) put these tables as typed data tables in memory on your servers. You can then implement logic to update your cache at regular intervals and / or then data has been changed / added.
With the BRE you could create a relational, distributed configuration store in memory that is quite dynamic. (you will not need to restart your host instances but instead wait for the refresh cycle to get new configuration up into memory)
There's a little coding to achive this. If you want to dig deaper the implementation is described in the "Professional biztalk server 2006" book by Daren Jefford, Kevin B Smith and Ewan Fairweather (page 354- 355).

Conclusion:

There're many different technologies that can be used to store configuration data for use in a Biztalk solution / installation.
The choice of technology to use is highly dependant of your needs (performance, complexity of configuration data, ease of use...)
My own opinion is that one should avoid to use the biztalk host configuration file. I also think that the extra effort during the build up of a new biztalk installation to outline the strategy and policy to use for configuration data is well worth.

Gracefully handling orchestration errors

Errors happen, it's sad but true. Services can become nonresponsive.

This "article" is about how to handle this when using biztalk server to orchestrate an integration flow.
The image below show a very simple orchestration flow: A message trigger the flow, an external WCF service is called, the response from the service call is fetched and delivered to some endpoint... i.e a really simple integration flow delivering some data between disparate systems. So what could go wrong? In this simple case not so much, the most critical part of this flow is the external service call made from the orchestration. This service could be unavaileble, the network could be down.
What happens if we don't write any dedicated code to handle this?
Well the send port calling the external service will retry according to the configured retry values (the default in biztalk is 4 retries in total 5 minutes appart). If these retries all fail the send port will be suspended along with the orchestration. That's ok -> then the network or service is upp again we just resume the port... or?
Unfortunately it doesn't work the way we would hope. The send port will indeed send away the message to the service once resumed but the response message from the service won't correlate back to our orchestration and hence the integration flow will be broken. No response payload will be delivered to our final destination (endpoint).

The design below is a way to solve this issue.

1 - first we must catch the exception thrown back from the failed service call.
(could be a custom service exception, a soap exception or System.Exception (when you'll catch all errors) )

2 - Next we suspend the orchestration with a suspend shape and with a relevant error message to the support operator. (a suspend flag has to be set when you catch the error which will be used to decide if an error happend in an if shape below the service call)

3 - A loop shape is used to retry the service call if the operator decides to resume the orchestration. The logic to exit the loop is simply handle by some flag that is reset then the service call is resumed.

A new "service message request" will be send to the message box resulting in a new instance of a send port calling the service. If the service call is successfull the response will be handled by the orchestration (thus resuming the integration flow) and (in this case) finally delivered to the endpoint.

You will end up with a least one suspended send port in the administration console that has to be manually killed once the orchestration is resumed. That's the best way I've found to recover from failed service calls (that exceeds the retry intervall on the ports).

If there's even a better way let me know!

Below you can see the orchestration design in this particular simple integration flow and how it looks in the tracking tool when a soap error is catched from a failed service call.

Orchestration design

Tracking tool -> catch SOAP error