BizTalk and configuration

In many integration scenarios you need some configuration values to make your integration process work smoothly and in an agile way. For instance you may not want to recompile your code just because some endpoint need an increased timeout value.

So what are the options to get these configuration values into your process? (i.e. orchestration)

1 - Configuration file

Normally then coding in .Net you put configuration in a config file. For the .exe processes it's the app.config file that's used for this. For web services and WCF services hosted in IIS you got the web.config file.

Biztalk host instances, that run your orchestration processes, are nothing more than windows services, that in turn are exe processes that run for a long time and without user interaction.
So for each Biztalk installation on a server you got a configuration file that is loaded into memory by each instances of your Biztalk hosts. This file is located in the biztalk installation directory, usually: C:\Program Files\Microsoft BizTalk Server XXXX\BTSNTSvc.exe.config
on each server where you have biztalk installed!

You could add keys to this file and pick them up in your orchestrations, but it's RISKY!
Ther're several reasons for this:

1 - You need to change the config file on every server that runs instances of your biztalk hosts.
2 - One single syntax error in this file and ALL your host instances on that server goes DOWN.
3 - If you upgrade your platform and do a new installation you need to reenter these configuration values.

So let's go ahead and look for other options!

2 - Windows Registry

The Windows Registry is a hierarchical database that stores configuration settings and options. You access the registry via the start menu just typing 'regedit' from the 'run' command window (XP, server 2003) or 'search' command text window (windows 7, server 2008). The structure looks like folders (called: keys) and you can create your own keys, subkeys and values.
You access your keys / values from .Net code. From within an orchestration you can call a 'Helper' .Net assembly and pick up the values at runtime.

As with the configuration file option you need to change the values on EACH server then you want to make a change. But in case of a syntax error it won't cause your biztalk runtime to go down. Only the related application / orchestration using that particular value will be affected.
The values are updated in runtime as opposed to the configuration file case where it needs to be reloaded in memory (i.e host instances need to be recycled).

Due to the cumbersome structure of regedit and to the fact that you can only put simple key / value pairs, I think one should try to limit the number of configuration values in the registry.

If you need complex configuration (i.e composed keys and a generic store for configuration values) you should use a relational database. That leads us to the third configuration option:

3 - SQL server database

Sooner or later I guess? that most BizTalk installations end up with a custom configuration database. From my own experience I can see that it's often used to keep temporary data (for instance data that at some point will be batched and delivered), state data and of course configuration.
With a relational database you can store any information. And it may be the only choice if you have relational data and want to store it in a reasonble way.
To write / retrieve data from the database just use the sql adapter or the newer WCFSQL-adapter.
The drawback of using a relational database is SPEED. The data has to be picked up from disk and if you use an adapter (ie not doing direct ODBC calls) you will in addition have persistence points before each configuration send request.

There's also a build in feature in BizTalk called "cross references". It consists of some relational tables in BizTalk mgmt database. Typically you would use these tables to do lookups in mappings but nothing prevents you from putting other kind of configuration here. The value lookups in mappings use some caching feature to improve performance while the id lookups are not cached. The length of the value field in the cross ref tables is limited to 50 chars.

So is there a faster way?

4 - Sigle sign on (SSO) store

The SSO store is already present in every BizTalk installation and is used by the adapters to store configuration so it's quite a natural place to keep custom configuration. You can create your own SSO applications in which you have key / value pairs. This can be done via a MMC snap in. To retrive the configuration you need to write some code.
(More details about this option can be found in the "Professional biztalk server 2006" book by Daren Jefford, Kevin B Smith and Ewan Fairweather (page 594- 595))
The configuration stored in SSO is secured and present in memory on the biztalk servers.
However you're still limited to key value pair kind of storage and in some situations it's not enough.

So is there any option that's both fast and relational?

5 - BRE (Business rule engine)

The BRE is typically used to evaluate XML data at runtime. But you can, with a bit of coding, use long-term facts as a distributed configuration store in your BizTalk server park!
You can for instance have your configuration tables in some SQL database and at runtime (first execution cycle of the policy) put these tables as typed data tables in memory on your servers. You can then implement logic to update your cache at regular intervals and / or then data has been changed / added.
With the BRE you could create a relational, distributed configuration store in memory that is quite dynamic. (you will not need to restart your host instances but instead wait for the refresh cycle to get new configuration up into memory)
There's a little coding to achive this. If you want to dig deaper the implementation is described in the "Professional biztalk server 2006" book by Daren Jefford, Kevin B Smith and Ewan Fairweather (page 354- 355).

Conclusion:

There're many different technologies that can be used to store configuration data for use in a Biztalk solution / installation.
The choice of technology to use is highly dependant of your needs (performance, complexity of configuration data, ease of use...)
My own opinion is that one should avoid to use the biztalk host configuration file. I also think that the extra effort during the build up of a new biztalk installation to outline the strategy and policy to use for configuration data is well worth.

Gracefully handling orchestration errors

Errors happen, it's sad but true. Services can become nonresponsive.

This "article" is about how to handle this when using biztalk server to orchestrate an integration flow.
The image below show a very simple orchestration flow: A message trigger the flow, an external WCF service is called, the response from the service call is fetched and delivered to some endpoint... i.e a really simple integration flow delivering some data between disparate systems. So what could go wrong? In this simple case not so much, the most critical part of this flow is the external service call made from the orchestration. This service could be unavaileble, the network could be down.
What happens if we don't write any dedicated code to handle this?
Well the send port calling the external service will retry according to the configured retry values (the default in biztalk is 4 retries in total 5 minutes appart). If these retries all fail the send port will be suspended along with the orchestration. That's ok -> then the network or service is upp again we just resume the port... or?
Unfortunately it doesn't work the way we would hope. The send port will indeed send away the message to the service once resumed but the response message from the service won't correlate back to our orchestration and hence the integration flow will be broken. No response payload will be delivered to our final destination (endpoint).

The design below is a way to solve this issue.

1 - first we must catch the exception thrown back from the failed service call.
(could be a custom service exception, a soap exception or System.Exception (when you'll catch all errors) )

2 - Next we suspend the orchestration with a suspend shape and with a relevant error message to the support operator. (a suspend flag has to be set when you catch the error which will be used to decide if an error happend in an if shape below the service call)

3 - A loop shape is used to retry the service call if the operator decides to resume the orchestration. The logic to exit the loop is simply handle by some flag that is reset then the service call is resumed.

A new "service message request" will be send to the message box resulting in a new instance of a send port calling the service. If the service call is successfull the response will be handled by the orchestration (thus resuming the integration flow) and (in this case) finally delivered to the endpoint.

You will end up with a least one suspended send port in the administration console that has to be manually killed once the orchestration is resumed. That's the best way I've found to recover from failed service calls (that exceeds the retry intervall on the ports).

If there's even a better way let me know!

Below you can see the orchestration design in this particular simple integration flow and how it looks in the tracking tool when a soap error is catched from a failed service call.

Orchestration design

Tracking tool -> catch SOAP error

Expose BizTalk endpoints / orchestration via windows azure app fabric

With the release of biztalk 2010 and microsoft new cloud services there's a completely new way of exposing service endpoints.
It's done via windows azure app fabric service bus technology and you can get it up and running simply by creating an account

This is, in my opinion, a huge improvement as compared to how you had to proceed before in order to expose services to the outside world.
It involved a lot of infrastucture like setting up fw servers - load balancing not to mention fw rules to open up for external partner applications...

Because I've been struggling so much with all of the above mentioned issues, I was eager to test this new technology.

I started with a simple console application hosting a WCF service.
-> That's quite useless in a production environment as you don't have any / very limited monitoring capacity on a console window. So I moved on to test exposing a biztalk orchestration to the cloud.

1 - EXPOSE THE SERVICE

In order to do that you need:

* windows 7 or windows 2008 R2 as operating system.
(these are the only operating systems supporting service activation in IIS which is needed for cloud hosting scenarios)

* visual studio 2010, sql server 2008 R2 and biztalk 2010
* BizTalk server 2010 feature pack ( http://go.microsoft.com/fwlink/?LinkId=204701)
* Windows Azure appfabric SDK (http://go.microsoft.com/fwlink/?LinkId=204209)
* Windows Server AppFabric (http://www.microsoft.com/downloads/sv-se/details.aspx?displaylang=sv&FamilyID=467e5aa5-c25b-4c80-a6d2-9f8fb0f337d2)

In addition to the above components we need an azure account.
You can sign up for an account here:
(http://www.microsoft.com/windowsazure/features/virtualnetwork)


Create a new Service Bus instance following the guidelines in the picture below:

After we've installed the above components and created our Service Bus we're ready to start coding!

We'll create an extremely simple biztalk assembly consisting of an orchestration, input schema (MultRequest.xsd), ouput schema (MultResponse.xsd) and map (TransformToResp.btm)

The orchestration receives our MultRequest message, transform it to our MultResponse message using the multiplication functoid. Finally the message is sent back to the logical request / response port that has to be configured as PUBLIC!!

See pictures below:

Create a strong key file and deploy the assembly to biztalk runtime.
Now we need to expose our "Multiplication Service Orchestration" to the cloud. This is done thanks to the WCF publishing wizzard and the BizTalk server 2010 feature pack we just installed.

Start Biztalk WCF publishing wizzard -> click next on the welcome screen -> Choose service endpoint, check the boxes and choose the application where you want the generated receive port to be created, click next. -> Check "Add a service bus endpoint", click next -> Choose "Publish BizTalk orchestration...", click next -> Choose the previously deployed dll, click next -> leave all the boxes checked, click next -> leave default target namespace, click next -> Check "Allow anonymous access to WCF service", click next -> configure the service bus endpoint according to picture below: (leave default bindings, specify your service buss endpoint and check the boxes)

Enter the service bus credentials:

Click next -> review the settings and click create.
(There's a more friendly version of how to configure these settings here:
http://social.technet.microsoft.com/wiki/contents/articles/expose-biztalk-applications-on-the-cloud-using-appfabric-connect-for-services.aspx)

There're 4 more steps we need to do in order to push out our service on the cloud.

1 - Change the application pool for the application in IIS to the ASP.NET 4.0 default app pool.
(make sure the app pool is running under the same account as the biztalk isolated host!)

2 - Configure our application in IIS manager to autostart.
Right click the application -> choose manage WCF, WF services -> choose configure -> click autostart on the left hand side meny -> activate.

3 - Bind the orchestration that we previously deployed to the autogenerated receive port.

4 - Start the application.

If everything was ok your service should now be registered in your azure app service bus and discoverable.
You can check if service activation was successfull by browsing to:
"https://[your service bus namespace].servicebus.windows.net"
You should see your service listed and there should be a mex endpoint discribing how to invoke your service.

2 - CONSUME THE SERVICE


The easiest way to test / and consume our service is to create a console application and add a service reference to the service we just created.

Add a new console project to the solution that contains your biztalk project.
Right click the console project and choose add service reference.
Enter the address to the discovery end point, in my case:
"https://jtapps.servicebus.windows.net/"
(replace jtapps with your namespace)
Select the service we just created in the window below and click ok.
This will automatically add a library reference: "System.ServiceModel.dll"
and create a basic configuration file for consuming this service.

We need to add credentials to access the service endpoint.
This is done by adding the following xml structure inside the <system.serviceModel> node:


<behaviors>
        <endpointBehaviors>
          <behavior name="sharedSecretClientCredentials">
            <transportClientEndpointBehavior credentialType="SharedSecret">
              <clientCredentials>
                <sharedSecret issuerName="owner" issuerSecret="[your secret key]" />
              </clientCredentials>
            </transportClientEndpointBehavior>
          </behavior>
        </endpointBehaviors>
      </behaviors>


we also need to tell the "client enpoint configuration" to use this "behavior" (ie these credentials)
it's done under the <client> node by configuring the attribute "behaviorConfiguration" (see the text in bold below). Also note the attribute "bindingConfiguration" that is set to "RelayEndpoint" -> the service bus just forward our request, it doesn't host anything.


<client>
            <endpoint address="sb://jtapps.servicebus.windows.net/ExposeOrchInCloudTest/WcfService_ExposeOrchInCloudTest.svc"
                binding="netTcpRelayBinding" bindingConfiguration="RelayEndpoint"
                contract="ExposedOrchInCloud.WcfService_ExposeOrchInCloudTest"
                name="RelayEndpoint" behaviorConfiguration="sharedSecretClientCredentials" />


Below is a snapshot overview of the configuration file:

That's the configuration of our client, now we just need some lines of code that will call our service bus endpoint.

We define our client to use the RelayEndpoint address, we define a simple request message and we call the service display the result.

Below is the code sample with the main points highlightened in bold and finally a picture of a sample call to the service. The client console window shows the request parameters and the result we get back. In the background you can see the biztalk admin console with a tracking query that shows the message flow inside biztalk runtime.

So we make a request from our local dev computer or server -> it calls an exposed endpoint on the azure service bus that can be a server center in northern america / europe / asia ... anywhere -> it goes back to our server / dev computer, handling the request and sending the response back the same way.
It's slow! And for the moment I think this it's only to be used for asynchronous type of service calls.
But it's extremely loosely coupled and I think a very fast (and secure!) way to expose services to the outside world.



using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace TestExposedOrchClient
{
    class Program
    {
        static void Main(string[] args)
        {
            ExposedOrchInCloud.WcfService_ExposeOrchInCloudTestClient myTestClient = new ExposedOrchInCloud.WcfService_ExposeOrchInCloudTestClient("RelayEndpoint");
            ExposedOrchInCloud.MultRequest myTestRequest = new ExposedOrchInCloud.MultRequest();
            myTestRequest.value1 = new decimal(2.5);
            myTestRequest.value2 = new decimal(3.0);
            try
            {
                Console.WriteLine("Calling biztalk cloud service to multiply:");
                Console.WriteLine("value1: " + System.Convert.ToString(myTestRequest.value1) + " with");
                Console.WriteLine("value2: " + System.Convert.ToString(myTestRequest.value2));
                ExposedOrchInCloud.MultResponse myResult = myTestClient.Operation_1(myTestRequest);

                Console.WriteLine("***************************");
                Console.WriteLine("*** Mult Result ***");
                Console.WriteLine("***************************\n");
                Console.WriteLine("Is: " + System.Convert.ToString(myResult.result));
                int i = Console.Read();
            }
            catch (Exception ex)
            {
                Console.WriteLine("\nException Message : " + ex.Message + "\n");
                Console.WriteLine("Exception Source : " + ex.Source + "\n");
                if (ex.InnerException != null)
                {
                    Console.WriteLine("\nInner Exception Message : " + ex.InnerException.Message + "\n");
                    Console.WriteLine("Inner Exception Source : " + ex.InnerException.Source + "\n");
                }
            }
        }
    }
}

Simple MSMQ tester over http

Well I guess I should have called this blog monthly instead of weekly developers notes...

Anyway here's a short input from my last week where I had to test the ability to send MSMQ over http.

It's very easy writing a simple console app to test this.

First you need to install the MSMQ role on your server or dev. computer.
In my case I used windows 7 so it looks like this: (sorry it's in swedish but I hope you can guess the settings anyway)

When you'll need to create your queue. Open computer management, expand service and programs and you should see at the bottom of this expanded view a new "messages" entry. Expand it and right click private queue to create a new one. Choose it to be transactional and click ok. That's all you need to do! Per default all users can send to the queue.

Our test queue is ready. Now we just need to write a simple client to send messages.

Start visual studio and a new console application project. Add a reference to System.Messaging and a using statement in your code:

using System.Messaging;

In order to send to a transactional message queue you need the message to be ... transactional! of course...
"MessageQueueTransaction" type is used for this.

Then you need to define a queue and a message to send on it:
"MessageQueue" &
"Message" types are used for this.

Begin the transaction, define the queue, the message with its properties, send it and commit the transaction...
Below is the code sample for this:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Messaging;

namespace myTestMSMQConsoleApp

{

    class Program

    {

        MessageQueueTransaction m_mqtrans = null;

        static void Main(string[] args)

        {

            Console.WriteLine("Start testing sending transactional messages to msmq over http...");

            Console.WriteLine("Please enter the server name where to send (default is localhost)");

            string serverName = Console.ReadLine();

            if (serverName == null || serverName == "" || serverName == ".")

                serverName = "localhost";

            Console.WriteLine("Please enter the path of the queue where to send (ex: private$/myprivatetestqueue)");

            string queuePath = Console.ReadLine();

            queuePath.Trim('\\');

            queuePath.Replace("\\", "/");

            string myMessageStr = "This is a test message!";

            string myLabel = "testLabel";

            Program myProgram = new Program();

            myProgram.SendStrMessage(myMessageStr, myLabel, "FORMATNAME:DIRECT=http://" + serverName + "/msmq/" + queuePath);

            Console.WriteLine("Message sent to queue");

            int i = Console.Read();

        }

        public void SendStrMessage(string sMessage, string sLabel, string TargetQueue)

        {

            if (m_mqtrans == null)

            {

                m_mqtrans = new MessageQueueTransaction();

                m_mqtrans.Begin();

            }

            MessageQueue mq = new MessageQueue(TargetQueue);

            Message msg = new Message();

            msg.Formatter = new ActiveXMessageFormatter();

            msg.UseJournalQueue = false;

            msg.UseDeadLetterQueue = true;

            msg.Recoverable = true;

            msg.Body = sMessage;

            msg.Label = sLabel;

            mq.Send(msg, m_mqtrans);

            m_mqtrans.Commit();

            m_mqtrans.Dispose();

        }

    }

}

This was written in a hurry but I hope that it'll help then you need to test sending MSMQ messages.

How to Configure IIS for an HTTP Receive Location, Biztalk 2010, windows 2008 R2, windows 7

Some time ago I had to migrate a biztalk application from 2006 to 2010 that used the http receive adapter to receive post requests. In order to make it work in IIS7, I had to perform quite a lot of steps and this is what I want to share here as I didn't find any 100% solution on the net.

The official microsoft guide is a good start and worth to take a look at but there some pieces missing in it to make this work. (http://msdn.microsoft.com/en-us/library/aa559072(v=bts.70).aspx)

Well lets start with the configuration. First, make sure that you got IIS installed with CGI and ISAPI coding functions. Go to program and functions in windows 7 or add feature if you're on windows 2008.

Then you have to perform these steps:

(before you proceed open IIS manager)

1 - AddISAPIHandler

First make sure that the ISAPI-dll handler is enabled

Top level Server settings -> Features view -> handler mappings

If ISAPI-dll handler is disabled right click it and select "edit feature permission"

check "Execute" box as on picture above.

2 - AddCGIHandler

Repeat the steps above for the CGI handler

3 - EditFeaturesSettings_addBtRestriction

Next add BTSHTTPReceive.dll to the ISAPI & CGI restrictions rules

Top level Server settings -> Features view -> ISAPI & CGI restrictions

Add

Browse to BTSHTTPReceive.dll

4 - EditFeatureSettings_permissions

Click on Edit Feature Settings and check the two boxes.

5 - Addmodulemapping

Go back to handler mappings (Top level Server settings -> Features view -> handler mappings)

Click on Module mapping and add a new module according to screen shots

6 - AddappPool

Now we need an application pool that runs under an account with access to BTSHTTPReceive.dll path.

7 - Finally, create an application with the application pool just created and under an account that has access to BTSHTTPReceive.dll.

(You can use the test functionality in IIS manager)

Test

Now you should be able to test the configuration.

Create a simple html form page that call the application you just created.

For instance:

"<html>

<body>

<h2>simple test form</h2>

<form method="post" action="http://localhost/MyTestHTTPapp/BTSHTTPReceive.dll?">

<fieldset>

<legend>testData</legend>

<label>someValue1</label><input type = "text" id = "value1" name="value1" value="some value 1" /><br/>

<label>someValue2</label><input type = "text" id = "value2" name="value2" value="some value 2" /><br/>

</fieldset>

<input class = "next" type = "submit" name="next" value = "send" />

</form>

</body>

</html>"

httpReceive

Create a http receive location according to screenshot above

Create for instance a file send location that subscribes to the BTS.receiveportname (for instance)

Start everything and post your form.

You should then "see" the post request in the file saved by your file send adapter:

In this sample request it should be;

"value1=some+value+1&value2=some+value+2&next=send"

And thats it, now your http post location should be operational!

First light

Hi,

The goal of this blog is to share my daily experience as a professional system developer and as a private android developer. My goal is to write weekly and to share about the issues I'm confronted with in my professional and private developer life. I hope to save time for people trying to solve the same issues and to give back to the "community" that via blogs forum ... have helped me so much over the years!

sincerely,
/Jakob