Handling binary data in SOAP - WMB V7 and MTOM/XOP

If you need to send large binary files over SOAP I suggest to use MTOM/XOP and not the common base64 encoding. ( One of the common types to transfer binary data over SOAP).
The benefit of using MTOM is that it not increasing the size of the original message. base64 converts each 6 bits to 8. if working with UTF-8 the encoded data will be larger in about 33%.
The data will be transfered as binary over soap. no conversion.
When setting type of an element as base46binary the Axis2 framework ( Web services in WMB handled by Axis2).

From Alex Linder's Blog - C&SI

Archive Service - Soap --> File

I made a sample project to test this technology with WMB V7.0.2.
I implemented an archive service, using SOAP Input node and FileOutput node.
The consumer invoke the service with file name and file data attached to soap using MTOM.
for the testing I worked with SoapUI, great testing tool for Web Services. use the following link to configure SoapUI work with MTOM : Adding Headers and Attachments

I exposed the following WSDL

The method get to input parameters, fileName and fileBuffer ( base64binary type).
here the image of soapUi console with the request and mtom configuration

From Alex Linder's Blog - C&SI

.

When invoking the WS the binary data sent in binary mode ( MIME).

From Alex Linder's Blog - C&SI

On the server side, the base64binary is encoded as BLOB, you can see this in the following debug screen shot :

From Alex Linder's Blog - C&SI

to save the file buffer to disk I create an output message of BLOB domain in the compute node.

From Alex Linder's Blog - C&SI

don't forget to built the soap reply message.

During my testing I notice that when handling files over 1 MB during debug mode the message flow use about 50% CPU and crash.
When not in debug mode I sent files over 6 MB with no problem.

links for the tester projects:
message flow project
message set project

Developing in WebSphere Message Broker Best practices guide Part 1

In this post I will provide with some guidelines you need to keep in mind when developing message flows in WMB (WebSphere Message Broker).

WMB is a good integration software but you need to know some internals for developing suit solution.

1. Avoid using hard wired looping in the message flows, Use PROPAGATE statement instead.

When using hard wired looping in message flow All the resources of the loop participant are stored on the stack storage. Nodes , message trees, variables copied and held in memory.
Therefore while message processing the stack storage of the Execution Group (process) can ran out.
You also probably will get abends and core dumps.

By using the PROPAGATE statement the data won't be copied and stored during the loop iterations. After each iteration all the iteration data will be freed and re-used.

2. Work with reference variable and don't navigate through message tree directly.

SET OutputRoot.XML.Root.Child.Field1 = 1;
SET OutputRoot.XML.Root.Child.Field2 = 2;


For those simple assigning statements about 20 navigation action will be made. And it's only if the OutputRoot element is empty.
If not it can be much more.


If you have complex message transformation and performance is your concern work with Reference variable and Create statement.

DECLARE ref REFERENCE TO OutputRoot;
CREATE LASTCHILD OF ref as ref DOMAIN('XML') NAME 'XML';
CREATE LASTCHILD OF ref as ref NAME 'Root';
CREATE LASTCHILD OF ref as ref NAME 'Child';
CREATE LASTCHILD OF ref NAME 'Field1' VALUE '1';
CREATE LASTCHILD OF ref NAME 'Field2' VALUE '2';


We no using SET statement and since all CREATES specify relative keywords no navigation are required. But we got more code :-(.
You can use the ROW statement to get more code visibility and simplicity combined with SET statement.

remember to work with REFERENCE.

3. Avoid using Environment tree to handle local variables.

All data which stored in the Environment tree are stored in Memory during the whole message flow and not freed after it's out of scope.
so if you need space to store local data for node work with the LocalEnvironment tree. Thus avoiding high memory consumption.
Even if you delete the fields from the Environment tree it still won't free the memory till message flow end processing the message. The broker use pool allocation and reuse an pointed resources of the Environment Tree.

more will come soon.

Websphere Message Broker HTTP Reply Node default behaviour changed after fix pack installation

I installed a 6.0.0.9 fix pack םn one of my clients site. After the installation I notice that some client applications that work over HTTP began to get strange errors. They couldn't parse the response. I checked the response with tester tool for WS SoapUI and notice that hebrew characters represented in something that looked like unicode encoding and not UTF-8 ( just hebrew characters ).
This was my mistake. will explain it shortly.

This was the behavior when I worked through biphttplistener.
But if working through IE01: WebSphere Message Broker - Proxy Servlet for the HTTP nodes the communication worked great. The response was interpreted in UTF-8, no errors in clients apps.

Problem

The problem is that IBM changed the default behavior of the HTTP Reply node. After the fix pack 4 of WMB 6 by default the Content-Type: text/xml;charset=UTF-8 was removed from the reply. Thus when the client got the reply without any specific encoding it tried to read it as Ascii and not UTF-8. The data returned in the same UTF-8 encoding as you can see below the raw data before and after the fix :

Before Fix Pack

POST /SOA/unicodeproblem/workdude HTTP/1.1
Content-Type: text/xml;charset=UTF-8
SOAPAction: "urn:recache"
User-Agent: Jakarta Commons-HttpClient/3.1
Host: localhost:9999
Content-Length: 248




hello
[0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] [0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d]




After the Fix Pack

POST /SOA/unicodeproblem/workdude HTTP/1.1
SOAPAction: "urn:recache"
User-Agent: Jakarta Commons-HttpClient/3.1
Host: localhost:9999
Content-Length: 248



hello
[0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] [0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d]




[0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] [0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] is equal to שלום שלום

Solution

You have couple of options:

1. set the content type manually in the code.
2. set default header by the checkbox on Http Reply Node. (Need to redeploy all your flows)
3. Work through IE01 sp, the content header is always set.

Websphere Message Broker Fixpack installation - Do backup before it

It is nice to know that IBM invested some thinking in the broker fix pack installation process, or maybe not.
The installation most of the time works fine, the problem is if something go wrong and you want to rollback the installation process.

The problem is that the only fix pack that you can uninstall are those of the toolkit. You can't uninstall any runtime fix packs.
If for some reason the broker or config manager fail to start post the fixpack installation than you probably would need to reinstall all server components.
With my great fixpack installation I encountered a nice feature, the installation massed up the jre library thus the components failed to start. I succeeded to fix it by copy the jre library from other working server.

It's very strange, really.

So before you install I suggest to backup the server, make a snapshot/image.

SOI with WMB

It's is very strange thing but I think that IBM missed a very important feature in websphere message broker design, the Infrastructure assets.
Each organization has its own assets which its develops and customize for their use.
In the SOA approach you build infrastructure services and application for reuse.
The reuse option in WMB can be achieved by the subflow ability.
Subflow is a regular message flow with small difference, It starts from Input node, logic inbound port and not physical one. ( Like direct ports in Biztalk).
You than can drag your subflow to the your message flow, it would be represented by single node.
What really happens there is an inline copy of the subflow to the message flow during the bar compilation. Thus you getting different copy of your subflow during each bar compilation.
It's like they didn't hear about REFERENCE in IBM.
So now for example you developed a logging subflow, or generic error handler. You put the subflow in all your projects.
Now if you make change in your subflow you need to recompile all of your message flows projects!
And here is the problem.
Every time you make change in your infrastructure you need to recompile all your projects.
This is a major development issue.

Some Tips to ease on this issue:

- Develop as many as you can of your infrastructure in java. thus won't need to
recompile your projects.
- Use scripting abilities of the webpshere message broker to recompile/redeploy the projects.
http://www.ibm.com/developerworks/websphere/library/techarticles/0706_spriet/0706_spriet.html

Working now on different approach to solved this, will post more details on it soon.
Hope to achieve the right agility with websphere message broker.

Error Handling in Websphere Message Broker V6/V6.1

A major issue in the WMB message flows is the error handling.
I saw many production message flows which loose messages meaning loose business information.

WMB platform give the developer freedom and variety of the error handling but it is very important to understand how it works.

So let's begin.

You can divide the broker elements in two categories under transactional objects and non transactional. On the transactional objects the engine makes checkpoints before entering each node. Thus if exception thrown in the entered node the object that you will get will be without any changes made in the node. If you need the changed object details you need to catch it in the node (java/esql try catch code). The non transactional elements save their state and the engine don't do any rollback on them.

Transactional Objects (those who have the Input/Output prefix):

• Body - Root
  
• LocalEnvironment
  

Non transactional :

• Environment
  
• ExceptionList
  

Node's Failure Terminal

Each node has a failure terminal. Message will route to this terminal only if there was an exception in the node and the failure terminal is connected otherwise the exception will be propagated to the previous up the flow. If an error occurred downstream it won't be routed to the node's failure terminal. I don't recommend using this pattern except maybe when calling sync service like.

When developing sub flows remember to connect all the terminal of the sub flow, because during the deployment of the flow the compiler will neglect the unconnected terminals, thus if failure occurred in the sub flow and you didn't connect it the failure won't be propagated to the main flow and thus you will lose track of the error.

Try Catch pattern and Catch terminal

My suggestion is to use this pattern; you get it implicit on the input nodes, by choosing the transaction mode.
Don't forget to throw exception at the end of the catch handling if you working under transaction otherwise no rollback will be executed because the whole transaction handled successfully.
If you developing synchronized service construct indicative reply message for the caller.
Very important if you putting dump messages for logging by the MQOutput set it to non transaction mode, if not your dump message will also be roll backed by the engine and you won't see your message in the queue.
Put your tryCatch node on common business parts, divide your flows wisely.

Trace Nodes

Nice function giving you the ability to dump your trace to log files, trace or custom files. You can dump whatever you want, ${Root}, and use the ESQL function like CURRENT_DATE.
The major improvement in 6.1 version is that now you don't need to delete the trace nodes after the development you can just disable them by this command :

mqsichangetrace –n [on off]

MQInput Node error handling flow when working under transaction (otherwise the message will be discarded) and the error occurred beyond the MQInput node.

1. If catch terminal connected the exception will be propagated there. Remember to throw custom exception for the Roll back process in which all action against external sources like DB or MQ queues will be back-out.
   It is also a good place for compensation process if needed.
   

   The message will be rolled back to input queue and the back out count will be raised by one.
   

2. If failure terminal connected the and the back-out count of the message equals treshold property the message will routed there if not it will be routed to the Back-out queue (property on the input queue) and if it not set then it tries to put it on the Qmgr back out queue.
   if no back out queue exist it will try to put the message on the Qmgr DLQ.
   If error occurred beyond the failure terminal then the engine will try to resend the message twice the treshold number ( new in version 6.1) and then try to put it on the Qmgr Back out Queue and if not then on the Qmgr DLQ.
   If no success yet then the message will loop infinitely thus become a poison message and manual interfering is needed.
   

Treshhold

MQ gives you set of properties by which you can handle back out messages. You can set on the queue the back out number and the back out queue name. the MQInput node will route the message implicitly if the treshold is reached.