tag:blogger.com,1999:blog-76656174501553597262024-03-08T15:17:41.893+02:00SW Engineering Blog - Alex Linder In this blog I plan to share things that I had learned as part of my career. technical (IC) and management related in the software engineering domains. Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.comBlogger30125tag:blogger.com,1999:blog-7665617450155359726.post-6538977283792519832012-04-09T11:24:00.000+03:002012-04-11T10:39:12.281+03:00DataPower get WSDL from MPGW - make it work with ?wsdl<div dir="ltr" style="text-align: left;" trbidi="on">
I needed to make the GET method work with POST on MPGW (Multiple Protocol Gateway). I made a proxy service which does some WTX COBOL <==> XML mapping ( on that I will expand in another post) over WMQ. The proxy exposed over SOAP.<br />
I wanted to enable the service consumers an option to get the WSDL from the DP on the same URL with the "?wsdl" method. Customer doesn't have WSRR or other registry solution on site.<br />
On the way I encountered on a strange issue in DP when tried to make the GET and POST methods enabled at the same time on the MPG. It just didn't worked at the same time. POST or GET not together.<br />
No error at all, after some investigation of the probe log I noticed the following information message :<br />
<br />
<b>"Event Code 0x80e0010e - The configuration has disabled rule processing in one direction.</b><br />
<i>The
configuration has disabled rule processing in one direction. This may
be due to an 'unprocessed' rule setting, or a dataflow which is
inherently unidirectional, such as HTTP GET transactions which only have
databodies in the response path."</i><br />
<br />
<br />
The issue was that by default the policy rule can handle only one method at a time like GET or POST and not together. IBM likes to hide stuff.<br />
<br />
<br />
To handle this issue you need to enable <b>non xml processing</b> on the processing rule under the <b>Objects --> Xml Processing --> Processing Rule --> [Your Rule]</b>, set the "Non-XML Processing" to on. This will solve the parallel processing of SOAP (POST) and HTML (GET) in the same policy.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-DcmnmqZ0bwE/T4KVVHHhghI/AAAAAAAAMZU/YqbXS4JVxMQ/s1600/ObjectsXmlProcessingProcessingRule.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="http://3.bp.blogspot.com/-DcmnmqZ0bwE/T4KVVHHhghI/AAAAAAAAMZU/YqbXS4JVxMQ/s320/ObjectsXmlProcessingProcessingRule.PNG" width="320" /></a></div>
<br />
<br />
<br />
<br />
<span style="font-size: large;">The Solution pattern :</span><br />
<br />
<br />
<ul style="text-align: left;">
<li>Set the MPG back end to dynamic proxy back end type. It need to be dynamic to enable basic Content Base Routing (CBR) on the response flow. One handles the WS request and the other the WSDL. The distinction will be made in the match pattern on the back end URI.</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-by2Atf3WZ74/T4KUu1UUQ7I/AAAAAAAAMYs/Qye8o4BjSAI/s1600/MPG_Config_Main.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="249" src="http://3.bp.blogspot.com/-by2Atf3WZ74/T4KUu1UUQ7I/AAAAAAAAMYs/Qye8o4BjSAI/s320/MPG_Config_Main.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<ul>
<li><span style="text-align: left;">Create a Client to Server processing rule which would handle the GET ?wsdl (also the match pattern "*?wsdl|WSDL) request. <br />I did simple fetch to the WSDL file from local store, It can be handled differently with registry/repository on site :-). Or it can be evolved to dynamic xslt which will generate WSDL with environment context like the address.<br />in the rule you need to set the back end URL to different from the original service so you can match on it in the response rule. I set it to loopback, you need to set the <b>var://service/routing-url = http://127.0.0.1/loopback. </b><br />Also because no back end processing needed set the <b>var://service/mpgw/skip-backside = 1, </b>by this the rule won't pass the request to the back end at all. loopback pattern.</span></li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-urohFtaB6rw/T4KUuwl8dyI/AAAAAAAAMYw/hSZ77XEBl7Q/s1600/WSDL_Request_Rule.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="249" src="http://2.bp.blogspot.com/-urohFtaB6rw/T4KUuwl8dyI/AAAAAAAAMYw/hSZ77XEBl7Q/s320/WSDL_Request_Rule.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul style="text-align: left;">
<li>Create the Server to Client rule to handle the WSDL response flow. Nothing special there, only the match pattern which set to "*loopback". Thus handling only the WSDL requests.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-JFVZ0YClF3I/T4KUwIcLz-I/AAAAAAAAMY4/lB30_9p8-7Y/s1600/WSDL_Response_Rule.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="249" src="http://4.bp.blogspot.com/-JFVZ0YClF3I/T4KUwIcLz-I/AAAAAAAAMY4/lB30_9p8-7Y/s320/WSDL_Response_Rule.PNG" width="320" /></a></div>
<div>
That's it.</div>
<div>
Hope it would help someone.</div>
<br />
<div>
<br /></div>
<div>
<br /></div>
<br />
<br /></div><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com2tag:blogger.com,1999:blog-7665617450155359726.post-89910394752332748972012-01-29T17:27:00.000+02:002012-01-29T18:04:52.114+02:00Biztalk future - it won't going to die soon<div dir="ltr" style="text-align: left;" trbidi="on">
I think that in the last two years there was a lot of online BUZZ regarding the Biztalk Server will go out of support.<br />
I believe that this won't happen in the nearest time.
<br />
Microsoft can't kill its middleware Integration solution. There are thousands of production Biztalk environments world wide.<br />
Biztalk will stay as a on premise server solution for some time, yes Microsoft roadmap is to move toward the cloud solution. AppFabric with Service Bus, Integration Services, Cache Service and other services will eventually implement the current Biztalk capabilities, the solution won't be called Biztalk.<br />
Microsoft aims to create a single middleware solution for cloud and On-Premises projects.<br />
Today Microsoft focus is on the cloud. And it will take time , years, to implement Biztalk functionality on the AppFabric cloud solution.<br />
Migration probably won't be smooth but eventually it will work, from what I saw online the migration process won't be able to handle whole applications, only artifacts (Orchestrations, maps, pipelines and ...).<br />
But I am sure that Microsoft will provide the needed support in a reasonable TCO to move from Biztalk Server to the AppFabric next-Generation middleware solution.<br />
<br />
Microsoft will continue to the develop and invest in Biztalk Server project till the cloud AppFabric project will have the needed and current Biztalk capabilities. <br />
The Biztalk project will be evolved in a moderate speed and investment.<br />
<br />
<b>Biztalk has a great integration features, like:</b><br />
<br />
<ul style="text-align: left;">
<li>Orchestrations - machine based integration flows, custom XBPEL files. The current .Net developers familiar with the Windows Workflow Foundation (WF). In the last 2010 version you still can't execute the WF on Biztalk runtime engine, but it is part of the product roadmap.</li>
<li>PUB/SUB engine - everything in Biztalk is based on the Publish/Subscribe patterns against the MessageBox DB for prof and cons. <br />This architecture is a major pitfall for low latency solution. But on the contrary it enables loosely couple integration solutions.<br />For low latency you can combine Biztalk map and adapter functianlity with WF in a regular .Net project in VS2010 and above. </li>
<li>Integration by W<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: x-small; line-height: 16px; text-align: -webkit-auto;">indows </span><em style="background-color: white; font-family: arial, sans-serif; font-size: small; font-style: normal; line-height: 16px; text-align: -webkit-auto;">Azure Connect</em><span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: x-small; line-height: 16px; text-align: -webkit-auto;"> </span> with Azure windows services.</li>
<li>B2B - EDI, SWIFT</li>
<li>BAM</li>
<li>BRE</li>
<li>...</li>
</ul>
<div>
A nice recorded presentation from the WPC2011 by <a href="http://digitalwpc.com/Videos/AllVideos/Permalink/e821e9f8-e379-45b0-8879-12fe271c86be#fbid=yrOWkA1h7rK">Tony Meleg's</a><span style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="color: white; font-size: 12px; line-height: 14px;"> </span><span style="font-size: 12px; line-height: 14px;">provides Microsoft long time strategy and commitment for Microsoft's Application Infrastructure future.</span></span></div>
<div>
<span style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="font-size: 12px; line-height: 14px;"><br /></span></span></div>
<div>
<span style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="font-size: 12px; line-height: 14px;"><b>In conclusion</b> </span></span></div>
<div>
<span style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="font-size: 12px; line-height: 14px;"><br /></span></span></div>
<div>
<span style="font-family: Verdana, Arial, Helvetica, sans-serif;"><span style="font-size: 12px; line-height: 14px;">Biztalk will stay in the market for a long time, eventually Biztalk capabilities will be implemented on the Azure AppFabric solution.</span></span></div>
</div><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com2tag:blogger.com,1999:blog-7665617450155359726.post-26029285877189084302011-08-03T18:43:00.004+03:002011-08-03T18:57:58.783+03:00Aspera Software – High-Speed File Transfer - Boost the file transferUsed <a href="http://www.asperasoft.com">Aspera's</a> solution in one of my latest integration projects.<br />We needed the ability to upload large files over geographical spread sites.<br />By Aspera solution we were able to use all the bandwidth of network for the upload and download.<br />We were able to transfer files ( various sizes 0.5 - 9 GB ) over 10 / 15 Mbps networks , over the internet with the maximum provided bandwidth provided ( 10/15 Mbps) by using the Aspera's "fasp" solution.<br />Aspera supports much much larger rate , 1 GBbit and more. <br />Aspera's algorithm can use all the bandwidth that is provided by the WAN / LAN.<br />Depends on the license.<br /><br />Before FTP was used and the max rate we saw was only round 500 kbps over the same 10/15 Mbps lines.<br /><br />We got great service by the support , I am really recommend to use this solution.<br />it saved us a lot of time and created a great solution to our customers.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com6tag:blogger.com,1999:blog-7665617450155359726.post-59545927316361456442011-08-03T18:05:00.006+03:002011-08-10T16:38:07.923+03:00Biztalk FTP adapter caused ACK Zombies messages in OrchestrationWe handled a strange behavior of logical FTP port in Orchestration which was set to Transmitted true.<br />
In the Orchestration we had several send ports in a single none transcriptional scope. this scope also included call and start orchestration shapes.<br />
The invocation of the orchestration left after each invocation two zombie messages , those was the ACK messages of the FTP adapter with the known error : <br />
<br />
<span style="font-style:italic;">0xC0C01B4C The instance completed without consuming all of its messages. The <br />
instance and its unconsumed messages have been suspended.</span><br />
<br />
I tried to tweak the propertices of the scope to Synchronized true and also switch to different type of transaction. I know that such changes influence on the orchestration behavior , this I won't describe here.<br />
Those changes didn't solve the issue.<br />
<span style="font-weight:bold;"><br />
Solution :</span><br />
Finally I created another Atomic scope that included only the send ports , this configuration solved the issue.<br />
That seemed to work fine but the error returned.<br />
The problem was caused by using Send Port Groups, we have one group with two ports FILE and FTP. when we removed the FILE port from group we stopped getting the error.<br />
<br />
In general you should always use Atomic scope if you have several send ports in a row , this save the Dehydration and Rehydration of the orchestration between each send. <br />
<br />
Strange behavior of the adapter and orchestration mediation in Biztalk 2009.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com1tag:blogger.com,1999:blog-7665617450155359726.post-76256272943361370472011-08-01T22:13:00.008+03:002011-08-02T11:15:22.705+03:00AIM - Application infrastructure and middleware - Cloud and ApplianceAIM<br /><br />Application infrastructure and middleware you can right it or buy it.<br /><br />The AIM solutions provided by most major vendors like Oracle , IBM and Microsoft.<br />The products that out in the market contains many features that suppose to make the AIM solutions easy to handle.<br />You must not forget that the purpose of those products not only to create the solutions but also to provide an easy maintenance for the solutions.<br /><br />The SMB market preferred not so far to do the integration by custom code because the AIM products were not so cost affective. If you need to do several integration processes a year why to spend tens of thousands on it? This was right in the past.<br /><br />Today you can find a good products out there that will not only speed the integration process but also will provide good platform for maintenance.<br /><br />If you don't have the man power or the resources you can get an appliance solution or a cloud.<br />appliance solution can be very easy to use , but keep in mind that it is an appliance , the update process can hard and you can have problems to scale your solution.<br /><br />Cloud is the world that everyone use , and you can do integration as a service , INaaS.<br /><br />some INaas vendors that you should check when seeking for integration solution for your business :<br /><br />- <a href="http://www.boomi.com/">Dell boomi</a><br />- <a href="http://www.castiron.com/">WebSphere Cast Iron</a><br />- <a href="http://www.mulesoft.org/cloud-connect">Mule Cloud connect</a><br />- <a href="http://web.progress.com/en/Product-Capabilities/enterprise-application-integration.html">Progress</a><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-13647498918154641202011-03-13T15:10:00.016+02:002011-03-13T17:57:15.139+02:00Handling binary data in SOAP - WMB V7 and MTOM/XOPIf you need to send large binary files over SOAP I suggest to use <a href="http://en.wikipedia.org/wiki/Message_Transmission_Optimization_Mechanism">MTOM/XOP</a> and not the common base64 encoding. ( One of the common types to transfer binary data over SOAP).<br />The benefit of using MTOM is that it not increasing the size of the original message. base64 converts each 6 bits to 8. if working with UTF-8 the encoded data will be larger in about 33%.<br />The data will be transfered as binary over soap. no conversion.<br />When setting type of an element as base46binary the Axis2 framework ( Web services in WMB handled by Axis2).<br /><br /><table style="width:auto;"><tr><td><a href="https://picasaweb.google.com/lh/photo/KzFci3oZNbNc_SPOVMgtSpthAqkHdabJqzIPu0qFuKU?feat=embedwebsite"><img src="https://lh4.googleusercontent.com/_0EFq8rT981w/TXzGRAJ_XTI/AAAAAAAAGVw/lbaymWyFpJw/s800/requestXsdBase64Binary.PNG" height="271" width="473" /></a></td></tr><tr><td style="font-family:arial,sans-serif; font-size:11px; text-align:right">From <a href="https://picasaweb.google.com/sasha.linder/AlexLinderSBlogCSI?authkey=Gv1sRgCM2-5MWo_oKZdA&feat=embedwebsite">Alex Linder's Blog - C&SI</a></td></tr></table><br /><span style="font-weight:bold;">Archive Service - Soap --> File</span><br /><br />I made a sample project to test this technology with WMB V7.0.2.<br />I implemented an archive service, using SOAP Input node and FileOutput node.<br />The consumer invoke the service with file name and file data attached to soap using MTOM.<br />for the testing I worked with <a href="http://www.soapui.org/">SoapUI</a>, great testing tool for Web Services. use the following link to configure SoapUI work with MTOM : <a href="http://www.soapui.org/SOAP-and-WSDL/adding-headers-and-attachments.html">Adding Headers and Attachments</a><br /><br />I exposed the following <a href="https://docs.google.com/leaf?id=0B24f-j__-uyUOTRmMGZjZGUtZTUxZS00MmRmLWEyODMtNWM4YmQzNjM2Mjdi&hl=en&authkey=CPmcvfAN" target="_new">WSDL </a><br /><br />The method get to input parameters, fileName and fileBuffer ( base64binary type).<br />here the image of soapUi console with the request and mtom configuration<br /><br /><table style="width:auto;"><tr><td><a href="https://picasaweb.google.com/lh/photo/pTGennf29J0NWtK9QFitKpthAqkHdabJqzIPu0qFuKU?feat=embedwebsite"><img src="https://lh4.googleusercontent.com/_0EFq8rT981w/TXzGR3CkXFI/AAAAAAAAGWM/yZIWwmbwoDA/s800/soapUIMimeConfigScreen.PNG" height="390" width="800" /></a></td></tr><tr><td style="font-family:arial,sans-serif; font-size:11px; text-align:right">From <a href="https://picasaweb.google.com/sasha.linder/AlexLinderSBlogCSI?authkey=Gv1sRgCM2-5MWo_oKZdA&feat=embedwebsite">Alex Linder's Blog - C&SI</a></td></tr></table>.<br /><br />When invoking the WS the binary data sent in binary mode ( MIME).<br /><br /><table style="width:auto;"><tr><td><a href="https://picasaweb.google.com/lh/photo/_CZNvR-psEOax9pC9L0t1pthAqkHdabJqzIPu0qFuKU?feat=embedwebsite"><img src="https://lh4.googleusercontent.com/_0EFq8rT981w/TXzGSBPL2yI/AAAAAAAAGWU/GZMuG7TO1b0/s640/binaryRequest.PNG" height="401" width="640" /></a></td></tr><tr><td style="font-family:arial,sans-serif; font-size:11px; text-align:right">From <a href="https://picasaweb.google.com/sasha.linder/AlexLinderSBlogCSI?authkey=Gv1sRgCM2-5MWo_oKZdA&feat=embedwebsite">Alex Linder's Blog - C&SI</a></td></tr></table><br /><br />On the server side, the base64binary is encoded as BLOB, you can see this in the following debug screen shot :<br /><br /><table style="width:auto;"><tr><td><a href="https://picasaweb.google.com/lh/photo/9pELaTwXecm82POX3a0XOJthAqkHdabJqzIPu0qFuKU?feat=embedwebsite"><img src="https://lh5.googleusercontent.com/_0EFq8rT981w/TXzGRB9AUeI/AAAAAAAAGV0/5OpxezgGG8Y/s640/debugFileBufferAsBLOB.PNG" height="452" width="640" /></a></td></tr><tr><td style="font-family:arial,sans-serif; font-size:11px; text-align:right">From <a href="https://picasaweb.google.com/sasha.linder/AlexLinderSBlogCSI?authkey=Gv1sRgCM2-5MWo_oKZdA&feat=embedwebsite">Alex Linder's Blog - C&SI</a></td></tr></table><br /><br />to save the file buffer to disk I create an output message of BLOB domain in the compute node.<br /><br /><table style="width:auto;"><tr><td><a href="https://picasaweb.google.com/lh/photo/JuXiq0wk9ASlP_yz4BXYXpthAqkHdabJqzIPu0qFuKU?feat=embedwebsite"><img src="https://lh3.googleusercontent.com/_0EFq8rT981w/TXzGRTsNxWI/AAAAAAAAGV8/68MSojsCGvY/s640/preFileOutputBlobMessage.PNG" height="379" width="640" /></a></td></tr><tr><td style="font-family:arial,sans-serif; font-size:11px; text-align:right">From <a href="https://picasaweb.google.com/sasha.linder/AlexLinderSBlogCSI?authkey=Gv1sRgCM2-5MWo_oKZdA&feat=embedwebsite">Alex Linder's Blog - C&SI</a></td></tr></table><br /><br />don't forget to built the soap reply message.<br /><br />During my testing I notice that when handling files over 1 MB during <span style="font-weight:bold;">debug mode</span> the message flow use about 50% CPU and crash.<br />When not in debug mode I sent files over 6 MB with no problem.<br /><br />links for the tester projects:<br /><a href="https://docs.google.com/leaf?id=0B24f-j__-uyUM2Q4Mzg4MGMtODBkNy00NmE5LWFhODktOWU3Y2ZhMzc1MWFi&sort=name&layout=list&num=50" target="_new">message flow project</a><br /><a href="https://docs.google.com/leaf?id=0B24f-j__-uyUYjZhMjljYTItZTYyOS00MDVlLTgxMjUtNmNlYmQ0MWFiOWE5&sort=name&layout=list&num=50" target="_new">message set project</a><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com4tag:blogger.com,1999:blog-7665617450155359726.post-70728507767783710922010-07-08T16:34:00.004+03:002010-07-08T17:35:40.400+03:00Developing in WebSphere Message Broker Best practices guide Part 1In this post I will provide with some guidelines you need to keep in mind when developing message flows in WMB (WebSphere Message Broker).<br /><br />WMB is a good integration software but you need to know some internals for developing suit solution.<br /><br />1. <span style="font-weight:bold;">Avoid using hard wired looping in the message flows, Use PROPAGATE statement instead.</span><br /><br />When using hard wired looping in message flow All the resources of the loop participant are stored on the stack storage. Nodes , message trees, variables copied and held in memory. <br />Therefore while message processing the stack storage of the Execution Group (process) can ran out.<br />You also probably will get abends and core dumps.<br /><br />By using the PROPAGATE statement the data won't be copied and stored during the loop iterations. After each iteration all the iteration data will be freed and re-used.<br /><br />2. <span style="font-weight:bold;">Work with reference variable and don't navigate through message tree directly.</span><br /><br /><span style="font-style:italic;">SET OutputRoot.XML.Root.Child.Field1 = 1;<br />SET OutputRoot.XML.Root.Child.Field2 = 2;<br /></span><br /><br />For those simple assigning statements about <span style="font-weight:bold;">20</span> navigation action will be made. And it's only if the OutputRoot element is empty.<br />If not it can be much more.<br /><br /><br />If you have complex message transformation and performance is your concern work with <span style="font-weight:bold;">Reference variable</span> and <span style="font-weight:bold;">Create statement</span>.<br /><br /><span style="font-style:italic;">DECLARE ref REFERENCE TO OutputRoot;<br />CREATE LASTCHILD OF ref as ref DOMAIN('XML') NAME 'XML';<br />CREATE LASTCHILD OF ref as ref NAME 'Root';<br />CREATE LASTCHILD OF ref as ref NAME 'Child';<br />CREATE LASTCHILD OF ref NAME 'Field1' VALUE '1';<br />CREATE LASTCHILD OF ref NAME 'Field2' VALUE '2';<br /></span><br /><br />We no using SET statement and since all CREATES specify relative keywords no navigation are required. But we got more code :-(. <br />You can use the <span style="font-weight:bold;">ROW</span> statement to get more code visibility and simplicity combined with SET statement.<br /><br />remember to work with REFERENCE.<br /><br />3. <span style="font-weight:bold;">Avoid using Environment tree to handle local variables.</span><br /><br />All data which stored in the Environment tree are stored in Memory during the whole message flow and not freed after it's out of scope.<br />so if you need space to store local data for node work with the LocalEnvironment tree. Thus avoiding high memory consumption.<br />Even if you delete the fields from the Environment tree it still won't free the memory till message flow end processing the message. The broker use pool allocation and reuse an pointed resources of the Environment Tree.<br /><br />more will come soon.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-74358381962825436422010-07-04T21:49:00.003+03:002010-07-04T22:00:17.391+03:00Force Remove of old WMQ Cluster information from QmgrIf you alter your WMQ Cluster, for example change it name or change the full repository Qmgr you may see error messages in the logs.<br />Errors like there is no full repository cluster for cluster "Name You have renamed" and more others.<br /><br />You can execute <span style="font-style:italic;">REFRESH CLUSTER(<CLUSTER_NAME>) REPOS(YES)</span> command on the partial repository Qmgrs or the reset cluster command.<br /><br />If those command not work properly I suggest to clean the SYSTEM.CLUSTER.REPOSITORY.QUEUE of the full repository Qmgr. Thus after the restart it will get the latest and the correct status of the WMQ Cluster.<br /><br />1. change the SYSTEM.CLUSTER.REPOSITORY.QUEUE to get <span style="font-weight:bold;">disable</span>.<br />2. restart the qmgr. (Because the repository queue is in get disabled mode the cluster process will failed to start and won't lock the queue for exclusive use)<br />3. Clear the SYSTEM.CLUSTER.REPOSITORY.QUEUE<br />4. Alter it to <span style="font-weight:bold;">get enabled</span> mode<br />5. restart the Qmgr.<br /><br />This should clean the noise log messages.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-35561727810233013632010-07-04T08:38:00.002+03:002010-07-04T08:58:27.149+03:00Consuming Web Service with SOAP header by Biztalk 2006There are a lot of information about this subject on-line,<br />but I think that it not so correct.<br /><br />The steps to create a working solution are :<br /><br />1. You need a WSDL which includes the declaration of the SOAP headers elements that are needed. <span style="font-weight:bold;">It won't work if you define an external schema for it.</span> as it is described in many posts.<br /><br />2. Create a property schema with the target <span style="font-weight:bold;">namespace</span> set to <span style="font-weight:bold;">http://schemas.microsoft.com/BizTalk/2003/SOAPHeader</span>, the root node name set to exactly the same as the root node name in SOAP header, and the <span style="font-weight:bold;">Property Schema Base property</span> set to <span style="font-weight:bold;">MessageContextPropertyBase</span>.<br /><br />3. Set the context of the request message with a XML of the SOAP Header structure.<br /> <br /><span style="font-style:italic;">XMLDOM = new System.Xml.XmlDocument();<br />XMLDOM.LoadXml(@"<ns0:Priority xmlns:ns0=""http://PurchaseOrderProcessing.Priority""><ID>123456</ID><Class>Silver</Class></ns0:Priority>"); WSReqMsg(SubmitPurchaseOrder.SOAPHeaderPropertySchema.Priority) = XMLDOM.InnerXml;<br /></span><br /><br />If you have an WSDL which not defines SOAP Header but it's needed (like WS-Security or WS-addressing) by WS provider you will need to ask the provider to provide updated WSDL or to update it by yourself. Otherwise it won't work in Biztalk 2006 with SOAP adapter.<br /><br />If you need to update the WSDL I suggest you to do it by exposing Orchestration as Web Service with the Web Service Wizard tool. You can use the defined property schema and you need to define custom schema which will include the needed SOAP Header.<br />During the wizard you can choose to expose SOAP Headers.<br /><br />In Biztalk 2006 R2 and later you can use WCF adapter which solve this issue without the need to update WSDL.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-78945556790565576512010-07-04T08:17:00.004+03:002010-07-05T23:41:23.325+03:00Exposing And Consuming Web Services by SAPYou can consume Web Services by SAP framework for web services and also expose RFC as Web Services without any need of external integration tools like SAP netweaver xi.<br />One of the most impotent things in SOA solution is the ability to monitor the traffic.<br />There is a great logging tool out of the box which is provided as part of SAP WAS, SOA Management. it's provide information on the calls time and also it's can capture the requests/responses.<br /><br />The process of consuming or exposing Web Services is not difficult at all, it evolves work with:<br />- Wizards : Creating the BAPI objects or exposing existing RFC.<br />- BAPI : Developing the Web Service proxy for consuming, working with objects and types created by the Wizard.<br />- SOA Management : Deploying the WSDLs/Exposing RFC , configure the logs.<br /><br /><br />There is some interoperability issues that you need to keep in mind when going for this solutions:<br /><br />- We didn't find a solution for consuming an existing WSDL as an interface and implement it.<br /><br />- xsi:nil attribute is not supported, opened OSS, waiting for SAP response.<br /><br />- You can't generate custom Namespace.<br /><br />There are more...<br /><br />The throughput was much more faster than thru XI. (10 times faster).<br /><br />In conclusion it is more than possible to expose SAP structures and types as standard Web Services and to consume them.<br />So if synchronous solution is suite for you you don't need to use adapters or XI. <br /><br />If you need help on this fill free to contact me for forward details.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com5tag:blogger.com,1999:blog-7665617450155359726.post-58592293714907257762010-01-27T16:57:00.012+02:002010-01-27T23:50:57.531+02:00Websphere Message Broker HTTP Reply Node default behaviour changed after fix pack installationI installed a 6.0.0.9 fix pack םn one of my clients site. After the installation I notice that some client applications that work over HTTP began to get strange errors. They couldn't parse the response. I checked the response with tester tool for WS <a href="http://www.soapui.org/">SoapUI</a> and notice that hebrew characters represented in something that looked like <span style="font-weight:bold;">unicode</span> encoding and not UTF-8 ( just hebrew characters ).<br />This was my mistake. will explain it shortly.<br /><br />This was the behavior when I worked through biphttplistener.<br />But if working through <a href="http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24009370&loc=en_US&cs=utf-8&lang=en">IE01: WebSphere Message Broker - Proxy Servlet for the HTTP nodes</a> the communication worked great. The response was interpreted in UTF-8, no errors in clients apps.<br /><br /><span style="font-weight:bold;">Problem</span><br /><br />The problem is that IBM changed the default behavior of the HTTP Reply node. After the fix pack 4 of WMB 6 by default the <span style="font-weight:bold;">Content-Type: text/xml;charset=UTF-8</span> was removed from the reply. Thus when the client got the reply without any specific encoding it tried to read it as <span style="font-weight:bold;">Ascii</span> and not UTF-8. The data returned in the same UTF-8 encoding as you can see below the raw data before and after the fix :<br /><br /><span style="font-weight:bold;">Before Fix Pack</span><br /><br />POST /SOA/unicodeproblem/workdude HTTP/1.1<br /><span style="font-weight:bold;">Content-Type: text/xml;charset=UTF-8</span><br />SOAPAction: "urn:recache"<br />User-Agent: Jakarta Commons-HttpClient/3.1<br />Host: localhost:9999<br />Content-Length: 248<br /><br /><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><br /> <soapenv:Body><br /> <recacheRequest><br /> <host>hello</host><br /> <port>[0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] [0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d]</port><br /> </recacheRequest><br /> </soapenv:Body><br /></soapenv:Envelope><br /><br /><span style="font-weight:bold;">After the Fix Pack</span><br /><br />POST /SOA/unicodeproblem/workdude HTTP/1.1<br />SOAPAction: "urn:recache"<br />User-Agent: Jakarta Commons-HttpClient/3.1<br />Host: localhost:9999<br />Content-Length: 248<br /><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><br /> <soapenv:Body><br /> <recacheRequest><br /> <host>hello</host><br /> <port>[0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] [0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d]</port><br /> </recacheRequest><br /> </soapenv:Body><br /></soapenv:Envelope><br /><br />[0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] [0xd7][0xa9][0xd7][0x9c][0xd7][0x95][0xd7][0x9d] <span style="font-style:italic;">is equal to שלום שלום</span><br /><br /><span style="font-weight:bold;">Solution</span><br /><br />You have couple of options:<br /><br />1. set the content type manually in the code.<br />2. set default header by the checkbox on Http Reply Node. (Need to redeploy all your flows)<br />3. Work through IE01 sp, the content header is always set.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-53102695317782937132009-12-20T23:12:00.004+02:002009-12-20T23:28:16.234+02:00Websphere Message Broker Fixpack installation - Do backup before itIt is nice to know that IBM invested some thinking in the broker fix pack installation process, or maybe not.<br />The installation most of the time works fine, the problem is if something go wrong and you want to rollback the installation process.<br /><br />The problem is that the only fix pack that you can uninstall are those of the toolkit. You can't uninstall any runtime fix packs.<br />If for some reason the broker or config manager fail to start post the fixpack installation than you probably would need to reinstall all server components.<br />With my great fixpack installation I encountered a nice feature, the installation massed up the jre library thus the components failed to start. I succeeded to fix it by copy the jre library from other working server. <br /><br />It's very strange, really.<br /><br />So before you install I suggest to backup the server, make a snapshot/image.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-26028592656957168842009-08-04T20:23:00.009+03:002009-08-04T20:49:39.468+03:00Http/Https asynchronous handle implementation in WMBHere I present simple option to work with the Http nodes in an asynchronous way.<br />The http/https in Websphere Message Broker handled by Http Listener process.<br />You may also work with IE01 support pack to implement the functionality of the WebSphere Message Broker HTTP listener in an external Servlet Container.<br /><br /><span style="font-weight:bold;">WMB Http/Https Handling in General:</span><br /><br />Http Listener communicates with the broker by those three queues:<br /> <br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_0EFq8rT981w/Snhv9bKRA9I/AAAAAAAABCE/Hry6ohZqHsA/s1600-h/wsQueues.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 213px; height: 54px;" src="http://1.bp.blogspot.com/_0EFq8rT981w/Snhv9bKRA9I/AAAAAAAABCE/Hry6ohZqHsA/s400/wsQueues.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5366162057065464786" /></a><br /><br />Each http request is written to the SYSTEM.BROKER.WS.INPUT queue which than picked up by message flow (correlation made by the URL written in the HTTP Input Node).<br />The HttpInputHeader is constructed from the http header data contained in the message.<br />Here is an example of the written request message by Http Listener to the SYSTEM.BROKER.WS.INPUT Queue:<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_0EFq8rT981w/SnhwgmKzJII/AAAAAAAABCM/flgzm6w8iWA/s1600-h/reqSample.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 134px;" src="http://4.bp.blogspot.com/_0EFq8rT981w/SnhwgmKzJII/AAAAAAAABCM/flgzm6w8iWA/s400/reqSample.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5366162661315912834" /></a><br /><br />After the message flow processing the reply message is written to the SYSTEM.BROKER.WS.REPLY queue by the Http Reply Node<br />(The correlation is done by MQ correlation, the correlation ID is written by the HttpInput Node to LocalEnvironment.Destination.HTTP.RequestIdentifier) which then picked up by the Http Listener process and return to the Http client.<br /><br />You can find more details on this li<a href="http://www.ibm.com/developerworks/websphere/library/techarticles/0608_braithwaite/0608_braithwaite.html"></a>nk.<br /><span style="font-weight:bold;"><br />The asynchronous Implementation:</span><br /><br />I implement this scenario <span style="font-weight:bold;">by two message flows</span>, First flow receives the request and the second flow sends the reply. They connected by WMQ and they can be deployed to different execution groups.<br /><br />First you receive the request. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_0EFq8rT981w/SnhyDatj_eI/AAAAAAAABCU/FBBW7BV_iN8/s1600-h/firstMessageFlow.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 258px;" src="http://3.bp.blogspot.com/_0EFq8rT981w/SnhyDatj_eI/AAAAAAAABCU/FBBW7BV_iN8/s400/firstMessageFlow.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5366164359047544290" /></a><br /> <br />Then construct the MQMD and store the RequestIdentifier in the MQMD header,MsgId.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_0EFq8rT981w/SnhymXr_Z-I/AAAAAAAABCc/IL7UHAPbFDc/s1600-h/firstMessageFlow_MQMD.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 137px;" src="http://3.bp.blogspot.com/_0EFq8rT981w/SnhymXr_Z-I/AAAAAAAABCc/IL7UHAPbFDc/s400/firstMessageFlow_MQMD.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5366164959531067362" /></a><br /> <br /><br />At the end puts the message on the queue.<br /><br /><br />Get the message from queue.<br /> <br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_0EFq8rT981w/SnhzJpnHIAI/AAAAAAAABCk/2rpQEvR-hOA/s1600-h/secondMessageFlow.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 135px;" src="http://4.bp.blogspot.com/_0EFq8rT981w/SnhzJpnHIAI/AAAAAAAABCk/2rpQEvR-hOA/s400/secondMessageFlow.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5366165565637861378" /></a> <br /> <br />Set the http reply correlation field by the MQMD MsgId and Optionally you can Remove the MQMD and the HttpInput headers.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_0EFq8rT981w/Snh0Osz8ycI/AAAAAAAABC0/9C2cl87UssM/s1600-h/secondMessageFlow_Headers2.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 112px;" src="http://1.bp.blogspot.com/_0EFq8rT981w/Snh0Osz8ycI/AAAAAAAABC0/9C2cl87UssM/s400/secondMessageFlow_Headers2.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5366166751907989954" /></a><br /><br />That is all you need to do to make a synchronous façade for an asynchronous transaction in WMB.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com1tag:blogger.com,1999:blog-7665617450155359726.post-9512929656999520782009-08-04T11:51:00.008+03:002009-08-04T12:23:52.290+03:00SOI with WMBIt's is very strange thing but I think that IBM missed a very important feature in websphere message broker design, the <span style="font-weight:bold;">Infrastructure assets</span>.<br />Each organization has its own assets which its develops and customize for their use.<br />In the SOA approach you build infrastructure services and application for reuse.<br />The reuse option in WMB can be achieved by the subflow ability.<br />Subflow is a regular message flow with small difference, It starts from Input node, logic inbound port and not physical one. ( Like direct ports in Biztalk).<br />You than can drag your subflow to the your message flow, it would be represented by single node.<br />What really happens there is a<span style="font-weight:bold;">n inline copy of the subflow</span> to the message flow during the bar compilation. Thus you getting different copy of your subflow during each bar compilation. <br />It's like they didn't hear about REFE<span style="font-style:italic;"></span>RENCE in IBM.<br />So now for example you developed a logging subflow, or generic error handler. You put the subflow in all your projects.<br />Now if you make change in your subflow you need to recompile all of your message flows projects!<br /><span style="font-weight:bold;">And here is the problem</span>.<br /><span style="font-weight:bold;">Every time you make change in your infrastructure you need to recompile all your projects.</span> <br />This is a major development issue.<br /><br /><span style="font-weight:bold;">Some Tips to ease on this issue:</span><br /><br />- Develop as many as you can of your infrastructure in java. thus won't need to <br /> recompile your projects.<br />- Use scripting abilities of the webpshere message broker to recompile/redeploy the projects.<br />http://www.ibm.com/developerworks/websphere/library/techarticles/0706_spriet/0706_spriet.html<br /><br />Working now on different approach to solved this, will post more details on it soon.<br />Hope to achieve the right agility with websphere message broker.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com2tag:blogger.com,1999:blog-7665617450155359726.post-39971286346575148062009-03-11T23:53:00.003+02:002009-03-12T00:00:33.067+02:00Error Handling in Websphere Message Broker V6/V6.1<span xmlns=""><p><span style="font-family:Georgia;">A major issue in the WMB message flows is the error handling.<br />I saw many production message flows which loose messages meaning loose business information.</span><br /></p><p><span style="font-family:Georgia;">WMB platform give the developer freedom and variety of the error handling but it is very important to understand how it works.</span><br /></p><p><span style="font-family:Georgia;">So let's begin.</span><br /><br /><span style="font-family:Georgia;">You can divide the broker elements in two categories under transactional objects and non transactional. On the transactional objects the engine makes checkpoints before entering each node. Thus if exception thrown in the entered node the object that you will get will be without any changes made in the node. If you need the changed object details you need to catch it in the node (java/esql try catch code). The non transactional elements save their state and the engine don't do any rollback on them. </span><br /></p><p><span style="font-family:Georgia;"><strong>Transactional</strong> Objects (<em>those who have the Input/Output prefix</em>):</span><br /></p><ul><li><span style="font-family:Georgia;">Body - Root</span><br /></li><li><span style="font-family:Georgia;">LocalEnvironment</span><br /></li></ul><p><span style="font-family:Georgia;"><strong>Non transactional</strong> :</span><br /></p><ul><li><span style="font-family:Georgia;">Environment</span><br /></li><li><span style="font-family:Georgia;">ExceptionList</span><br /></li></ul><p><span style="font-family:Georgia;"><strong>Node's Failure Terminal</strong></span><br /></p><p><span style="font-family:Georgia;">Each node has a failure terminal. Message will route to this terminal only if there was an exception in the node <strong>and the failure terminal is connected </strong>otherwise the exception will be propagated to the previous up the flow. If an error occurred downstream it won't be routed to the node's failure terminal. I don't recommend using this pattern except maybe when calling sync service like. </span><br /></p><p><span style="font-family:Georgia;">When developing <strong>sub flows</strong> remember to connect all the terminal of the sub flow, because during the deployment of the flow the compiler will neglect the unconnected terminals, thus if failure occurred in the sub flow and you didn't connect it the failure won't be propagated to the main flow and thus you will lose track of the error.</span><br /></p><p><span style="font-family:Georgia;"><strong>Try Catch pattern and Catch terminal</strong></span><br /></p><p><span style="font-family:Georgia;">My suggestion is to use this pattern; you get it implicit on the input nodes, by choosing the transaction mode.<br />Don't forget to <strong>throw exception at the end</strong> of the catch handling if you working under transaction otherwise no rollback will be executed because the whole transaction handled successfully.<br />If you developing synchronized service construct indicative reply message for the caller.<br />Very important if you putting dump messages <strong>for logging by the MQOutput set it to non transaction mode</strong>, if not your dump message will also be roll backed by the engine and you won't see your message in the queue.<br />Put your tryCatch node on common business parts, divide your flows wisely.</span><br /></p><p><span style="font-family:Georgia;"><strong>Trace Nodes<br /><br /></strong>Nice function giving you the ability to dump your trace to log files, trace or custom files. You can dump whatever you want, ${Root}, and use the ESQL function like CURRENT_DATE.<br />The major improvement in 6.1 version is that now you don't need to delete the trace nodes after the development you can just disable them by this command :<br /><br />mqsichangetrace –n [on off] </span><br /></p><p><span style="font-family:Georgia;"><br /><strong>MQInput Node error handling flow when working under transaction (otherwise the message will be discarded) and the error occurred beyond the MQInput node.</strong></span><br /></p><ol><li><div><span style="font-family:Georgia;">If catch terminal connected the exception will be propagated there. Remember to throw custom exception for the Roll back process in which all action against external sources like DB or MQ queues will be back-out.<br />It is also a good place for compensation process if needed.</span><br /></div><p><span style="font-family:Georgia;">The message will be rolled back to input queue and the back out count will be raised by one.</span><br /></p></li><li><span style="font-family:Georgia;">If failure terminal connected the and the back-out count of the message equals treshold property the message will routed there if not it will be routed to the Back-out queue (property on the input queue) and if it not set then it tries to put it on the Qmgr back out queue.<br />if no back out queue exist it will try to put the message on the Qmgr DLQ.<br />If error occurred beyond the failure terminal then the engine will try to resend the message twice the treshold number ( new in version 6.1) and then try to put it on the Qmgr Back out Queue and if not then on the Qmgr DLQ.<br />If no success yet then the message will loop infinitely thus become a <strong>poison message</strong> and manual interfering is needed.</span><br /></li></ol><p><span style="font-family:Georgia;"><br /><strong>Treshhold<br /><br />MQ </strong>gives you set of properties by which you can handle back out messages. You can set on the queue the back out number and the back out queue name. the MQInput node will route the message implicitly if the treshold is reached. </span><br /></p></span><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com7tag:blogger.com,1999:blog-7665617450155359726.post-85758300320650510622009-03-11T20:06:00.004+02:002009-03-12T00:48:11.502+02:00ellipsis my new homeWish me luck.<br />This month I started working at <a href="http://www.ellipsis.co.il/" target=”_new”>ellipsis </a>consulting services.<br />This firm is a great place to be at, great minds to learn from and challenging integration projects to design and develop.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com1tag:blogger.com,1999:blog-7665617450155359726.post-43420422311799005482009-01-09T18:35:00.003+02:002009-03-12T00:00:11.089+02:00MQSeries 2.0 Adapter Latency HotFixIf you working with the MQSeries 2.0 biztalk adapter and letancy is your concern you definitely need to install this <a href="http://support.microsoft.com/kb/938986">hotfix</a>.<br />By this hotfix you change the adapter working mode from polling to event driven when receiving messages from queue.<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-13882416314148266802009-01-05T23:13:00.002+02:002009-01-05T23:23:39.873+02:00SQL 2005 and Dates formatYesterday I and my work college encounter a strange date format issue.<br />We have a SSIS on SQL 2005 which imports csv file to a DB. One of its fields was a date field, represented in the following format : dd/mm/yyyy.<br /><br /><br />The strange thing was that if the day number was less than 12 , ex. 5/2/2009 then the date was stored in DB as <span style="font-weight: bold; color: rgb(255, 0, 0);">2/5/</span>2009. Different date. The import worked fine if the day number was above 12. 15/1/2009 was imported correctly.<br /><br />We tried to change the regional settings on the SQL Instance machine, played with the date format, checked the collation of the DB. Nothing helped.<br /><br /><span style="font-size:130%;"><span style="font-weight: bold;">Solution</span></span><br />We solved our problem by changing the date format to : dd/mmm/yyyy (6/JAN/2009)<div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-50559197690196920532008-07-28T20:25:00.006+03:002008-07-28T21:19:45.063+03:00MQ 6 on Windows 64 bit operating systems<span style="font-size:130%;">So, is there such thing as WMQ 64 bit on 64 bit Windows platform?<br /></span><br /><span style="font-size:130%;">I discovered that there is no such thing.<br />WMQ 6 on windows is great 32 bit queueing system that supports 64 bit windows platforms.<br />This issue doesn't exist on Linux/unix systems.<br /><br />If you developing .Net application for 64 bit systems you should compile it as an x86 program and not 64 bit, because WMQ 6.0 doesn't support 64 bit and you will get several unexpected errors while compilation/ running of the program.<br /><br />go to this link </span><a href="http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp"><span style="font-size:130%;">http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp</span></a><span style="font-size:130%;"> and search for "installing on 64-bit windows operating systems" for more details.</span><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-14636169344359641832008-07-13T22:25:00.005+03:002009-01-08T00:07:26.195+02:00SOA & BPM what is the connection<div>In the past few years the words SOA and <a href="http://en.wikipedia.org/wiki/Business_Process_Management">BPM </a>became common thing.</div><br /><div>I won't explain about SOA or BPM in this post, I will explain the connection between those aspects.</div><br /><div> </div>Why do organization want to enter the SOA world?<br /><div>Those are some of the reasons:<br /></div><ul><li>Business Agility - react fast to changes with Orchestration solutions.</li><li>Greater visibility of organization assets,services.</li><li>Business empowerment by IT abstration.</li><li>Reduction in integration expense, services reuse.</li></ul><p>Why BPM, I already have SOA?<br />SOA is the enabler of BPM, with SOA you can simplify the BPM solution and gain the next things:<br /></p><ul><li>Transparency of Business processes.</li><li><a href="http://en.wikipedia.org/wiki/Business_analyst">Business Analyst</a> empowerment.</li><li>With <a href="http://en.wikipedia.org/wiki/Business_Process_Modeling_Notation">BPMN</a> you can model your business process.</li><li>Business process optimization.<br /></li></ul>For Conclusion<br /><br />You making SOA for the BPM. Your goal is to be ready for the changes to come and in our word changes happing all the time. If your business will be inflexible than in a short time it won't be relevant.<br /><br />For BPM to succeed you need a good service infrastructure in your organization.<br />If not your BPM projects will soon become integration projects.<br />Your mind will be busy with services issues and not with the process.<br /><br /><br /><br /><div> </div><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-74177064699014593392008-06-28T23:21:00.002+03:002009-01-07T23:54:31.323+02:00optimize Biztalk projects by using outer mapping tools<div>I suggest everyone who work with biztalk to look at the xslt the mapper makes. You will be amezed with the poor xslt that it generates.<br /><br />There is a good feature in the map properties which allows you to insert xslt for the map to run.<br />My recomendation is to use biztalk map only as a host and to make the xslt with other tool like Altova MapForce.<br /><br />By this you will gain better performance of the mapping process.<br /></div><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-32455351633041847172008-06-11T23:29:00.003+03:002008-06-28T23:17:22.133+03:00Memory Leak using Biztalk mapper<span style="font-size:130%;">In one of our ESB projects that I worked on with my team we encountered on a huge memory leak issue with biztalk projects. In our project we had a couple orchestrations and maps (simple, only text functoids).<br /><span style="color:#ff0000;">The problem was that after running the map the first time the memory usage jumped for 200-300 MB.</span> After working on 4/5 messages the biztalk host service took about <strong>2 GB of memory. And with no load at all messages began to stuck. Messages run into message box but no dequeue was made at all (Tracking DB).<br /></strong>The messages was about 1-2 kb size and the orchestration was standart work flow, without any looping.</span><span style="font-size:130%;"><br /></span><span style="font-size:130%;"><br />After some resarch made by us and my work colegue Matanya Weinshtock that found the next kb :<br /><a href="http://support.microsoft.com/kb/918643">http://support.microsoft.com/kb/918643</a><br /><br /><strong><span style="font-size:180%;">summary of it :<br /><br /></span></strong>The System.Policy.Security.Evidence object is often used in transforms and can consume a lot of memory. Whenever a map contains a scripting functoid that uses inline C# (or any other inline language), the assembly is created in memory. The System.Policy.Security.Evidence object uses the object of the actual calling assembly. <strong>This situation creates a rooted object that is not deleted until the BizTalk service is restarted.Most of the default BizTalk functoids are implemented as inline script. These items can cause System.Byte[] objects to collect in memory.</strong><br /><br /><strong><span style="font-size:180%;">Suggested Solution by Microsoft<br /></span><br />To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly. </strong><br /><br /><strong><span style="font-size:180%;">The Solution</span></strong><br /><br />We decided not split each map to different assemblybecause as you can see at kb almost all of the functoids are implemented with inline scripts.<br />I suggest the next develop metodology with biztalk project :<br /><br /><strong>Alwayes split the project to at least three projects :</strong><br /><ol><li><strong>Schema project.</strong></li><li><strong>Maps project.</strong></li><li><strong>Orchestration project.<br /></strong></li></ol><p>with this develop metodology we succeded to almost stop the leaking at all.<br /></p><p>Again don't take biztalk as It is simple and supposed to work, always check it.<br /><br /><br /></p></span><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0tag:blogger.com,1999:blog-7665617450155359726.post-33950058137233142982008-06-10T16:03:00.008+03:002008-06-10T17:09:39.033+03:00MQ correlation set initialization in biztalk<span style="font-weight: bold;font-size:130%;" ></span><br /><span style="font-size:130%;">WMQ manager can generate a unique message id (MQMD_MsgID) for you if you put message to queue with MQMD_MsgID set to null.<br />In MQ/MQC biztalk adapters there is a great feature to get the MQMD_MsgID after putting the message to WMQ queue using solicit response port. After putting the message to queue you get an xml which includes the MQMD_MsgID.<br /><br /></span><div style="text-align: left;"> <div style="text-align: left;"><span style="font-size:130%;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_0EFq8rT981w/SE6AMsxrj1I/AAAAAAAAABM/N3vZAdSeQbM/s1600-h/mqsResponse.jpg"><img style="cursor: pointer;" src="http://2.bp.blogspot.com/_0EFq8rT981w/SE6AMsxrj1I/AAAAAAAAABM/N3vZAdSeQbM/s400/mqsResponse.jpg" alt="" id="BLOGGER_PHOTO_ID_5210242774580563794" border="0" /></a></span></div> </div><span style="font-size:130%;"><br />So if you want to send a request and get the response by WMQ you need to do the next steps :<br /></span><ol><li><span style="font-size:130%;">Create send solicit-response port, configure it to Mqseries/MQC adapters.</span></li><li><span style="font-size:130%;">Create Correlation Set based on MQSeries.Biztalk_CorrelationID<br /></span></li><ol><li><span style="font-size:130%;">Add Mqseries.dll to your project, from Biztalk bin directory.</span></li><li><span style="font-size:130%;">Create the correlation type,You can make the correlation only by the MQSeries.Biztalk_CorrelationID because the adapters update it automatically. On send to queue they copy the MQMD_MsgID to MQSeries.Biztalk_CorrelationID and on the receive the adapter copies the value of MQSeries.Biztalk_CorrelationID to MQMD_CorrelationID.</span></li></ol><li><span style="font-size:130%;">Initialize the Correlation set using "dummy send" port (Null/Nope adapters)<br />You cannot initialize it on the receive shape like in the following orchestration , because you will get this error when compile :</span><span style="font-style: italic;font-size:130%;" ><span style="color: rgb(255, 0, 0);"> "</span><span style="color: rgb(255, 0, 0);">in a sequential convoy the ports must be identical<span dir="rtl">" </span></span></span><span style="color: rgb(255, 0, 0);font-size:130%;" >.</span><span style="font-size:130%;"><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_0EFq8rT981w/SE6FbJq8a-I/AAAAAAAAABU/i6ybSBUZtD8/s1600-h/mq+convoy+issue.jpg"><img style="cursor: pointer; width: 331px; height: 145px;" src="http://1.bp.blogspot.com/_0EFq8rT981w/SE6FbJq8a-I/AAAAAAAAABU/i6ybSBUZtD8/s400/mq+convoy+issue.jpg" alt="" id="BLOGGER_PHOTO_ID_5210248520413244386" border="0" /></a><br /><br />you get this error because biztalk cannot determine which message it will get first, the message of the solicit-response (receive shape which initialize the correlation set) or the message of the response followed up the Correlation set.</span></li><li><span style="font-size:130%;">Set the follow correlation set attribute on the needed receive shape.<br /><br />enjoy<br /><br /></span></li></ol><br /><br /><div class="c" style="margin-left: 1em; text-indent: -2em;"><span class="m"><</span> <span class="t"> </span></div><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com1tag:blogger.com,1999:blog-7665617450155359726.post-13219081374983783652008-06-08T14:54:00.013+03:002008-06-10T18:43:57.845+03:00Biztalk and MQ Correlation Set with NOPE/ NULL adapter causing Race Condition.<span style="font-size:130%;">In my integration scenario I needed to implement request/response interface between application running on Mainframe (Supplier) platform and windows (Biztalk ) . The communication between the platforms done with WMQ.<br /></span><br /><span style="font-weight: bold;font-size:130%;" >General MQ Stuff</span><span style="font-weight: bold;font-size:180%;" ><br /></span><span style="font-size:130%;"><br />In WMQ you have great feature for Correlation. When putting a message to a queue with MQMD_MsgID set to Null, WMQ Server generates a unique ID for this message and thus you can correlate the transaction with this MQMD_MsgID. The supplier get the message from queue, do it's stuff, sets the Correlation_ID of the response message to the MQMD_MsgID of the received message and puts it to queue.<br />All the client need to do is to listen on the queue for a message with Correlation_ID equal to it request MSG_ID.</span><br /><br /><span style="font-weight: bold;font-size:130%;" >Biztalk implementation</span><br /><br /><span style="font-size:130%;">Biztalk support this way of work with it's MQSeries/MQC adapters using the solicit response port type. When issue the put to queue command with the adapter, you get the MQMD_MsgID which was assigned to your message by the WMQ Manger. The adapter copies it's also to the MQSeries.BizTalk_CorrelationId, thus you can work only with it and not using the MQMD_MsgID/MQMD_CorrelationID (on receive).<br /><br />To implement this you need to do the next steps, by best practices :<br /></span><ol><li><span style="font-size:130%;">Create solicit-response port for the destination wmq queue.</span></li><li><span style="font-size:130%;">Create Correlation Set based on MQSeries.BizTalk_CorrelationId. this context property will be initiate automatically by the adapter with the MQMD_MsgID created by the WMQ Manager after issue the send to queue port.<br /></span></li><li><span style="font-size:130%;">Create "dummy send" port for dev null/ NOPE adapter for the</span><span style="font-weight: bold;font-size:130%;" > Correlation Set initialization</span><span style="font-size:130%;">. (*)<br /></span></li><li><span style="font-size:130%;">Setting the Correlation set on the receive shape, waiting for the responce. The adapter will set the MQSeries.BizTalk_CorrelationId with the value of the MQMD_CorrelationID.<br /><br />(*) <span style="font-style: italic;">see my <a href="http://linderalex.blogspot.com/2008/06/mq-correlation-set-initialization-in.html">previous post </a>on this subject.</span><span style="font-style: italic;"></span><br /></span></li></ol><span style="font-size:130%;"><span style="font-weight: bold;">The Problem</span><br /><br />seems simple yes, but there is a huge problem in step 3 and 4 because they are not an atomic transaction.<br />What will happen if biztalk will get a response message prior the setting of the Correlation Set, you will get the no subscribers error, because the Orchestration did not succeeded yet to subscribe it self in the message engine for this particular message.<br /></span><span style="font-weight: bold;font-size:130%;" ><br />We got a Race Condition problem.<br />Yep I got it , the roundtrip of the message from Biztalk to Mainframe , suppliers work, and back took less time than the Correlation Set initialization in biztalk. Surprised I WAS too.</span><br /><span style="font-size:180%;"><br /><br /><span style="font-weight: bold;">Solution</span></span><br /><br /><span style="font-size:130%;">So to solve it I needed to write my own code for unique message id generation and set the correlation set before sending the request message with it.</span><br /><span style="font-weight: bold;"><br /><span style="font-size:180%;">Summery</span><br /><br /><span style="font-size:130%;">Be aware of biztalk publish/subscribe latency.<br />On my next post I will give some tips for it optimization by administration table configuration. In this scenario it didn't help.</span><br /><br /></span><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com1tag:blogger.com,1999:blog-7665617450155359726.post-19688206499577043402008-06-08T14:40:00.004+03:002008-06-08T22:12:55.727+03:00Biztalk SQL Server management and optimization<span style="font-size:130%;">I found two great posts about biztalk server 2004/2006 DB optimization, deadlock solutions.<br />I suggest everyone who work with biztalk to read this posts.</span><span style="font-weight: bold;"><br /></span><ol><li><span style="font-size:130%;"><a href="http://support.microsoft.com/kb/917845">http://support.microsoft.com/kb/917845</a></span></li><li><span style="font-size:130%;">Biztalk core engine's WebLog - <a href="http://blogs.msdn.com/biztalk_core_engine/archive/2007/01/04/what-you-can-and-can-t-do-with-the-messagebox-database-server.aspx">What you can and can't do with the Messagebox Database server.</a><br /></span></li></ol><span style="font-weight: bold;font-size:130%;" >Summery :<br /></span><ul><li><span style="font-size:130%;">Don't enable/use statistics.</span></li><li><span style="font-size:130%;">Shutdown the parallelism.</span></li><li><span style="font-size:130%;">Don't built maintenance plan for indexes, use only <span style="height: 10px; font-family: arial; font-size: 9pt; color: rgb(0, 0, 153);" dir="ltr" id="spnTrans1"><span style="font-weight: bold;"></span></span><span style="font-weight: bold; font-family: arial;"> </span>bts_RebuildIndexes stored procedure, </span><span style="font-size:130%;"><span style="font-weight: bold;">shutdown biztalk engine first.<br /></span></span></li><li><span style="font-size:130%;">Don't change the stored procedure code.<span style="font-weight: bold;"><br /></span></span></li></ul><span style="height: 10px; color: rgb(0, 0, 153);font-family:arial;font-size:9;" dir="ltr" id="spnTrans1" ><span style="color: rgb(0, 0, 0);font-size:100%;" ><span style="font-size:130%;">enjoy,</span><span style="font-weight: bold;"><br /></span></span></span><div class="blogger-post-footer">Alex Linder.</div>Alex Linderhttp://www.blogger.com/profile/11481294606203072803noreply@blogger.com0