IBM Business Process Manager - Mastering Pitfalls

IBM Business Process Manager erstwhile Teamworks Lombardi is one of the top selling BPM suites in the market. IBM Business Process Manager has some excellent features and some innovative concepts which enhance BPM development. However it is imperative for an organization to know about its key features and some of the disadvantages it brings along with its features. These short comings can be easily overcome and one easiest way to do that is to become aware of the short coming itself.

Branching and Merging in IBM Business Process Manager

IBM Business Process Manager has a shared development model. Unlike the traditional software development, the IBM Business Process Manager code base is maintained in a Process Center environment. A Process Center environment is not a source control by itself; however the IBM Business Process Manager artifacts are stored in the Process Center. The code base for a process application is maintained in a Process Center.

The run-time instance (executable) of the process application (code base) is known as Snapshot. A Snapshot can be deployed into any of the IBM Business Process Manager run-time process servers. The code base for a process application cannot be branched nor merged. Although, branching can be done by creating a Workspace in IBM Business Process Manager, there is no way to merge the branches.

This important aspect of this short coming in IBM Business Process Manager is that the Agile projects have to be planned meticulously. If a project plan mandates parallel development for a same process application, then there is no way in IBM Business Process Manager to merge the parallel branches into a single trunk.

Process instance data cannot be modified

Long running process instances are vulnerable to failures. Some of the failed instances may be irrecoverable due to data issues. For example, a particular variable in a business process needs to be set to a certain value to take a certain path in the business process flow. The business process data due to system issues or code issues might have got a wrong set of data and it would become necessary to read and update the business process data when the process is active. IBM Business Process Manager has an API to read the data, unfortunately does not have a good way to update the data, since the process data is stored as BLOB.

To circumvent this problem, the usage of business process data has to be minimized. Only business key attributes or in layman terms, primary key attributes in the application data needs to be mapped to the business process data. By mapping the primary key data as business process data, a loose coupling between the business process and the application data is achieved, thereby the application data would have more freedom to change without the necessity of keeping the business process data in sync.  There is a huge debate on what needs to map to business process data and what has to go into the application data, which itself becomes a separate topic to explore and it is well outside the context of this article.

IBM Business Process Manager stores most of the process related data as BLOB's

A typical software application’s life cycle regions include Development (DEV), System Integration Test (SIT), User Acceptance Environment (UAT) and Production (PROD). After every release to production, a good practice is to mirror the entire PROD data into different regions. This is no exception for BPM applications, and IBM Business Process Manager presents itself another challenge to implement this setup. Since process data in IBM Business Process Manager are stored as BLOB’s, there is a challenge for DBA’s to replicate the BLOB’s in lower region databases.

Also if the application data model and the IBM Business Process Manager data model are two different schemas, then there is an additional challenge in maintenance of data states after the production refresh. It becomes imperative for DBA’s to understand the complete data model of the Process Manager to synchronize the tables required to reflect the state and to ignore the tables which represents the run-time dynamics.

Also refer, BPM Data Model Architecture.

Workspace cannot be deleted

The need for Workspace arises when the code needs to be branched out of a particular Snapshot. For example, if there is a defect in a production release and the current Development release has progressed into the next version of Snapshot, a branch of production Snapshot is required and Workspace comes for rescue. Any version of the Snapshot can be branched out to a Workspace, however it cannot be merged to another Workspace as discussed already. Workspace created cannot be deleted from Process Center leaving a scar for potential misuse. Multiple Workspace would lead to confusion and there is always a possibility of developer choosing a wrong Workspace.

Creation of Workspace cannot be avoided, but with proper naming convention and standards, a potential misuse or accidental usage of a wrong Workspace for a wrong version can be avoided.

Process applications in Process Center cannot be deleted

Process applications in Process Center cannot be purged. This is mostly a sanity related aspect of maintaining the code base in the Process Center. Over a period of time the Process applications tend to grow and there would be a need to purge some of the obsolete Process Applications which would be no longer required. Unfortunately there is no way in IBM Business Process Manager, a Process application can be purged. A better way to avoid this situation is to have tighter governance and strict privileges within the developer community on who can create a Process application.

REST Services with Lombardi

This article is a supplement of Lombardi on mobile, in which it was discussed to have Lombardi services exposed as REST API for enabling mobile communication. REST is a very sought after API for any product due to the fact that REST is capable of emitting JSON which can be easily consumed by Web 2.0 and mobile applications. Lombardi support provides a REST API which extensively uses Lombardi Web API which Lombardi does not recommend after TeamWorks version 6. The objective of this article is to outline an approach to expose RESTful services of Lombardi API without using Lombardi Web API.

This post outlines a methodology to expose RESTful services for some of the common tasks of Lombardi like, process initiation and adding comments to a process instance taking them as examples. Any API call from Lombardi which needs to be exposed as REST should be exposed as a SOAP Webservice, since Lombardi does not provide the capability to directly expose the API calls as RESTful service. If a BPD needs to be invoked, a general system service needs to be exposed as SOAP Webservice. The actual instantiation of the process should happen with the general system service.




For example, a BPD can be invoked using the following command, embedded in a Server script activity of general system service.

// to start a process var processInstance = tw.system.startProcessByName(processName, InputParameterMap); // to add comment to a process instance var processInstance = tw.system.findProcessInstanceByID(processId); processInstance.addComment("Hello!");

Lombardi provides a Webservice implementation component which can be used to expose general system service as SOAP Webservices. A stand alone service layer built on Spring3 framework would consume the SOAP Webservices and convert them to RESTful services using Spring3 REST support. Stand alone services can decouple the transformation logic from Lombardi. Also, deploying them as stand alone would mean that Lombardi product code is not tinkered and hence not endangering future product patches and also this helps fully leveraging Lombardi warranty support.

Below are the examples of code snippets of Spring controller which calls a SOAP Webservice (Lombardi API exposed as SOAP Webservice) and exposes the method as RESTful service which would emit XML or JSON.

// controller method to invoke a process @RequestMapping(method = RequestMethod.GET, value = "/startCalcGDPProcess", headers = "Accept=application/xml, application/json") public @ResponseBody String startCalcGDP() throws Exception { RunGDPPortTypeProxy proxy = new RunGDPPortTypeProxy(); String pi = proxy.startGDP(); return pi; } // controller method to add a comment to a process instance @RequestMapping(method = RequestMethod.POST, value = "/addCommentToGDP", headers = "Accept=application/xml, application/json") public ModelAndView addCommentToGDPProcess(@RequestBody String body) { Source source = new StreamSource(new StringReader(body)); ProcessInstance e = (ProcessInstance) jaxb2Mashaller.unmarshal(source); addCommentToGDPProcess.add(e); return new ModelAndView(XML_VIEW_NAME, "object", e); }
A sample Spring servlet configuration for REST based services.
<!-- To enable @RequestMapping process on type level and method level --> <bean class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping" /> <bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter"> <property name="messageConverters"> <list> <ref bean="marshallingConverter" /> <ref bean="atomConverter" /> <ref bean="jsonConverter" /> </list> </property> </bean> <!-- Client --> <bean id="restTemplate" class="org.springframework.web.client.RestTemplate"> <property name="messageConverters"> <list> <ref bean="marshallingConverter" /> <ref bean="atomConverter" /> <ref bean="jsonConverter" /> </list> </property> </bean> <bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller"> <property name="classesToBeBound"> <list> <value>dw.spring3.rest.bean.ProcessInstance</value> </list> </property> </bean> <bean id="lombardiRestAPI" class="org.springframework.web.servlet.view.xml.MarshallingView"> <constructor-arg ref="jaxbMarshaller" /> </bean> <bean class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver"> <property name="mediaTypes"> <map> <entry key="xml" value="application/xml" /> <entry key="html" value="text/html" /> </map> </property> <property name="viewResolvers"> <list> <bean class="org.springframework.web.servlet.view.BeanNameViewResolver" /> <bean id="viewResolver" class="org.springframework.web.servlet.view.UrlBasedViewResolver"> <property name="viewClass" value="org.springframework.web.servlet.view.JstlView" /> <property name="prefix" value="/WEB-INF/jsp/" /> <property name="suffix" value=".jsp" /> </bean> </list> </property> </bean> <bean id="marshallingConverter" class="org.springframework.http.converter.xml.MarshallingHttpMessageConverter"> <constructor-arg ref="jaxbMarshaller" /> <property name="supportedMediaTypes" value="application/xml" /> </bean> <bean id="atomConverter" class="org.springframework.http.converter.feed.AtomFeedHttpMessageConverter"> <property name="supportedMediaTypes" value="application/atom+xml" /> </bean> <bean id="jsonConverter" class="org.springframework.http.converter.json.MappingJacksonHttpMessageConverter"> <property name="supportedMediaTypes" value="application/json" /> </bean>

This approach might sound little bit awkward for having a RESTful service invoking a SOAP Webservice, however this methodology effectively decouples the Lombardi API calls from generating RESTful services.

IBM Business Process Manager on Mobile

There is an increasing demand and need for BPM suites to extend its capabilities to Mobile platforms. IBM Business Process Manager does not provide out of the box features for Mobile applications; however IBM Business Process Manager does have a REST API which can be used for Mobile communication. The REST API provided by IBM Business Process Manager utilizes the Web API which has been discarded from TeamWorks 6. The REST API still exists and could prove to be useful too, but it is getting deprecated. Also, IBM Business Process Manager recommends all process data access through its JavaScript API from TeamWorks 7 and it extends for WebSphere editions too. This article aims at providing an approach for enabling mobile users communicate with IBM Business Process Manager.

Exposing IBM Business Process Manager API as RESTful services

Applications built on IBM Business Process Manager lives on intranet (corporate domain), which means a browser from a mobile phone with internet cannot be used to access IBM Business Process Manager portal on corporate domain. Even if an organization has exposed IBM Business Process Manager portal over internet, the usability feature of the portal will be hampered by the mobile browser. This boils down to the fact that dedicated mobile applications should be built for IBM Business Process Manager specific tasks. This article provides an approach to enable mobile app communication on platforms like Android, iOS and Symbian from IBM Business Process Manager and is not a tutorial to build the mobile application itself. A subsequent post would be published on building an Android application for IBM Business Process Manager tasks.

Some of the common activities users would wish to perform from their mobile application would be like, initiate a process, add comments to a process instance, complete a task, etc. One common protocol mobile apps use to perform for cross platform communication is Java Script Object Notation (JSON). Mobile application tool kits and SDK already have lot of libraries to write and read JSON. Choosing some other protocol other than JSON/HTTP would mean, that a lot of logic for mobile apps needs to written from scratch.

The activities required to be performed from a Mobile application needs to have corresponding services in IBM Business Process Manager which can throw JSON so that mobile apps can consume. So for instance, an activity like Create Bank Account process, if needs to be invoked from a mobile application, the mobile app would invoke a REST IBM Business Process Manager service (General System Service), which in turn would trigger the Create Bank Account process. IBM Business Process Manager has a JSON toolkit which has very limited capabilities and lot of other functions needs to be built to make it as a complete JSON toolkit. There are different approaches in which a IBM Business Process Manager service can be converted into a Restful service.

• Use JSON toolkit provided by IBM Business Process Manager support.
• Use REST API provided by IBM Business Process Manager support. REST API extensively uses Web API to access process data. As a word of caution, Web API is deprecated from TeamWorks6.
• Build a stand-alone REST services with Spring3, where the stand-alone REST services would be consumed by the Mobile apps and the REST services would invoke IBM Business Process Manager services via SOAP based Webservices.
• Another deviation from the 3rd option can be instead of REST services invoking IBM Business Process Manager services via SOAP based Webservices; REST services can invoke exposed URLs from IBM Business Process Manager. But, exposed URLs from IBM Business Process Manager are limited to executing a process and completing a task.

The 3rd option seems like a more reasonable and scalable option since, a lot of mobile app features can be incorporated in house. Moreover Spring3’s REST capabilities can be fully leveraged for JSON generation. The weird thing with this approach is that it uses REST services to invoke SOAP based services.



In real world, mobile apps are outside corporate domain network, it is imperative to expose the REST based services in internet. So a robust authentication mechanism is required to secure the REST based services over internet. Spring security and Spring Android security can be fully utilized with OAuth mechanism to fulfill any security constraints.

This article will be supplemented with two others posts, one for Developing Spring3 REST services which would expose IBM Business Process Manager Mobile API functions, and other post for building an Android app for IBM Business Process Manager tasks.

Advanced Reporting in Lombardi

Lombardi provides two flavors for Reporting. Adhoc Reporting let developers create quick charts and graphs in no time, without coding and with minimal configurations. The other flavor is custom reporting which is quite flexible with one or more SQL queries and data transformation logic with filtering of data.

Lombardi documentation provides enough information about creating Adhoc and custom reports. This article is aimed at creating better custom and advanced reports using Lombardi and this is not a tutorial or a guide to create reports. Please browse Lombardi documentation for user guides to create reports. Before getting into Lombardi custom reports, it is imperative for an analyst to understand the Lombardi Performance Data Warehouse Architecture and key concepts in performance reporting.

Basically, reports are just representation of business data, and Lombardi creates reports using business data which are tracked in the business process. Tracing business data is a key concept and in Lombardi business data can be either auto-tracked or it can be manually tracked with tracking events.



Tracking business data manually with tracking events offers several advantages. A tracking group needs to be created for manual tracking, which is used to group similar business data in a business process. From the business process, business data is fed into the tracking group using tracking events. Please see ‘Start tracking GDP’ as tracking event in the above business process diagram. So a tracking event essentially captures business data which needs to be tracked and sends it to the Performance Warehouse DB. In technical terms, a tracking group translates to a database view in Lombardi Performance Warehouse DB. The same is the case for auto-tracked business data, but Lombardi captures auto-tracked events during the start and end of each activity, deteriorating the overall performance of the BPM system. But in manual tracking, tracking events can be placed in business process exactly at an activity where tracking is essential, thereby reducing the round trips between Performance Warehouse server and Process server.

Lombardi comes with default tracking groups such as PROCESSFLOWS, SLASTATUS and SLATHRESHOLDTRAVERSALS. With default tracking groups, reports on SLA’s and violations can be easily created.

In custom reporting, a SQL query needs to be plugged into a chart as data source. The below link has variety of techniques to formulate an SQL query for analysis and reporting. http://download.oracle.com/docs/cd/B19306_01/server.102/b14223/analysis.htm The SQL query needs to be framed with aggregate functions for data columns with a GROUP BY clause as a must. Along with the query, data transformation logic has to be plugged into the data source as well. Lombardi provides around 7 in-house built report transformation services. Analysts can write their own data transformation logic by developing report transformation which parses the data which is fed from the SQL query and then inputs them to a Lombardi JavaScript function addDataToSeries(seriesValue, labelValue, displayValue).

Having created a data source, it’s time to build a chart. Lombardi comes to rescue with 9 default chart layouts defined in Lombardi system toolkit. The input for charts is data source. The data source along with its embedded report transformation logic converts the data into chart. Custom chart layouts can be defined using Chart Definition Language (CDL). Existing Lombardi chart layouts can be altered by replicating their CDL and altering the chart properties.

Finally, one or more charts can be embedded as pages (HTML) and can be published as scoreboards. Below are some of chart samples generated using Lombardi.





Adam Deane had an interesting point to make about business data being not used in BPM Reporting capabilities. Fortunately, Lombardi just does that. Lombardi captures process business data and generates reports out of it. Process business data is captured in performance DB as relational data unlike in process server where process data is persisted as CLOBs and BLOBs.

IBM Business Process Manager Best Practices

There is no dearth of information available in IBM Business Process Manager support site about best practices. The best practices recommendations in the support site are highly recommended. However this article is not aimed at repeating those technical aspects of the best practices, but rather would highlight some of the issues which had to be dealt from an architectural, design and requirement gathering perspective.

I have already posted an article on Best Practices for BPMS Design and Architecture, and the article is generic rather than tool centric. I would highly recommend reading the same and all the best practices and guidelines mentioned in that article, are applicable to IBM Business Process Manager as well.

In-flight instance management

One of the most challenging aspects of BPM architecture is in-flight instance management. It is a bitter truth that people learn about the best practices involved in managing in-flight instances after their first successful project release, which is too late. There are various aspects of design elements which need to be applied to take care of in-flight instances.

Process Modeling – “Encapsulate what varies”

A process model must be an abstraction of the business process. Going by these lines, it is critical to identify the part of the process model which is going to change frequently. Of course, this needs some serious business thinking. It is good idea to encapsulate the variations in process model into a sub-process. Any changes in future would not affect the top level business model but rather changes in sub-process. So if there are major changes in sub-process from version1, then a new version (version2) of sub-process can be replaced with changes from version1. Again, this won’t have any effect on the parent process. But parent process should have a variable based routing mechanism. For instance, a variable named version number should be declared in the parent process, and this variable should be incremented for each release. Based on the version number in parent process, either version1 sub-process or version2 sub-process can be invoked.

Process Data Structure – “Open for extension, closed for modification”

Another design element which is quite crucial for in-flight instance management is process data structure. IBM Business Process Manager recommends no drastic changes to process data structure, and this will directly affect the in-flight instances. The process data structure should strictly represent the domain model, but not necessarily with all attributions, but only with key elements and identifiers. Any element removed from the data structure would break the in-flight instances. So, detailed attention is required in designing process data structure. In short process data structure should be closed for modification but open for extension.

Testing of migrated instances

With having the best in-flight instance handling mechanism, it is no guarantee that the migrated instances would work without proper testing. It is a good idea to mimic the production environment to a stage environment and test all the migrated instances. The IBM Business Process Manager production database must be replicated in stage so that the process instances would be left in the same state as in production. This would give a good insight of the actual behavior of migrated instances in stage before going LIVE.

Coaches

One principle which has to be religiously followed when it comes to IBM Business Process Manager coaches is DRY (Don’t Repeat Yourself). It can be as simple as a header logo of a company which should be embedded as a custom HTML component rather than embedding in every coach. If a coach needs to be duplicated then it is rather a good practice to change the encapsulating general system service which holds the coach. For instance, if a same coach needs to be presented with different data set, then the services have to be duplicated to load the respective data set, without duplicating the coaches.

The same principle applies for authorization of HTML elements. It is advisable to encapsulate the common HTML elements in a custom HTML component and invoke the custom component from different coaches.

Some of business requirements may mandate for extreme customization of coaches. IBM Business Process Manager recommends the usage of Yahoo User Library (YUI). In my personal opinion, jQuery may not be bad option at all.

Process Instance Search

A ubiquitous requirement for any BPM solution would be a process instance search framework. From a user point of view, it is quite important and fundamental functionality since without a specific instance intended for the user, the user will have no activity to work on. Zeroing on a particular process instance would depend on the business data which often does not reside in the IBM Business Process Manager database. But IBM Business Process Manager comes to rescue in terms ‘Shared Search’ feature. However this comes with additional caveats. The user would require exposing some search parameters (business data) in process and these would be available in search criteria. This imposes several limitations, and it is not often that all business data would be available in process data structure which can be exposed as a search parameter.

This drives technical folks to circumvent the search functionality. A better suggestion would be to build a custom search framework. So here it is imperative to associate the process instance identifier of IBM Business Process Manager with the business data which can reside in an application schema. So if business data can be filtered, its associated process instance identifier can be fetched and in turn it can be associated with BPM artifacts.

Exception Handling
http://bpmstech.blogspot.com/2011/05/exception-handling-in-bpm.html

Conclusion

This is of course not a definitive list of items. But due to limited time, I’ve published my thoughts, but there is more to come.