Agenda
BPM in the Cloud - Advantages
- Multitenancy
- Dynamic Scaling
- Elastic Resource Capacity
Identity Management for BPM Solutions
Cloud-to-on-premise Connectors
BPM Topologies
BPM in the Cloud Architecture
Thursday, August 15, 2013
Why process mining is important to BPM?
A different perspective on process mining approach - my latest post published on BPMLeader.com
Why process mining is important to BPM?
Why process mining is important to BPM?
Friday, February 24, 2012
BPM System Architecture
The BPM System Architecture is
the workhorse of an organization’s BPM vision to fulfill its strategic BPM
needs and objectives. This article aims to provide insights into the BPM System
Architecture and how it should be devised and evolved to fully support a
seamless, scalable and dynamic platform for process implementation. Most
organizations in today’s world would purchase a packaged BPM suite which has
abundant features out of the box that support’s the entire BPM lifecycle,
rather than an in-house built custom BPM platform. The need for a BPM System
Architecture is even more imperative for a packaged BPM suite, since the
packaged products must be judiciously aligned within the scope of the
enterprise architecture (EA) of an organization. A home grown BPM platform
would have a lesser challenge in terms of fitment into enterprise architecture
since it would have evolved within the EA scope right from the beginning.
The packaged BPM suites would
seem to be more exciting and successful when implemented on a small scale like
proof of concepts. However when these suites are stretched for a full fledged
implementation, its short comings may lead to a colossal failure. A BPM suite can handle 10 process
implementations with ease, but when it comes to hundreds and thousands of
processes, a trivial mistake or a minuscule short coming in the BPM suite would
proportionately increase the rate of failure.
It is imperative that a BPM
System Architecture is built so that a packaged BPM suite can be seamlessly
integrated into an Enterprise Architecture. This article will provide insights
and glimpses of BPM reference architecture.
Process Data versus Business Data
The process data is the one which supplements the process, but not crucial for the actual completion of the process. KPI, SLA and metrics are few examples of process data. A KPI or SLA might be based on a business data but as such it may not be required for process execution. Business data is the actual driver of the process and would take it to completion. In an Insurance industry the business data would look like, an insurance id, a claim id, insurance expiry date etc. Business data is critical for process execution and the process directly depends on the business data for the process to complete.
The BPM System Architecture
should clearly differentiate between the process data and the business data.
The business data would most probably reside in an independent repository
(database) and would be accessed via an abstraction layer like SOA. The process
data would be derived from the business data by the BPM suite and would reside
in the BPM suite database.
The BPM System Architecture must
clearly define the business data identifiers that a process should be aware of.
For instance, an insurance process must be only aware of insurance id. Anything
beyond that would be a nightmare to sync the process and business data. If the
process data is defined with insurance id and insurance expiry date, then any
changes to the insurance expiry date in the external systems must be synced
with the process data, which would be a nightmare if the number of business
attributes in the process data grows. Instead the process should only be aware
of the insurance id and when the process requires the insurance expiry date, it
needs to be pulled up from the external system using the insurance id. In other
words, the best practice to define process business attributes is to define it
with unique business identifiers. An insurance id would be a unique identifier
and all relevant details pertaining to the insurance policy can be pulled from
the insurance id.
Process data like KPI and SLA
can be derived again from business data. If an SLA needs
to be defined on the insurance expiry date from the process, with the insurance
id, the insurance expiry date required for SLA
computation can be pulled from the external system and SLA
can be derived. The SLA data for the insurance expiry
date can reside in the process repository.
The BPM System Architecture should
define a framework (like an integration component) which would act like a nerve
centre to pull business date from the process with the help of unique business
key attributes.
Long running process
Failure of process instance cannot be always avoided. But what would be unavoidable, is of course the graceful recovery of the process instance. A process instance can fail for variety of reasons ranging from unavailability of system resources, data issues, network outages and technical glitches. The more prominent of these are the data issues.
A process instance in a telecom infrastructure provider would typically execute for months. If the telecom provider has a process implementation, to lay trans-Atlantic cables from continental United States to Europe, such a process instance can get executed for multiple months. In due course of time, if the process instance fails, the situation may not demand to create the process instance altogether from the beginning of the process and it is more likely it may not be possible at all, since the business data has undergone changes in due course of the process traversal. In such scenarios the process instance no matter what has to be gracefully recovered.
Failure of process instance cannot be always avoided. But what would be unavoidable, is of course the graceful recovery of the process instance. A process instance can fail for variety of reasons ranging from unavailability of system resources, data issues, network outages and technical glitches. The more prominent of these are the data issues.
A process instance in a telecom infrastructure provider would typically execute for months. If the telecom provider has a process implementation, to lay trans-Atlantic cables from continental United States to Europe, such a process instance can get executed for multiple months. In due course of time, if the process instance fails, the situation may not demand to create the process instance altogether from the beginning of the process and it is more likely it may not be possible at all, since the business data has undergone changes in due course of the process traversal. In such scenarios the process instance no matter what has to be gracefully recovered.
Let us assume that for the
process described above, the process instance gets stuck at an activity where
the process instance expects the length of the cable between pier A and pier B
to be of certain length, whereas the business data for cable length is out of
that range for what the process instance expects. For the process instance to execute, the
business data for cable length needs to be updated in the source record. The
BPM System Architecture should have a mechanism to refresh the process with
latest business data after the update and should recover gracefully.
Also, all long running processes
which are vulnerable to failures and are critical to business wherein the re-creation
of the instance is not possible, the actual business process must be
supplemented with a background recovery process. The background recovery
process must be designed in such a way that it must be capable enough to
replicate any process instance and can hold the same state as the failed
process instance before the failure. The input to the recovery process must be
the business key attribute and the activity of the process from where the
process instance has to start. This input activity would be the same as the
failed process instance step. In order to design such a recovery process, it is
absolutely essential to separate the business data and process data. Without
this it would be very difficult to create a process instance from a specific activity
within the process, since the newly created process instance should reflect the
exact state of the failed process instance. This is only possible by
associating the business data of the failed process instance to the newly
created instance.
A large scale BPM implementation would require a unified user interface for the business user community. The out of the box portal packaged in the BPM suite may not be flexible enough for extreme customizations. Also, process instance search is a ubiquitous functionality which would act as a first interfacing point for the user.
The BPM System Architecture is
responsible for a process instance search framework. The challenge to build
such a framework would be to integrate both the process and the business data
in one single view. A process instance can be searched with a business
attribute as a parameter and also the search can be performed with process data
such as SLA.
One possible technical solution is to combine the process data and business data together as Database view, by which expensive database joins can be avoided between multiple tables of the business data schema and process data schema.
BPM Exception Handling
Exception handling is one of the prime aspects of software design which attributes to the robustness of the application. Exception handling gains more significance in BPM space, since exceptions had to be dealt both at a system level and at a business level.
The significance of exception handling at a process modeling step cannot be ignored. There is a need to handle ACID transactions in BPM, with a business solution and it has to be captured in the model with business semantics. For instance, a classic example of flight, hotel and rental car booking when deemed as a single atomic transaction cannot always be dealt at a system level. If flight booking and hotel booking are two different applications in two different domains (if flight carrier and hotel are different companies), then system transactions cannot be propagated and even if it seems to be possible, system level locks for transaction may not be possible to be obtained since the whole transaction may exceed beyond days to complete. The ideal way to deal with this situation is to have three distinct transactions as business process and even if one fails the other transactions must be canceled using a business process. If flight booking has been confirmed and if hotel booking transaction fails, then flight booking should also be canceled, wherein flight cancellation must be modeled as a separate business process.
It becomes imperative for a BPM solution to handle system level exceptions gracefully. Since process instances can span across business days, in case of system exceptions the process state has to be persisted and should be able to recover from the state where it was left. This may not be possible always for each and every process instance. Sometimes there is a good chance that a process instance may get into an irrecoverable state. In such scenarios, it may not be desirable for the users to create a new process instance altogether. The alternatives for irrecoverable instances due to system exceptions are few, and one probable solution could be to create a new process instance and associate the
application process data with the new instance, and programmatically navigate process the instance to the workflow step where it failed.
Another alternative is to create an exception business process. Any system level exception occurring in any of the application business process would trigger the exception business process, which would notify the user about the exception. Also exception process can notify a repair/admin team about the exception. Having a repair queue of exceptions can help the repair/admin team to get first hand information about the exception without the need for notification from the business users about what had happened. Also, irrecoverable process instances can be dealt internally within the IT team, without burdening the business users.
Process Versioning
A process undergoes lot of changes in a BPM lifecycle with multiple iterative implementations. The BPM System Architecture must be potent enough to handle multiple versions of process instances. Most of the modern BPM suites come pre-packaged with utilities to migrate the process instances from an obsolete version to higher versions. However, a process instance in an older version may not be always possible to migrate to a higher version, due to external system dependencies, when the external system itself would have undergone a version change along with the process. This is again in-line with the process data versus business data debate. The battle lines between process data and business data have to be carefully drawn in such way that any data on external system should not be a part of process data. A better way to hold reference to external application data is to hold reference to the unique business key attribute of the external system.
Conclusion
These are the some of the many items that must be considered for a BPM System Architecture. If the segregation between process data and application data can be achieved then having the BPM System Architecture would bring the additional flexibility in terms of solution design.
One possible technical solution is to combine the process data and business data together as Database view, by which expensive database joins can be avoided between multiple tables of the business data schema and process data schema.
BPM Exception Handling
Exception handling is one of the prime aspects of software design which attributes to the robustness of the application. Exception handling gains more significance in BPM space, since exceptions had to be dealt both at a system level and at a business level.
The significance of exception handling at a process modeling step cannot be ignored. There is a need to handle ACID transactions in BPM, with a business solution and it has to be captured in the model with business semantics. For instance, a classic example of flight, hotel and rental car booking when deemed as a single atomic transaction cannot always be dealt at a system level. If flight booking and hotel booking are two different applications in two different domains (if flight carrier and hotel are different companies), then system transactions cannot be propagated and even if it seems to be possible, system level locks for transaction may not be possible to be obtained since the whole transaction may exceed beyond days to complete. The ideal way to deal with this situation is to have three distinct transactions as business process and even if one fails the other transactions must be canceled using a business process. If flight booking has been confirmed and if hotel booking transaction fails, then flight booking should also be canceled, wherein flight cancellation must be modeled as a separate business process.
It becomes imperative for a BPM solution to handle system level exceptions gracefully. Since process instances can span across business days, in case of system exceptions the process state has to be persisted and should be able to recover from the state where it was left. This may not be possible always for each and every process instance. Sometimes there is a good chance that a process instance may get into an irrecoverable state. In such scenarios, it may not be desirable for the users to create a new process instance altogether. The alternatives for irrecoverable instances due to system exceptions are few, and one probable solution could be to create a new process instance and associate the
application process data with the new instance, and programmatically navigate process the instance to the workflow step where it failed.
Another alternative is to create an exception business process. Any system level exception occurring in any of the application business process would trigger the exception business process, which would notify the user about the exception. Also exception process can notify a repair/admin team about the exception. Having a repair queue of exceptions can help the repair/admin team to get first hand information about the exception without the need for notification from the business users about what had happened. Also, irrecoverable process instances can be dealt internally within the IT team, without burdening the business users.
Process Versioning
A process undergoes lot of changes in a BPM lifecycle with multiple iterative implementations. The BPM System Architecture must be potent enough to handle multiple versions of process instances. Most of the modern BPM suites come pre-packaged with utilities to migrate the process instances from an obsolete version to higher versions. However, a process instance in an older version may not be always possible to migrate to a higher version, due to external system dependencies, when the external system itself would have undergone a version change along with the process. This is again in-line with the process data versus business data debate. The battle lines between process data and business data have to be carefully drawn in such way that any data on external system should not be a part of process data. A better way to hold reference to external application data is to hold reference to the unique business key attribute of the external system.
Conclusion
These are the some of the many items that must be considered for a BPM System Architecture. If the segregation between process data and application data can be achieved then having the BPM System Architecture would bring the additional flexibility in terms of solution design.
Tuesday, February 14, 2012
IBM Business Process Manager - Mastering Pitfalls
IBM Business Process Manager erstwhile Teamworks Lombardi is one of the top selling BPM suites in the market. IBM Business Process Manager has some excellent features and some innovative concepts which enhance BPM development. However it is imperative for an organization to know about its key features and some of the disadvantages it brings along with its features. These short comings can be easily overcome and one easiest way to do that is to become aware of the short coming itself.
Branching and Merging in IBM Business Process Manager
IBM Business Process Manager has a shared development model. Unlike the traditional software development, the IBM Business Process Manager code base is maintained in a Process Center environment. A Process Center environment is not a source control by itself; however the IBM Business Process Manager artifacts are stored in the Process Center. The code base for a process application is maintained in a Process Center.
The run-time instance (executable) of the process application (code base) is known as Snapshot. A Snapshot can be deployed into any of the IBM Business Process Manager run-time process servers. The code base for a process application cannot be branched nor merged. Although, branching can be done by creating a Workspace in IBM Business Process Manager, there is no way to merge the branches.
This important aspect of this short coming in IBM Business Process Manager is that the Agile projects have to be planned meticulously. If a project plan mandates parallel development for a same process application, then there is no way in IBM Business Process Manager to merge the parallel branches into a single trunk.
Process instance data cannot be modified
Long running process instances are vulnerable to failures. Some of the failed instances may be irrecoverable due to data issues. For example, a particular variable in a business process needs to be set to a certain value to take a certain path in the business process flow. The business process data due to system issues or code issues might have got a wrong set of data and it would become necessary to read and update the business process data when the process is active. IBM Business Process Manager has an API to read the data, unfortunately does not have a good way to update the data, since the process data is stored as BLOB.
To circumvent this problem, the usage of business process data has to be minimized. Only business key attributes or in layman terms, primary key attributes in the application data needs to be mapped to the business process data. By mapping the primary key data as business process data, a loose coupling between the business process and the application data is achieved, thereby the application data would have more freedom to change without the necessity of keeping the business process data in sync. There is a huge debate on what needs to map to business process data and what has to go into the application data, which itself becomes a separate topic to explore and it is well outside the context of this article.
IBM Business Process Manager stores most of the process related data as BLOB's
A typical software application’s life cycle regions include Development (DEV), System Integration Test (SIT), User Acceptance Environment (UAT) and Production (PROD). After every release to production, a good practice is to mirror the entire PROD data into different regions. This is no exception for BPM applications, and IBM Business Process Manager presents itself another challenge to implement this setup. Since process data in IBM Business Process Manager are stored as BLOB’s, there is a challenge for DBA’s to replicate the BLOB’s in lower region databases.
Also if the application data model and the IBM Business Process Manager data model are two different schemas, then there is an additional challenge in maintenance of data states after the production refresh. It becomes imperative for DBA’s to understand the complete data model of the Process Manager to synchronize the tables required to reflect the state and to ignore the tables which represents the run-time dynamics.
Also refer, BPM Data Model Architecture.
Workspace cannot be deleted
The need for Workspace arises when the code needs to be branched out of a particular Snapshot. For example, if there is a defect in a production release and the current Development release has progressed into the next version of Snapshot, a branch of production Snapshot is required and Workspace comes for rescue. Any version of the Snapshot can be branched out to a Workspace, however it cannot be merged to another Workspace as discussed already. Workspace created cannot be deleted from Process Center leaving a scar for potential misuse. Multiple Workspace would lead to confusion and there is always a possibility of developer choosing a wrong Workspace.
Creation of Workspace cannot be avoided, but with proper naming convention and standards, a potential misuse or accidental usage of a wrong Workspace for a wrong version can be avoided.
Process applications in Process Center cannot be deleted
Process applications in Process Center cannot be purged. This is mostly a sanity related aspect of maintaining the code base in the Process Center. Over a period of time the Process applications tend to grow and there would be a need to purge some of the obsolete Process Applications which would be no longer required. Unfortunately there is no way in IBM Business Process Manager, a Process application can be purged. A better way to avoid this situation is to have tighter governance and strict privileges within the developer community on who can create a Process application.
Thursday, June 16, 2011
REST Services with Lombardi
This article is a supplement of Lombardi on mobile, in which it was discussed to have Lombardi services exposed as REST API for enabling mobile communication. REST is a very sought after API for any product due to the fact that REST is capable of emitting JSON which can be easily consumed by Web 2.0 and mobile applications. Lombardi support provides a REST API which extensively uses Lombardi Web API which Lombardi does not recommend after TeamWorks version 6. The objective of this article is to outline an approach to expose RESTful services of Lombardi API without using Lombardi Web API.
This post outlines a methodology to expose RESTful services for some of the common tasks of Lombardi like, process initiation and adding comments to a process instance taking them as examples. Any API call from Lombardi which needs to be exposed as REST should be exposed as a SOAP Webservice, since Lombardi does not provide the capability to directly expose the API calls as RESTful service. If a BPD needs to be invoked, a general system service needs to be exposed as SOAP Webservice. The actual instantiation of the process should happen with the general system service.
For example, a BPD can be invoked using the following command, embedded in a Server script activity of general system service.
Lombardi provides a Webservice implementation component which can be used to expose general system service as SOAP Webservices. A stand alone service layer built on Spring3 framework would consume the SOAP Webservices and convert them to RESTful services using Spring3 REST support. Stand alone services can decouple the transformation logic from Lombardi. Also, deploying them as stand alone would mean that Lombardi product code is not tinkered and hence not endangering future product patches and also this helps fully leveraging Lombardi warranty support.
Below are the examples of code snippets of Spring controller which calls a SOAP Webservice (Lombardi API exposed as SOAP Webservice) and exposes the method as RESTful service which would emit XML or JSON.
A sample Spring servlet configuration for REST based services.
This approach might sound little bit awkward for having a RESTful service invoking a SOAP Webservice, however this methodology effectively decouples the Lombardi API calls from generating RESTful services.
This post outlines a methodology to expose RESTful services for some of the common tasks of Lombardi like, process initiation and adding comments to a process instance taking them as examples. Any API call from Lombardi which needs to be exposed as REST should be exposed as a SOAP Webservice, since Lombardi does not provide the capability to directly expose the API calls as RESTful service. If a BPD needs to be invoked, a general system service needs to be exposed as SOAP Webservice. The actual instantiation of the process should happen with the general system service.
For example, a BPD can be invoked using the following command, embedded in a Server script activity of general system service.
// to start a process
var processInstance = tw.system.startProcessByName(processName, InputParameterMap);
// to add comment to a process instance
var processInstance = tw.system.findProcessInstanceByID(processId);
processInstance.addComment("Hello!");
Lombardi provides a Webservice implementation component which can be used to expose general system service as SOAP Webservices. A stand alone service layer built on Spring3 framework would consume the SOAP Webservices and convert them to RESTful services using Spring3 REST support. Stand alone services can decouple the transformation logic from Lombardi. Also, deploying them as stand alone would mean that Lombardi product code is not tinkered and hence not endangering future product patches and also this helps fully leveraging Lombardi warranty support.
Below are the examples of code snippets of Spring controller which calls a SOAP Webservice (Lombardi API exposed as SOAP Webservice) and exposes the method as RESTful service which would emit XML or JSON.
// controller method to invoke a process
@RequestMapping(method = RequestMethod.GET, value = "/startCalcGDPProcess",
headers = "Accept=application/xml, application/json")
public @ResponseBody
String startCalcGDP() throws Exception {
RunGDPPortTypeProxy proxy = new RunGDPPortTypeProxy();
String pi = proxy.startGDP();
return pi;
}
// controller method to add a comment to a process instance
@RequestMapping(method = RequestMethod.POST, value = "/addCommentToGDP",
headers = "Accept=application/xml, application/json")
public ModelAndView addCommentToGDPProcess(@RequestBody String body) {
Source source = new StreamSource(new StringReader(body));
ProcessInstance e = (ProcessInstance) jaxb2Mashaller.unmarshal(source);
addCommentToGDPProcess.add(e);
return new ModelAndView(XML_VIEW_NAME, "object", e);
}
A sample Spring servlet configuration for REST based services.
<!-- To enable @RequestMapping process on type level and method level -->
<bean
class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping" />
<bean
class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<property name="messageConverters">
<list>
<ref bean="marshallingConverter" />
<ref bean="atomConverter" />
<ref bean="jsonConverter" />
</list>
</property>
</bean>
<!-- Client -->
<bean id="restTemplate" class="org.springframework.web.client.RestTemplate">
<property name="messageConverters">
<list>
<ref bean="marshallingConverter" />
<ref bean="atomConverter" />
<ref bean="jsonConverter" />
</list>
</property>
</bean>
<bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
<property name="classesToBeBound">
<list>
<value>dw.spring3.rest.bean.ProcessInstance</value>
</list>
</property>
</bean>
<bean id="lombardiRestAPI"
class="org.springframework.web.servlet.view.xml.MarshallingView">
<constructor-arg ref="jaxbMarshaller" />
</bean>
<bean
class="org.springframework.web.servlet.view.ContentNegotiatingViewResolver">
<property name="mediaTypes">
<map>
<entry key="xml" value="application/xml" />
<entry key="html" value="text/html" />
</map>
</property>
<property name="viewResolvers">
<list>
<bean class="org.springframework.web.servlet.view.BeanNameViewResolver" />
<bean id="viewResolver"
class="org.springframework.web.servlet.view.UrlBasedViewResolver">
<property name="viewClass"
value="org.springframework.web.servlet.view.JstlView" />
<property name="prefix" value="/WEB-INF/jsp/" />
<property name="suffix" value=".jsp" />
</bean>
</list>
</property>
</bean>
<bean id="marshallingConverter"
class="org.springframework.http.converter.xml.MarshallingHttpMessageConverter">
<constructor-arg ref="jaxbMarshaller" />
<property name="supportedMediaTypes" value="application/xml" />
</bean>
<bean id="atomConverter"
class="org.springframework.http.converter.feed.AtomFeedHttpMessageConverter">
<property name="supportedMediaTypes" value="application/atom+xml" />
</bean>
<bean id="jsonConverter"
class="org.springframework.http.converter.json.MappingJacksonHttpMessageConverter">
<property name="supportedMediaTypes" value="application/json" />
</bean>
This approach might sound little bit awkward for having a RESTful service invoking a SOAP Webservice, however this methodology effectively decouples the Lombardi API calls from generating RESTful services.
Subscribe to:
Posts (Atom)