BPM System Architecture

The BPM System Architecture is the workhorse of an organization’s BPM vision to fulfill its strategic BPM needs and objectives. This article aims to provide insights into the BPM System Architecture and how it should be devised and evolved to fully support a seamless, scalable and dynamic platform for process implementation. Most organizations in today’s world would purchase a packaged BPM suite which has abundant features out of the box that support’s the entire BPM lifecycle, rather than an in-house built custom BPM platform. The need for a BPM System Architecture is even more imperative for a packaged BPM suite, since the packaged products must be judiciously aligned within the scope of the enterprise architecture (EA) of an organization. A home grown BPM platform would have a lesser challenge in terms of fitment into enterprise architecture since it would have evolved within the EA scope right from the beginning.

 

The packaged BPM suites would seem to be more exciting and successful when implemented on a small scale like proof of concepts. However when these suites are stretched for a full fledged implementation, its short comings may lead to a colossal failure.  A BPM suite can handle 10 process implementations with ease, but when it comes to hundreds and thousands of processes, a trivial mistake or a minuscule short coming in the BPM suite would proportionately increase the rate of failure.

It is imperative that a BPM System Architecture is built so that a packaged BPM suite can be seamlessly integrated into an Enterprise Architecture. This article will provide insights and glimpses of BPM reference architecture.

Process Data versus Business Data

The process data is the one which supplements the process, but not crucial for the actual completion of the process. KPI, SLA and metrics are few examples of process data. A KPI or SLA might be based on a business data but as such it may not be required for process execution. Business data is the actual driver of the process and would take it to completion. In an Insurance industry the business data would look like, an insurance id, a claim id, insurance expiry date etc. Business data is critical for process execution and the process directly depends on the business data for the process to complete.

The BPM System Architecture should clearly differentiate between the process data and the business data. The business data would most probably reside in an independent repository (database) and would be accessed via an abstraction layer like SOA. The process data would be derived from the business data by the BPM suite and would reside in the BPM suite database. 

The BPM System Architecture must clearly define the business data identifiers that a process should be aware of. For instance, an insurance process must be only aware of insurance id. Anything beyond that would be a nightmare to sync the process and business data. If the process data is defined with insurance id and insurance expiry date, then any changes to the insurance expiry date in the external systems must be synced with the process data, which would be a nightmare if the number of business attributes in the process data grows. Instead the process should only be aware of the insurance id and when the process requires the insurance expiry date, it needs to be pulled up from the external system using the insurance id. In other words, the best practice to define process business attributes is to define it with unique business identifiers. An insurance id would be a unique identifier and all relevant details pertaining to the insurance policy can be pulled from the insurance id.

Process data like KPI and SLA can be derived again from business data. If an SLA needs to be defined on the insurance expiry date from the process, with the insurance id, the insurance expiry date required for SLA computation can be pulled from the external system and SLA can be derived. The SLA data for the insurance expiry date can reside in the process repository.

The BPM System Architecture should define a framework (like an integration component) which would act like a nerve centre to pull business date from the process with the help of unique business key attributes.

Long running process

Failure of process instance cannot be always avoided. But what would be unavoidable, is of course the graceful recovery of the process instance. A process instance can fail for variety of reasons ranging from unavailability of system resources, data issues, network outages and technical glitches. The more prominent of these are the data issues.

A process instance in a telecom infrastructure provider would typically execute for months. If the telecom provider has a process implementation, to lay trans-Atlantic cables from continental United States to Europe, such a process instance can get executed for multiple months. In due course of time, if the process instance fails, the situation may not demand to create the process instance altogether from the beginning of the process and it is more likely it may not be possible at all, since the business data has undergone changes in due course of the process traversal. In such scenarios the process instance no matter what has to be gracefully recovered.

Let us assume that for the process described above, the process instance gets stuck at an activity where the process instance expects the length of the cable between pier A and pier B to be of certain length, whereas the business data for cable length is out of that range for what the process instance expects.  For the process instance to execute, the business data for cable length needs to be updated in the source record. The BPM System Architecture should have a mechanism to refresh the process with latest business data after the update and should recover gracefully.

Also, all long running processes which are vulnerable to failures and are critical to business wherein the re-creation of the instance is not possible, the actual business process must be supplemented with a background recovery process. The background recovery process must be designed in such a way that it must be capable enough to replicate any process instance and can hold the same state as the failed process instance before the failure. The input to the recovery process must be the business key attribute and the activity of the process from where the process instance has to start. This input activity would be the same as the failed process instance step. In order to design such a recovery process, it is absolutely essential to separate the business data and process data. Without this it would be very difficult to create a process instance from a specific activity within the process, since the newly created process instance should reflect the exact state of the failed process instance. This is only possible by associating the business data of the failed process instance to the newly created instance.

BPM Portal

A large scale BPM implementation would require a unified user interface for the business user community. The out of the box portal packaged in the BPM suite may not be flexible enough for extreme customizations. Also, process instance search is a ubiquitous functionality which would act as a first interfacing point for the user.

The BPM System Architecture is responsible for a process instance search framework. The challenge to build such a framework would be to integrate both the process and the business data in one single view. A process instance can be searched with a business attribute as a parameter and also the search can be performed with process data such as SLA.

One possible technical solution is to combine the process data and business data together as Database view, by which expensive database joins can be avoided between multiple tables of the business data schema and process data schema.

BPM Exception Handling


Exception handling is one of the prime aspects of software design which attributes to the robustness of the application. Exception handling gains more significance in BPM space, since exceptions had to be dealt both at a system level and at a business level.

The significance of exception handling at a process modeling step cannot be ignored. There is a need to handle ACID transactions in BPM, with a business solution and it has to be captured in the model with business semantics. For instance, a classic example of flight, hotel and rental car booking when deemed as a single atomic transaction cannot always be dealt at a system level. If flight booking and hotel booking are two different applications in two different domains (if flight carrier and hotel are different companies), then system transactions cannot be propagated and even if it seems to be possible, system level locks for transaction may not be possible to be obtained since the whole transaction may exceed beyond days to complete. The ideal way to deal with this situation is to have three distinct transactions as business process and even if one fails the other transactions must be canceled using a business process. If flight booking has been confirmed and if hotel booking transaction fails, then flight booking should also be canceled, wherein flight cancellation must be modeled as a separate business process.


It becomes imperative for a BPM solution to handle system level exceptions gracefully. Since process instances can span across business days, in case of system exceptions the process state has to be persisted and should be able to recover from the state where it was left. This may not be possible always for each and every process instance. Sometimes there is a good chance that a process instance may get into an irrecoverable state. In such scenarios, it may not be desirable for the users to create a new process instance altogether. The alternatives for irrecoverable instances due to system exceptions are few, and one probable solution could be to create a new process instance and associate the
application process data with the new instance, and programmatically navigate process the instance to the workflow step where it failed.

Another alternative is to create an exception business process. Any system level exception occurring in any of the application business process would trigger the exception business process, which would notify the user about the exception. Also exception process can notify a repair/admin team about the exception. Having a repair queue of exceptions can help the repair/admin team to get first hand information about the exception without the need for notification from the business users about what had happened. Also, irrecoverable process instances can be dealt internally within the IT team, without burdening the business users.

Process Versioning

A process undergoes lot of changes in a BPM lifecycle with multiple iterative implementations. The BPM System Architecture must be potent enough to handle multiple versions of process instances. Most of the modern BPM suites come pre-packaged with utilities to migrate the process instances from an obsolete version to higher versions. However, a process instance in an older version may not be always possible to migrate to a higher version, due to external system dependencies, when the external system itself would have undergone a version change along with the process. This is again in-line with the process data versus business data debate. The battle lines between process data and business data have to be carefully drawn in such way that any data on external system should not be a part of process data. A better way to hold reference to external application data is to hold reference to the unique business key attribute of the external system.

Conclusion

These are the some of the many items that must be considered for a BPM System Architecture. If the segregation between process data and application data can be achieved then having the BPM System Architecture would bring the additional flexibility in terms of solution design.

IBM Business Process Manager - Mastering Pitfalls

IBM Business Process Manager erstwhile Teamworks Lombardi is one of the top selling BPM suites in the market. IBM Business Process Manager has some excellent features and some innovative concepts which enhance BPM development. However it is imperative for an organization to know about its key features and some of the disadvantages it brings along with its features. These short comings can be easily overcome and one easiest way to do that is to become aware of the short coming itself.

Branching and Merging in IBM Business Process Manager

IBM Business Process Manager has a shared development model. Unlike the traditional software development, the IBM Business Process Manager code base is maintained in a Process Center environment. A Process Center environment is not a source control by itself; however the IBM Business Process Manager artifacts are stored in the Process Center. The code base for a process application is maintained in a Process Center.

The run-time instance (executable) of the process application (code base) is known as Snapshot. A Snapshot can be deployed into any of the IBM Business Process Manager run-time process servers. The code base for a process application cannot be branched nor merged. Although, branching can be done by creating a Workspace in IBM Business Process Manager, there is no way to merge the branches.

This important aspect of this short coming in IBM Business Process Manager is that the Agile projects have to be planned meticulously. If a project plan mandates parallel development for a same process application, then there is no way in IBM Business Process Manager to merge the parallel branches into a single trunk.

Process instance data cannot be modified

Long running process instances are vulnerable to failures. Some of the failed instances may be irrecoverable due to data issues. For example, a particular variable in a business process needs to be set to a certain value to take a certain path in the business process flow. The business process data due to system issues or code issues might have got a wrong set of data and it would become necessary to read and update the business process data when the process is active. IBM Business Process Manager has an API to read the data, unfortunately does not have a good way to update the data, since the process data is stored as BLOB.

To circumvent this problem, the usage of business process data has to be minimized. Only business key attributes or in layman terms, primary key attributes in the application data needs to be mapped to the business process data. By mapping the primary key data as business process data, a loose coupling between the business process and the application data is achieved, thereby the application data would have more freedom to change without the necessity of keeping the business process data in sync.  There is a huge debate on what needs to map to business process data and what has to go into the application data, which itself becomes a separate topic to explore and it is well outside the context of this article.

IBM Business Process Manager stores most of the process related data as BLOB's

A typical software application’s life cycle regions include Development (DEV), System Integration Test (SIT), User Acceptance Environment (UAT) and Production (PROD). After every release to production, a good practice is to mirror the entire PROD data into different regions. This is no exception for BPM applications, and IBM Business Process Manager presents itself another challenge to implement this setup. Since process data in IBM Business Process Manager are stored as BLOB’s, there is a challenge for DBA’s to replicate the BLOB’s in lower region databases.

Also if the application data model and the IBM Business Process Manager data model are two different schemas, then there is an additional challenge in maintenance of data states after the production refresh. It becomes imperative for DBA’s to understand the complete data model of the Process Manager to synchronize the tables required to reflect the state and to ignore the tables which represents the run-time dynamics.

Also refer, BPM Data Model Architecture.

Workspace cannot be deleted

The need for Workspace arises when the code needs to be branched out of a particular Snapshot. For example, if there is a defect in a production release and the current Development release has progressed into the next version of Snapshot, a branch of production Snapshot is required and Workspace comes for rescue. Any version of the Snapshot can be branched out to a Workspace, however it cannot be merged to another Workspace as discussed already. Workspace created cannot be deleted from Process Center leaving a scar for potential misuse. Multiple Workspace would lead to confusion and there is always a possibility of developer choosing a wrong Workspace.

Creation of Workspace cannot be avoided, but with proper naming convention and standards, a potential misuse or accidental usage of a wrong Workspace for a wrong version can be avoided.

Process applications in Process Center cannot be deleted

Process applications in Process Center cannot be purged. This is mostly a sanity related aspect of maintaining the code base in the Process Center. Over a period of time the Process applications tend to grow and there would be a need to purge some of the obsolete Process Applications which would be no longer required. Unfortunately there is no way in IBM Business Process Manager, a Process application can be purged. A better way to avoid this situation is to have tighter governance and strict privileges within the developer community on who can create a Process application.