IT Directions

Explanation soon … watch this space!

In my last post, I quoted an article on this site by Jon Pyke in which he looked forward to 2 new generations of BPM tools that provide support for knowledge work. The first of these generations integrates Case Management with process technology. The second goes further to support Human Interaction Management (HIM).
Jon is now the Chief Strategy Officer of Cordys, who along with other key BPM players are now bringing to market the first of these developments. What is involved in the second step? And why is it necessary?
As regards what is involved, I have tried to illustrate in this blog series some features of HIM that go beyond the capabilities of current process and case management technology, even offerings that appear to be at the leading edge. HIM notation directly addresses five aspects of management with which organizations are currently struggling:

1. Teams
2. Communication
3. Knowledge
4. Time
5. Planning

Further, the current inexorable trends towards outsourcing, partnering and sub-contracting as the fundamental means of doing business in a globalized economy mean that in each of the 5 aspects of management above it is critical to support decentralized, cross-boundary processes where there is not necessarily a single process “owner”.
To meet these requirements, a new paradigm for process description is required - one that is based not on state machines in which the process is a clockwork mechanism that moves from stage to stage, controlled centrally by a single engine, but on object models where a process is a set of objects in different domains, whose interaction and synchronization is controlled collaboratively by agents acting on behalf of each player. This new paradigm is what HIM notation, and the underpinning HIM semantics, provide.
As regards why this is necessary, one only needs to read a newspaper. The global economy is in meltdown, for which the deep reason is simply the advent of the Web, with the consequent rise of “Asia, Automation and Abundance”. Computing has brought a huge sea change to our world, a change to which we are too close to see the true scale. The only way to survive such change is to adapt, which means taking dramatic steps early on. For once, only the early adopters will survive! Those that wait until new ideas have been fully tested will be out of business before they get a chance to put them into practice for themselves.

Scientists are starting to realize that information processing may lie at the heart of many different aspects of nature (see, for example, Charles Seife’s “Decoding the Universe”). Similarly, mathematicians are starting to understand how the notion of computation explains many of the dramatic results of the last hundred years (see, for example, Gregory Chaitin’s “Metamaths - the Quest for Omega”).
In business, an equivalent conceptual breakthrough is taking place. Pareto’s law tells us that the 20% of “exceptional cases” account for 80% of the costs - but it does not tell us why. To discover why, and deal with it, one must appreciate that “exceptional cases” are not exceptional at all! They are the norm, since they occur all the time - further, they are what truly test your business practices.
To deal with the “long tail” - i.e., to operate efficiently and effectively in a globalized economy - one must abandon the hopeful notion that business processes can be defined once then run thousands of times with only minor change. One must create an operational environment in which change is not only possible, but structured, encouraged and aligned with strategic objectives.
This means taking a much richer view of “process” - a view in which people, communication channels, knowledge, time and plans are all managed along with the activities that are more easily visible - across multiple domains that include not only you and all your trading partners but also your customers. Bottom-up empowerment is not enough. Top-down control is not enough. You need an enterprise management framework that supports both, at the same time, using the same approach.
Change is never easy, especially when the luxury of taking your time is not available. However, as you can see by watching lynchpins of the financial world topple one by one, right now there really is little choice. Yes, you need to improve the 80% of processes that are mechanistic, using current mainstream BPM techniques, but this only brings you up to the level of your competitors. To stay ahead, and stay in the game, you need to improve the other 20% of processes that cannot be fully planned in advance - and do it at enterprise level.
Find out more about HIM here.

In the 3 years since publication of my book “Human Interactions - the Heart and Soul of Business Process Management”, reactions from BPM specialists have been mixed.
One frequent response is: “at last someone is saying what I’ve always thought”. In other words, the mixture of consultancy observations and insights from varied disciplines that is Human Interaction Management (HIM) seems like simple common sense.
Another frequent response is “that can’t be right”. In other words, please don’t let this be right - because if it is, we’ve got to start again, almost from scratch. Both consultants and software vendors are going to have to make dramatic changes to the solutions they are offering.
Current BPM solutions (both method and theory) relegate human collaboration to the fringes of organizational life, treating as an backwater of unmanageable “ad-hoc” work where “here be dragons”. This is not good enough, is it. Organizations need knowledge workers to be:

1. More efficient - work faster and leave tidy information behind. Basex estimate that 28% of knowledge worker time is lost to poor control of interactions. So we need better teams, better communication channels, better generation of knowledge from information, better time management, and better planning tools (these are the 5 HIM principles, by the way). How are current BPM and case management solutions going to give us any of these? To the worker, “human-centric” software offerings are still just task lists, with the added complication that they are now expected to create their own tasks.
2. More effective - adapt their work to changing objectives. The key word is “changing”. It is not enough to allow people to plan their own activities in ad-hoc fashion - this is just a gateway to chaos, in which people go off at tangents the whole time, leaving behind a sea of unstructured audit trails. Human work needs to be part of a management framework geared around driving top-down change (i.e., from current strategy) while empowering bottom-up decision-making (i.e., allowing people to negotiate next steps in a structured way as they go through a process).

In this blog series, I am relating these needs specifically to BPMN, by showing how BPMN notation cannot express constructs that are critical to the efficient and effective management of human work - while HIM notation can do so easily. Here is another example.
In the HIM diagrams featured in previous posts in this series, you will notice that each Role (yellow rectangle) has the word INSTANCE underneath its name. This means that the Role is active in the process - it represents a real, working participant. Some Roles, however, are used differently. Here is another version of the HIM solution to Respond to RFP from a few posts back, again generated using the free HumanEdj:

Notice that this time the Technical Consultant Role is shown as a TYPE rather than an instance. Also, the Account Manager has a new Activity, to assign a Technical Consultant. What is going on?
The new diagram reflects the fact that, in some cases, the Salesperson will not need technical consultancy - they will be familiar enough with the solution to be offered to the client that they can prepare the proposal unaided. In the new version, the Salesperson and Account Manager can discuss this via the Initiate Submit Proposal Interaction - and depending on the outcome of that discussion, the Account Manager may or may not create an “instance” of the Technical Consultant Role “type”.
How would this be depicted in BPMN? You would have to create a swimlane for the Technical Consultancy work, and make all its activities optional in the process. In other words, work has been assigned but may never happen. Further, there is no explicit depiction of how the work will be assigned - under what circumstances, when, who by, to whom, the sort of skills and experience required, and so on. Critical aspects of human resource planning are omitted from the process - of necessity, since there is nowhere to put them.
TAKE AWAY
To remedy process description deficiencies such as those described in this blog series, software vendors typically advocate the use of business rules in conjunction with process execution software based on BPMN and/or XPDL.
While one of the 6 key HIMS object types is a Rule, which can be applied in many ways, Rules alone are not the answer, since in large quantities they are unmaintainable. For routine work, we are moving toward an agent-based approach in which rules are key - when you run a process a million times, the effort of maintaining the rules is worth it. But for human-driven work, in which every process instance is different, there is no value-add in burdening it with huge numbers of rules. Not only does it constrain the work artificially (just ask a doctor whether they would let a system diagnose for them), but there is zero motivation for the people involved to help maintain the rules, since they are only putting themselves out of a job - so people will work around the rules and let them degrade. I’ve seen this in many organizations over the years.
The key issue is that human-driven work is fundamentally different to routine work, for a variety of reasons ranging from socio-economic issues such as globalization (in particular, the availability of very cheap workers) to psychological issues such as team building (as Handy predicted 20 years ago, corporations are becoming virtual). Organizations ignore this at their peril! As Jon Pyke, Chair of the Workflow Management Coalition, wrote in an article on this site in 2006:

Process-based technology that understands the needs of people and supports the inherent “spontaneity” of the human mind is the next logical step, and we might be tempted to name this potential paradigm shift “Knowledge Intensive Business Processes.”

KIBPM falls into two main categories, which will probably merge over time, and the vendor that recognizes that potential will steal a march on the others. At the simplest level we have case management, and secondly, we have Human Interaction Management. I doubt there are many BPM products on the market today which will be able to meet this seismic shift in requirements - certainly those that rely on BPEL and SOA won’t; what’s more, any that have been in the market for longer than five years will need radical surgery to meet the coming challenge.

Why Workflow Sucks

In other words, radical new ideas and accompanying new technology are required, that go way beyond case management. So why have the major software vendors stuck with the kind of workflow and state-based case management solutions that have been available for years? Well, all suite vendors have a huge code base to maintain, and it’s easier to tinker with what they’ve got than to develop new code from scratch. But this isn’t serving the market, or ultimately, themselves.
To thrive in the next 50 years, we need new insights, new practices and (ultimately) new forms of software.

In the last entry to this blog, I challenged readers to depict a very simple process in BPMN - development of a branding package (colours, fonts, logo, artwork, etc), in which:

1. The account manager, designer and artist throw ideas back and forward for review and debate (concepts, images, documents, etc), usually but not always by email;
2. The same person acts as designer and artist at the start, but has the option of handing over the artwork creation to others if their workload becomes too heavy;
3. It is possible to add more artists as needed during the life of the process, all of whom participate in the same communications.

So far no-one has responded to the challenge. This is not surprising, since it is not possible to show any of this in BPMN.
With regard to “throwing ideas back and forward”, message flow in BPMN is limited to a single, one-way send from one pool to another. The send can be repeated if the appropriate looping constructs are used, but it is very hard to depict message flow between more than 2 parties, and any attempt to reproduce the flexible manner in which people exchange messages is doomed to failure. Mainstream BPMS software currently deals with this limitation by claiming that message exchange between colleagues is not really part of a work process - rather, they claim, it is an ad-hoc activity on which no structure can (or needs to) be placed. In other words, what is perhaps the most fundamental tool of knowledge work is relegated to floating around under the organizational radar, in an unmanageable backwater.
With regard to “handing over work”, BPMN has no concept of who does the work. It simply shows what needs to be done, by someone or something. Hence it is impossible to include in a process diagram any indication of the people, or sorts of people, involved. Again, fundamental aspects of knowledge work (acceptance and delegation of responsibility, capabilities, personal characteristics, and so on) are quite literally out of the picture.
Finally, with regard to “adding more” of a certain role, there is no means to achieve this in a BPMN process. It is hard even to imagine how the notation could be extended to support the notion, since the flowchart principles on which BPMN are based do not support such a concept. Yet human resource planning is fundamental to process management, as it is to all management. How is process software supposed to support human resource planning using a notation in which it is not possible to depict, let alone adjust, the resource levels assigned to a work package?
By contrast, here is a simple diagram using Human Interaction Management (HIM) notation that shows all the above constraints:

The horizontal, dashed lines are Interactions - they show message exchange between the 3 parties to the process. Messages can flow in any direction, repeatedly as necessary.
The yellow rectangles are Roles in the process, and the ovals at the top are Users of those Roles. At the start, Dee Zeiner is playing both the Designer and the Artist Roles. However, the diagram includes an extra User, Richard Tist, to whom the Artist Role can be reassigned during the process if Dee takes on other work and has to delegate.
Finally, should it become necessary to add another Artist as extra resource, this can be done in a Human Interaction Management System (HIMS) simply by right-clicking on the Artist Role and asking for another one. The HIMS will prompt for the details of the person to be assigned the work and then duplicate the Role, automatically assigning the new Role to the new person and including the new Role in the same Interactions as the current Role.
TAKE AWAY
There are many other aspects of HIM notation that make it more suitable for collaborative, adaptive human work than BPMN. I will discuss these aspects in future posts in this series. While waiting for the next post in this series, you might like to consider the following puzzle.
The Artist in the process above must create a draft set of colours before starting to create fonts. However, once they have created a draft set of colours, [the semantics of a HIM diagram mean that] they can repeat and interleave the colour and font creation activities as they wish. Similarly, they must create a draft set of fonts before starting to create a logo. However, once they have created a draft set of fonts they can repeat and interleave the colour, font and logo creation activities as they wish. Simple common sense! Yet can you depict this incredibly simple and commonplace pattern of work in BPMN?
You will find that it is necessary to jump through some very artificial hoops in order to draw such a pattern in BPMN - just try to show how the Artist can go directly from colour creation to logo creation only once they have created fonts. Unlike people, a flowchart has no memory. A HIMS does, however. If you are interested to try out the next generation of workplace software, the free HumanEdj (via which the above diagram was generated) is available online for download.

This week I spoke at BPM Europe 2008 in London. Talking to people there, and listening to the other presentations, it seems to me that BPM has reached, at last, an inevitable watershed. Home truths that people have been trying to ignore for years are finally and painfully apparent to everyone.
As you will read on this Web site and others, BPM software is now mainstream - the market has consolidated and all major suite/platform vendors offer some kind of BPM solution. Further, the cultural practices that practitioners consider actually to be BPM (software deployment being advisable only after adopting a process-oriented approach to management) have become reasonably consistent.
Unfortunately, however, the management practices are not keeping up with needs - and as a consequence, the software market is shrinking.
To see this, consider various analyst reports of the last few years. Although all analyst firms and hence the reports are funded mainly by software vendors, the reports themselves tell a grim story. For instance, Gartner in 2004 reported that you could expect ROI of “at least 15%” from BPM. But you can get 10% from a high street savings account! Why go to the expense, trouble, and risk of a BPM project to get 15%?
Around the same time, Butler Group reported that you were unlikely to get any ROI from BPM, since most corporate processes were already fairly efficient. This makes sense, since the management practices most usually associated with BPM date from the 1950s - Lean, Six Sigma, et al are essentially refinements of TQM and Hoshin Kanri as developed by Juran, Deming and others during the reconstruction of post-war Japan.
That was 2004. Now, in 2008, a highly publicised report by BEA, “State of the BPM Market in 2008″, tacitly confirms Butler’s viewpoint.
For a start, the report makes no quantitative estimate of ROI from a BPM project at all. In a document whose only purpose can be to promote adoption of BPM software, this is a telling omission.
Further, the BEA report also includes estimates of the size of the BPM software market from various sources. On average, the size is believed to be around $1bn, predicted to rise to several times that in a few years. This is exactly the same prediction that has been given year on year for a long time now - every year the total size is still around $1bn, predicted hopefully to rise “soon”. When you take into account the fact that every year more and more of this sum is residual revenue - support and maintenance payments on existing installations - the market as a whole is clearly decreasing in size.
The reason that BPM software appears to be doing OK is that the market has consolidated dramatically, from hundreds of vendors to a handful of major players. With so many less vendors, for now at least there is enough to go around.
On a less quantitative note, any BPM practitioner will tell you that enterprise BPM - the Third Wave predicted by Smith and Fingar - does not exist. Many of the speakers at BPM Europe 2008 made this point, including Howard Smith himself. BPM is currently deployed, whether as management practice alone or also using software to automate work, as a point solution - to fix a specific low-level process such as invoicing, claims processing, or membership approval.
What has all this to do with BPMN?
TAKE AWAY
BPMN is a symptom of a deep problem with BPM, which is that its management practices completely ignore leading management thinking since 1960. The great figures of the last half century - Drucker, Handy, Senge, and others - all pointed out that organizations are systems, in which feedback loops cross boundaries. For the organization to operate effectively, people must collaborate in order to make real time changes to running processes. This requires both visibility of what is happening at many levels and empowerment to implement such changes.
Yet mainstream BPM practice, as it currently stands, is this: describing an organization as a hierarchical tree of processes, starting with separate value streams/chains, and descending with inexorable rigour to routine, repetitive flowcharts more suited to enactment by CPUs than to enactment by humans. Such an approach is well suited to the factories with which Deming and Juran were concerned in 1950, but not at all to a modern, globalized organization in which a huge part of the work is collaborative human activity.
In this blog series, I am focusing on a specific aspect of mainstream BPM practice - the use of flowcharts to represent processes, via the de facto standard notation BPMN. To end this post, consider how you might use BPMN to depict the following process - development of a branding package (colours, fonts, logo, artwork, etc), in which:

1. The account manager, designer and artist throw ideas back and forward for review and debate (concepts, images, documents, etc), usually but not always by email;
2. The same person acts as designer and artist at the start, but has the option of handing over the artwork creation to others if their workload becomes too heavy;
3. It is possible to add more artists as needed during the life of the process, all of whom participate in the same communications.

Next time, I will show how these requirements, which are typical of human collaborative activity, cannot be represented in BPMN in any way that is useful. I will also show how very simple it is to represent the process in HIM notation. In doing so, I will (as promised) explain more about the notation demonstrated during my last post.
In the meantime, you may like to find out more about HIM, and the associated software, a HIMS (of which the reference implementation is free). It is the only way you will ever realize the Third Wave.

A commenter on the last post to this blog wrote:
“I agree that BPMN is not targeted to the behaviour of people to people and people to system interaction. Using it for such a purpose is akin to using a chisel to open a paint tin.”
This sums up very neatly a typical attitude to what I call human-driven processes. Like paint swirling around in a tin, collaborative human work is seen as lacking any structure. Human interactions appear to be so complex, so ad-hoc, that it is impossible to apply precise descriptive techniques to what is going on.
The result of this viewpoint is that collaborative processes are brushed under the carpet by business analysts, process designers, solution architects, and other people in a position to help organizations work both more efficiently and more effectively. The latest crop of “human-centric” BPM tools, for example, treat knowledge work as an inconvenient adjunct to routine processes - a sort of exceptional case, in which the polished simplicity of a routine process is unfortunately disrupted by the human desire to think, communicate and achieve. The manner in which editing documents, sending emails, and other mainstays of business life is treated by such BPM products relegates them to the fringes of management awareness - a short-sighted strategy, to say the least, since it is this kind of work that delivers most value. Treating such activities as a temporary step outside the “real” process certainly gives little opportunity for the 59%-80% of the US workforce who are knowledge workers to contribute to the sucess of their organization.
Regular readers of this blog will know that the principles and patterns of Human Interaction Management exist to remedy the situation. Using the simple notation of HIM, in conjunction with a small number of standard patterns, it is possible to understand and improve any type of collaborative work. However, these principles and patterns are relatively new (they were documented for the first time in my 2005 book), so in this blog series I am attempting to illustrate them by example.
The particular example used is a simplified process for Response to Request For Proposal. For comparison, in my last post I drew the process in BPMN and discussed the weaknesses of the representation. In this post I will illustrate how the process can be represented using HIM notation.

You will see immediately that the diagram looks quite different from BPMN. The similarities that exist are in fact superficial - don’t be misled by them into thinking that the notational constructs are equivalent. Here are some key aspects of the diagram:
Roles
Instead of lanes/pools as in a BPMN diagram, with documents/data floating about in thin air, each player in a HIM process has a Role - the vertical rectangles with yellow background. A Role is more than a collection of actions - it is a mini-workspace, that provides you with a lot more than “things to do in a specific order”. For a start, each Role has its own goals and responsibilities - which by agreeing to play a Role, you commit to meeting. To help you meet the goals and responsibilities, each Role has its own private documents/data that you can update and share with colleagues when appropriate. A Role also contains business rules that help you decide when to do what, and that help ensure that your work stays in line with higher-level policies and regulations.
Interactions
Instead of messages from one party to another as in a BPMN diagram, a HIM process has Interactions between Roles - the purple, mainly horizontal lines. A HIM Interaction is a purposeful communication channel - a means by which messages can be exchanged, repeatedly and in any direction, between any number of parties. Think of Interactions as “email plus process context”, although any form of messaging can be used to implement an Interaction.
Collaborative Transactions
Instead of sub-processes as in a BPMN diagram, a HIM process has Collaborative Transactions - stages or phases of a process, as highlighted in the marked-up version of the diagram below. Collaborative Transactions start with an Interaction between all parties (to establish the purpose of the ensuing work), contain various actions for each party, and conclude with another Interaction (to agree that the work is completed and decide on next steps). Collaborative Transactions can be nested, as shown here, without introducing a hierarchical structure foreign to human interactions - in reality, people can talk to one another at any time, whatever part of a process they are supposed to be carrying out.

There is a lot more to say about this diagram, and I will return to it in future posts.
TAKE AWAY
Collaborative human work is a quite different thing from the routine, mostly automated work handled by mainstream BPM techniques and tools. Trying to capture human work using BPMN is not so much like using a chisel to open a paint tin as it is like using a chisel to play the violin - the end result will be frustration with your chisel, serious damage to your violin, and a horrible noise.
The diagram above was drawn using HumanEdj, the reference implementation of a Human Interaction Management System - a modelling and execution environment for human work. The diagram not only shows the work, but can be used by all parties to carry it out. If this blog post strikes a chord with you, why not try out HumanEdj - design some collaborative processes for yourself, and see how easy it is to bring order to the apparent chaos. It may not be possible to eliminate real-world complexity, but it is certainly possible to bring it under control. You just need new techniques and new tools.
NOTE: The diagram shown here was drawn using the latest version of HumanEdj, which will be on public release later this week. If you are using an earlier version, your diagrams will look different.