IT Directions

In my last post I promised to discuss mashups - in particular, how there is more than one type of mashup.
Let’s start at the beginning - what is a mashup?
To help answer this, here are some pictures taken from publicly available presentations by the CTOs of 2 firms supplying software commonly used to create mashups, Coach Wei of NexaWeb and John Crupi of JackBe.
Coach Wei gives an overview of where a mashup sits with respect to SOA and Web 2.0, terming it “Enterprise Web 2.0″:

Coach’s principle is that a mashup is how you leverage back-end services, namely by applying a rich Web 2.0 interface in order to expose them for business use. John Crupi provides 2 pictures that explain this how this differs from previous approaches, which typically use portal technology. Here we see what John calls “Today’s Content-Driven Web”:

Here, by contrast, we see what John calls “Tomorrow’s Data-Driven Web”:

Essentially, a mashup moves the integration of disparate forms of information behind the firewall. By doing so, it becomes possible to add additional controls, not just for security purposes but also in order to connect data more usefully - for example, by showing on a single chart information that previously was displayed in separate “portlets” (i.e., separate areas of a Web page).
The growth of NexaWeb and JackBe testifies to the utility of mashup technology. However, the tools available today are just the start of the mashup trend. Why?
Useful as mashups are in their current form, software applications built in this way are essentially competing with process support systems as service consumers. Like mashups, both the BPMS (for routine, semi-automated work) and the HIMS (for human collaborative work) use services to implement business processes.
In practice, a current mashup can be thought of as equivalent to a step in a business process - a node in a BPMN diagram, for instance, or a Task in a HumanEdj Role. In fact, it would be quite possible to design a BPMS or HIMS process that incorporated one or more mashups in this way. However, at this point mashup and process tools are not integrated. For now at least, there is a clear division in the market between mashup creation tool vendors and process support system vendors, and this can only lead to problems.
Most organizations have come to accept that managing work activity better means managing work processes better. Hence they are using process techniques, and in due course most will adopt process execution software. If they are also using mashups, the consequence is that they will then have 2 siloed forms of design activity for “service-oriented business applications”. As a result, features belonging really to processes will creep into mashups, and vice-versa.
In particular, processes of either BPMS or HIMS type tend to use data items to preserve their “state” - i.e., where the work has currently got to. For example:

• A BPMS process representing a product assembly might contain data items corresponding to quality check results, and use these to determine whether or not the product is ready to ship.
• A HIMS process representing a marketing campaign might contain data items corresponding to focus group results, and use these to determine whether or not the collateral is ready for release.

If such data items are hidden inside a mashup application, the process designer has two choices. They can duplicate the data items in the process definition - a redundancy potentially leading to error, even if the data is extracted originally from the mashup. Alternatively, they can try to avoid redundancy problems by insisting that the data is held only in the process - an articial workaround that restricts the usefulness of the mashup.
TAKE AWAY
Like the Web, mashups will evolve. The next stage is surely to provide a means of integrating Rich Internet Applications with BPMS and/or HIMS processes, to permit safe sharing of data between applications and processes.
Such integration is likely to start with the HIMS rather than with the BPMS, since the processes that they deal with are very different. Processes for which a BPMS is suited are routine, repetitive and semi- or fully-automated. Human involvement in such processes is limited to key points, and takes the form of low-level data entry and decision-making. This is not the territory of mashups, which are aimed usually at higher-level knowledge work activities. Mashup activities are much more closely related to HIMS processes - innovative, collaborative, adaptive human work. It is HIMS processes that can leverage the rich content of a mashup, and hence it is here that the integration will probably start.
Returning to the theme of this post, there are essentially 2 types of mashup: siloed and process-aware. For now, mashup tools are siloed. However, in due course they will become process-aware via integration with HIMS technology - and once this happens, the combination of HIMS processes and mashup applications will be an extremely powerful way to leverage both Web 2.0 and your legacy infrastructure.

Just back from SOA India 2007, where I gave the keynote.  The conference was a fascinating experience.  In particular, it was interesting to talk to delegates from many different sizes and types of organization who are actually doing SOA!  Not just talking about it

India really is where it is all happening, IT-wise.

And the discussions I had over the week, both with delegates and with other speakers, bear witness to an urgent need for a deeper understanding of SOA governance.  Many people working in the field are technically savvy, and have a good understanding of SOA techniques and tools, but still possess little or no appreciation of governance issues.  I heard some startling things …

In the end, the sole justification for SOA is change.  In particular, SOA should enable you to align business change with system change - if you don’t expect to attempt this, there is little or no point doing SOA.  However, change in an SOA environment is deeply risky - analogous to rewiring an F16 in mid-flight.  Scary stuff, considering that standard software QA techniques such as static analysis and automated testing have yet to mature for systems that are assembled rather than programmed.

Hence, if you’re going to do SOA, you can expect to be doing on-the-fly transformation - and in the absence of standard, systematic control techniques you need to have a very good understanding of how to control change manually.  In other words, you need to define your governance processes, and manage their implementation,  very carefully indeed.  Such processes are not routine, and are only partial candidates for automation - they are, in fact, exactly the kind of processes dealt with by Human Interaction Management (HIM) (collaborative human work processes).

However, no-one I spoke to last week seemed to understand the need for process control in SOA governance, however sophisticated their ideas on SOA in general. Certainly none of the vendors in the governance space have yet incorporated HIM techniques into their tools. The next step for SOA governance tools is to integrate the functions of a Human Interaction Management System (HIMS) such as HumanEdj into an existing, repository-based offering - and the first entrant in this space will have the market to themselves.

TAKE AWAY

I suspect that understanding of the need for process control in SOA governance will only become widespread once there have been a few public disasters - and these cannot be far off.  Soon, early adopters of SOA will attempt to use their brand new system environment in the way promised by its original advocates - namely, to make rapid changes to align IT with new business requirements.  And without effective process controls, reconfiguration of a hugely complex network of interacting services is bound to bring problems.

I see no reason why the impact of such problems should be limited.  Rather, they are quite likely to break some organizations entirely.  It may be that for now, SOA change problems have been covered up - but large-scale operational or financial disaster is not so easy to hide.

So why wait until it happens to you?  Would it not be better to bring some order to the chaos?  In other words, take responsibility for your SOA!  Do not treat SOA governance as if you were going out to dinner with colleagues - each choosing a few items from the menu on a whim.  Rather, treat SOA governance as if you were all working together in the kitchen - collaborating with just enough structure to ensure efficiency, and adjusting your work patterns from day to day as necessary to meet requirements.

You may be interested in my slides from the keynote, which addressed these issues.  Also, following discussions at the conference around data management, I have prepared a new version of my Balanced Scorecard for SOA Governance.  The organizers, SDA India, videotaped a long interview with me which will be online in due course - when I know the link, I’ll post it to this blog.

In my next post, I will discuss a related topic - namely, mashups, of which there are more than one kind.  Stay tuned.

In my last post, I discussed the similarities and, more importantly, the differences between architecture of a very old-fashioned kind - mediaeval cathedral-building - and architecture of a more modern kind, namely SOA.  In this post I’d like to take the analogy a little further.

Most cathedrals were built before the basic principles of physical dynamics were worked out.  It was not until the 18th century that mathematical physicists such as Poisson and Cauchy developed the ideas that enabled structural engineers to predict stress patterns in a large body.  Until then, architects - or master masons, as they were known - relied on knowledge handed down orally, from one guild member to another.  This knowledge amounted effectively to rules of thumb, albeit highly sophisticated ones.

Hence, even the supreme achievements of mediaeval architecture were constructed without a true knowledge of what actually held the structure together.  The trial and error of centuries had established that certain approaches were valid, and that these approaches scaled well.  This was enough to construct masterpieces such as Chartres.  But they didn’t really know why it worked.

Recognizing this, the master masons took steps to ensure the safety of their structures: they employed a technique that modern engineers call "similar redundancy".  Basically, they built more columns, buttresses, arches and so on than they probably needed.  Similar redundancy works well if you are confident in the validity of your approach.

Skipping over the next few centuries to the modern day, engineers in fields that demand high system integrity (such as aerospace, for example) now prefer a technique called "dissimilar redundancy".  This technique involves first identifying the separate functions of a system (raise the ailerons, decrease the pressure, turn on lighting, etc) - then commissioning a solution for each function from several different manufacturers, who are not permitted to discuss the matter amongst themselves at all.  In this way, it is hoped that any fundamental error in one design will not be propagated into the other designs.  All the designs are used together in the production system, so that each function has a main operating solution plus a number of "dissimilar" backup solutions.  Dissimilar redundancy is a stronger technique than similar redundancy.

Turning to SOA, what is the relevance of this discussion?  In an enterprise computing environment pre-SOA, there are generally a number of competing "sources of truth" - for example, different ERP systems, front-end operational systems, data marts, data warehouses, and so on.  It is common to shield senior management from this complexity via a layer of reconciliation activity inside the organization, that attempts to understand the differences, or at least choose a set of figures for presentation to the board.  Nevertheless, one can safely say that most people working on business systems - and certainly all accountants - accept that a "single source of truth" is a complete myth in enterprise life.

With SOA, this starts to change.  The underlying systems are generally still there, of course, but the reconciliation is more likely to be done by IT staff than by managers and accountants, as part of the design of an SOA infrastructure.  Which service do we use to present "total revenue per region per quarter" on a dashboard?  Service XYZ123, from system XYZ.  Or the mean result of XYZ123 and service PQR678 from system PQR.  Or the median result of 10 services … Or …

Whatever solution is chosen, the people who really understand what to do about reconciliation (namely, managers and accountants) are out of the loop in daily working practice, even if they were involved in the initial design.  Effectively, dissimilar redundancy has been dispensed with, since the new solution treats the various sources of truth not as alternative implementations, but as part of a single implementation.

Further, even similar redundancy is often dispensed with!  It is quite possible to build an SOA infrastructure in which key services have no parallel implementation (running on different servers, using different middleware, etc).  This is not good practice, of course, but in a large enterprise with thousands of services, it can easily happen - and if it does, a single bug in such a service could bring down your entire operation like a house of cards.

TAKE AWAY

Next week I will be delving deeper into these issues as part of my keynote speech at SOA India 2007, and how to solve them via governance techniques based on Human Interaction Management.  I will also be running a workshop on the last day of the conference (you have to pay separately for this, but apparently, over 150 people have already signed up). If you’re at the conference, come up and say hi.

You might also like to have a listen to the podcast that Elizabeth Book (Editor-in-Chief of ebizQ) and I just recorded, in which we discuss the latest version of the free HumanEdj, that provides software support for Human Interaction Management principles.  Elizabeth writes: "I am actually about to download this thing to use in my own office".  Why not follow her example?  And I am always interested in your feedback - do let me know what you think.

With only a couple of weeks to go until SOA India 2007, I’ve been putting thoughts together for my keynote speech. Something I may or may not mention (haven’t decided yet) is a comparison between “architecture” of the service-oriented kind, and “architecture” as practised by the cathedral-building stonemasons of the middle ages. Cathedrals were effectively the mediaeval equivalent of middleware - the large-scale infrastructure via which stakeholders in the Christian religion (i.e., church officials) serviced their customers (i.e., the Christian community).
Looking back so far, it is hard to know exactly what a master mason did when working on one of these huge buildings. However, we have some idea of the principles they used, and some idea also of the working practices employed by the guilds of the time. Interestingly with regard to a comparison with SOA, we also know that most of the work would effectively have been modification rather than construction from scratch. For a start, many of the major cathedrals were constructed over a period longer than a single working lifetime. Further, early cathedral building often proceeded on a trial-and-error basis - for example, there are records of buttresses being built only when certain parts of the structure started falling down.
This somewhat experimental approach changed in the gothic period, when new design principles meant that a small and elegant set of supports could hold up a huge edifice - and moreover, do so while letting in huge amounts of light. This transformed not only the construction process, but the experience of those inside the building - from being huddled in a cold and gloomy environment surrounded by yards of thick stone, to being bathed in golden rays of warming and uplifting sunshine.
So am I saying that SOA is like gothic architecture? A new and vibrant experience for the users, thanks to a step change in technological sophistication?
It may be, but that’s not the point I’m trying to make. Rather, my point is about the difference between the two forms of architecture.
In general, cathedrals change their structure, very slowly, if at all. When they do change, it is usually by accretion - the addition of new buildings, rather than modification to the old ones. This means that the original design principles are largely unaffected by change. Hence, if a cathedral is still standing a few years after construction, it will probably still be standing a few hundred years after construction (barring the usual acts of god, or more often, acts of man).
With SOA, on the other hand, we can expect constant and often dramatic change both to the structure and functions of a system. After all, the main USP of SOA is that it helps facilitate such change! This means that original design principles are little or no guide to the longevity of the system. In other words, however clever the original designers were, the whole system can be compromised hopelessly by their successors at any time.
TAKE AWAY
A capacity to deal with change is often touted as the main attraction of an SOA approach. Yet this pre-supposes that change is managed properly. I see no evidence for this happening. Change on an enterprise scale, that spans multiple applications, is a complex human-driven process involving many different parties: usually many different departments, and often many different organizations altogether. The only tools SOA vendors are providing to deal with such processes are repositories for technical artefacts, that sometimes include the ability to make issue lists. In a nutshell, this isn’t even close to enough, since it provides no mechanism for Human Interaction Management.
Further, the problem is exacerbated by subtle issues of system robustness - which we can glean from a deeper comparison between cathedrals and enterprise applications. Tune in to my next post for more mediaeval illumination