Service-Oriented Architecture
Perhaps the oldest "new thing" in the equation is the service-oriented architecture. First conceived alongside technologies like DCE and CORBA, service-oriented architecture separates the core of an application into a series of fairly coarse-grained components - chunks of business logic that provide "services" to client programs. These services can be shared between presentation (GUI) and integration applications (for example, as steps in a business process).

Processes communicate only through documented interface "contracts," which reinforce the design principles of modularity and encapsulation. Hiding the details of how each module works - and tying down its specification - makes it easier for external developers to reuse the component.

A particular class of service-oriented architecture uses complete documents - business messages if you like - to trigger each step in an overall business transaction. That's because there may be no single database that the process steps can all share. This style of interaction can be called "document-centric processing," to distinguish it from the database-centric model commonly used to build internal systems such as ERP and CRM. The document - the message passed from system to system - contains all necessary data items, and takes the same role as input parameters in a well-structured function call. Indeed, it is more flexible; because the document encapsulates all necessary data, there is no need for each step to know which specific data items are needed by the next step.

Using message-oriented middleware, it's easy to plug services together. And because it's easier, that makes it cheaper too. Interfaces tend to be simpler and cleaner. An enterprise can assemble systems from best-of-breed pieces, which may be hosted in different departments or even companies. There is a clear separation of concerns; components - each of which can be owned, developed, and deployed autonomously - can be separately scaled, replicated, or replaced. Different services can have different development life cycles, languages, and platforms. An enterprise can mix and match its 30-year-old mainframe systems - using IBM WebSphere MQ to kick off CICS transactions - with J2EE/Unix, Windows/.NET, and any other legacy processing (see Figure 1).

 

Easier integration also promotes more frequent reuse rather than redevelopment of components. When millions of dollars are invested in software assets, the last thing a business wants is to trash them just because it doesn't match their current preferred platform.

Content-Based Filtering and Routing
A key requirement of any integration platform is to identify which data to process and where it needs to be sent. Selection rules - typically specified using XPATH - identify the key sections of an XML document. These sections can be extracted and routed to processing components, and the results delivered to the next processing stage, based on its content. For example:

/order/item[price>10.80]

will select all order items with a price greater than 10.80, while

/order/[count(item)=1]

selects all orders that are for just a single item.

There's real value in exposing these selection rules as configuration metadata rather than simply hiding them in code. Business requirements are changing faster than applications can be created and modified. Rules offer a way of encapsulating the business semantics and promoting them to the surface, where they are much easier and cheaper to manage.

Transformation Capability
As it moves from system to system, data needs to be validated, translated, and enriched by calculations or lookups. These operations may be performed by the ESB itself, or they may be delegated to external systems (by synchronous or asynchronous calls). The ESB offers an environment in which these actions can be processed independently (and concurrently) using "actions," which are processing rules written in a range of scripting or declarative languages. XSLT is one common approach. But although XSLT engines are easily available at no cost, the engines can be rather slow and XSLT itself struggles to achieve the kind of complex transformations needed for application integration. ESB vendors need to add their own more powerful transformation engines so that documents can be processed swiftly and efficiently and then reassembled into one or more output documents that can be routed - based on their content - to the next step in an orchestrated sequence.

One important transformation capability is "any to XML" - translating legacy formats (EDI, structured files, and relational data) into XML for processing and onward transmission, and from XML back to legacy formats where required. Because of its semantic power, XML provides an excellent interchange and internal processing format even for non-XML inputs and outputs. The ESB executes these syntactic transformations at the edge - as documents enter and leave the engine - ensuring that a single technology can be used for core processing.

Support for Standard Interfaces
Of course, the ESB will support Web services interfaces, together with JMS for enterprise messaging, JCA for applications, and JDBC for databases. Widely adopted "industry standards" - proprietary interfaces like IBM WebSphere MQ messaging and CICS transaction monitor, and Microsoft's MSMQ messaging, for example - may also be supported. Increasingly, applications and services will communicate pragmatically, based on widely used protocols such as POP3/ SMTP and instant messaging, as well as HTTP, SOAP, and MOM. This makes it easier to manage and share infrastructure between applications and humans.

Once any middleware (such as a MOM) has been deployed, it can be hard to replace or upgrade. That's why so many organizations have two or more existing middleware products deployed, and why it is such a big mistake to tie an ESB to any single vendor's MOM. An ESB needs to adapt to the realities of today's businesses: bridging between multiple MOMs, as well as bridging between asynchronous (messaging) and synchronous (RPC - Remote Procedure Call) domains. An ESB is there to support and integrate whatever is already in use, not to replace it.

Distributed Operation and Management
Early integration products tended to follow a hub-and-spoke architecture - a straightforward approach that simplifies integration topology and centralizes configuration and management. In a batch environment, the slight risk of failure could be insured against by operating a standby hub for cold failover.

As software pervades the modern business, and interchange frequency increases, the centralized approach is no longer tenable and the costs of interruption are too great. Businesses need distributed networks with no single point of failure - to optimize network traffic, to provide alternate routes in case of failure, and to distribute the processing load. Offering services closer to participating applications - where possible - spreads the processing load as well as reducing network bandwidth utilization.

Of course this complicates system management, so it must be possible to manage today's globally distributed ESB from a single operational console, and to adjust its configuration without bringing down the entire integration network. Downtime must be kept to a minimum - remember that "five nines" (99.999%) availability allows for just five minutes for downtime per year, and even four nines leaves less than an hour.

What's in It for Developers?
Developers have often disliked integration products, preferring to use frameworks of their own devising. The main reasons given for this are:

However, there are long-term implications in adopting a "roll your own" policy. Will the organization be able to support the framework going forward? Use of a product can deliver significant productivity gains, reducing development time and cost by allowing the developer to focus on business logic rather than framework building or infrastructure coding, so that he or she needs only to write a fraction of the code normally required. Moreover, the code is fully open and transferable and no new skills are required, enabling rapid adoption and return on investment. Now that license costs have fallen, and products are much easier to work with, there's really no excuse for taking this kind of risk.

Existing developers and business users can rapidly deliver new solutions with minimum disruption to existing systems and maximum leveraging of existing assets and skills. XML-based ESB products can complement and readily integrate with familiar IDEs and configuration management systems; this is much less intrusive on the development process.

Challenges for XML Processing Architectures
One perceived impediment to the exponential uptake of XML is that it can deliver a rather verbose representation of a chunk of data. Mushrooming quantities of data being exchanged present a serious challenge to corporate infrastructure. This is not caused just by the addition of all those "human readable" tags. XML succeeds precisely because its structure is so flexible and extensible. Just as the introduction of the relational model made it easy to add new tables and columns to a database, so XML's syntactic power allows organizations to refine and specialize XML Schemas to handle all of their specific information needs. If necessary, entire document sub-trees can be added onto a standard schema without breaking third-party functionality. By encapsulating all contextual information needed by a series of business processes, XML encourages the development of loosely coupled, "document-oriented" processing. New systems are easier to plug in as messages can be extended to include all required data - there's no need for a multiplicity of calls back and forth for additional items.

First-generation XML integration pilots successfully demonstrated the potential of the technology, but they are under increasing strain as the quantity, size, and complexity of XML documents continues to grow. XML cannot be adopted for core systems until these problems are recognized and addressed. Among the causes of poor scalability are:

Routes to Scalability
What is needed is a processing architecture that helps the architect/developer organize:

A broker that can manage these techniques without requiring expert coding and configuration massively reduces the risks of project failure and consequent business damage caused by inadequate performance. Core applications, previously out of bounds by reason of data volume and complexity, can now be confidently addressed.

Conclusion
There has been a tectonic shift away from complex, proprietary, centralized, and costly integration brokers toward more lightweight, distributed, standards-based and inexpensive enterprise service bus technology; this has been driven by the adoption of open standards like XML, Web services, and J2EE.

The ESB offers a powerful and extensible integration platform that supports your development aims without forcing you to adopt proprietary technology, retrain your staff, or radically change your development methodology. By leveraging the substantial community investment in XML and other technologies, ESBs can equal or better the capabilities of earlier integration brokers at a far lower cost, bringing integration capabilities to smaller businesses and more marginal projects. Easy adoption of familiar technologies further reduces the cost of uptake and helps maximize return on investment for any size of business.