I never quite fathomed the software development methodology craze that was gripping enterprise computing when I came onto the scene in the early '90s. In those days development teams were managing complexity and enforcing quality via draconian software development life cycles (SDLCs).

A lot of big enterprises were using SDLCs that they had purchased as mindshare from system integrators or tool vendors. The ones people paid for came to be known as big-M methodologies, to denote their hegemony in the field. There was even a meta-process to measure how refined your methodology was. It was called CMM - the "Capability Maturity Model." CMM had five levels of "maturity" to which enterprises aspired, with each level adding an increasing number of artifacts, expensive tools, consultants, and documentation reams. If you ever toiled as a developer under a CMM-focused methodology, you would have gotten the sense that your job was to produce CMM artifacts, not software, in search of that elusive level-5 competency certification.

Stringent methodologies are perfectly suitable for the design of systems that never change, like an airplane flight control system. Since airplanes don't grow new wings, or add extra pilots over the course of time, or have to talk to other airplanes much, a software development methodology that makes it next to impossible to change the system makes sense. In fact, I would say it is highly desirable once the system works. The fact that it takes 10 years to develop new systems in this way is not a problem...it takes longer than that to create the plane's hardware anyway. This is not the case with systems predicated on integration patterns or having anything to do with the Internet. Change is the order of the day in these systems, and CMM-happy enterprises did not fare well in this climate. If a dollar figure could be placed on the cost of system development (actual cost and missed opportunity cost) to business because such methodologies were used, I am sure it would be astounding.

When I decided I wanted to be an SOA architect a few years ago, the old '90s developer in me wanted some type of methodology to follow, while the skeptic in me demurred. Further confusing the issue was the fact that SOA, the business-focused concept, has not been well defined for that long. It took a while for the dots to connect between the Web services and EAI platforms that we were using a few years back, all the way through to a business-focused agile execution paradigm. Along the way we were cautioned by analysts that there was great risk in doing SOA, that the business would undoubtedly be skeptical - especially when we asked for money to retool - and that we should do small pilots to validate ROI and assumptions about reuse in our enterprises. "Don't try to boil the ocean," they said. "Limit the scope of the problem domain or you will fail." Then along came 2006, which has been a year in which en masse SOA projects seem to be appearing with more regularity. As usual, those of us in IT do not have the luxury of scoping the problem domain. The business drives our projects, and the promise of efficiency and competitive advantage of SOA has derived the need for an all-encompassing way to proceed.

In thinking about the problem, it occurred to me that the ability of a services-based architecture to manage complexity (and thus engender agility and efficiency) stems from how architects can divide and conquer problem domains by using its precepts. The divide-and-conquer approach is the oldest method known to man for addressing complex problems. The scale of the problem is irrelevant as far as the method is concerned - the larger and more complex the problem is, the more dividing and conquering you do. With the big-M methodologies, when the problem got more complex, you were in even more trouble. So I wondered if a methodology (little-m) could be distilled for en masse SOA enablement by dividing enterprise development aspects along SOA boundaries as dictated by the fundamental concepts of SOA, including business and organizational aspects. I use the term "enablement" to denote the fact that we are not concerning ourselves with implementing an application (or set of applications). We are focused on creating an infrastructure that supports creation and assembly of services and related enterprise aspects via metadata management at sufficient change velocity to support our vision of business agility, as well as repurposing existing components and services for use in SOA. Specific applications and service compositions then leverage the SOA-enabled enterprise to meet specific business requirements.

Conventional wisdom has dictated that SOA projects start with and focus primarily on all things services - service creation, lifecycle management, and governance. Not so much has been put forth on repurposing an existing enterprise for a large-scale SOA project. Such an SOA enablement effort necessarily focuses more on the build out of an SOA infrastructure and leverages the well-worn service delivery and management methods axiomatically. In Part 1 I will focus on analysis for en masse SOA enablement, with design/build/test being addressed in Part 2. It is the purpose of the analysis phase to elicit requirements for our SOA infrastructure and its associated change management facilities, thereby providing to the design phase what it needs to produce a detailed infrastructure design. Though it has been said that we are building an SOA enterprise and not a specific application, the fact remains that the problem domain will not be expressed in the abstract; it will be the usual mix of actors, process flows, interactions, use cases, etc. As such, separate requirements sets must be derived - system requirements that define specific process flows, services that support them, and consumers that consume them, and an infrastructure (non-functional) requirements set that will drive the design of all of the non-functional aspects of the enterprise SOA, i.e., the SOA "enablement" of the enterprise. The trick (and focus of Part I of this series) is to glean enough from the analysis of the problem domain to drive design of the SOA infrastructure such that it can serve the business's notion of agility and efficiency for as long as possible without too much modification. A series of broad work steps will be proposed, moving from the more abstract to the more specific, which should allow the infrastructure requirements set to be completed if applied to a specific enterprise.

First, Develop an Overarching Agility Vision
It has long been understood that the business value proposition of SOA is that it engenders agility and efficiency in enterprises. SOA agility is defined as the measure of the cost of change in IT systems: dollar cost and opportunity cost. If your enterprise can accommodate change easily, then changes demanded by business are going to be cheaper. In addition, accommodating more change allows business to offer a higher quality of service to customers and likely to achieve competitive advantage against less agile competitors. Conventional wisdom for piecemeal SOA rollouts has told us that to be successful in our efforts, we needed to establish success criterion and even detail ROI metrics by which we can chart our progress. In an en masse SOA enablement effort, we take this to the nth degree and paint an overarching agility vision. The analysis phase of the project must create this vision. It should be grandiose - idealistic in fact. It should be simple; easily understood by everyone on the team within a couple of minutes. It should make people ask, "How could we possibly get there from here?" SOA offers the ability to deliver extreme agility in due course of time (likely years) and project architects and business analysts would be remiss not to shoot for it. In keeping with the spirit of divide and conquer, the vision can well be broken out by sub-domain, like lines of business. By staying true to such a futuristic vision of your enterprise, downstream analysis and design tasks become easier. Start your analysis by creating an overarching agility vision for your enterprise.

Reengineer the Business Domain
Any kind of system analysis is going to focus on the business problem domain. Detailed domain decomposition is critical for accurate development and should be conducted by a team of business analysts with representation from the IT architecture team. If the business cannot accurately detail the task at hand, there is no hope for IT to deliver. With en masse SOA, it is essential to ensure that the business domain is taking advantage of the new capabilities it is being offered - the business domain should be re-engineered. For example, if it adds competitive advantage to the business to be able to interact with 100 new business partners per year electronically, each having a different access policy profile, then demand it up front. "Business re-engineering" is a loaded phrase but an important concept here - an enterprise recast as an SOA can offer business a degree of agility that may be unexpected, such that it may well need changing itself. In en masse SOA enablement, the business is basically telling IT to reinvent the way that it enables business. Hence, a virtuous cycle is possible whereby business re-engineers and IT war-games enterprise changes. This cycle should play out throughout the entire analysis phase. The result should be a highly focused set of system and infrastructure requirements for your "new" enterprise that supports your overarching agility vision.

Inventory Existing IT Assets
Every enterprise making a shift to SOA is going to have lots of assets that can (and should) be reused in the new effort, without compromising the vision of the new SOA. This has been recognized previously as a key SOA analysis step, most notably as a key element of the IBM SOMA process. In en masse SOA analysis the boundaries are extended beyond services and enterprise aspects are considered as well - we are trying to repurpose enterprise aspects for SOA. Any database, service (Web service or otherwise), workflow engine, mainframe, identity store, policy repository, and even client applications that trap rules or policies at the client tier are fair game. The mining process can be quite complex, and this is where the divide/conquer approach comes in handy. Create a team with representation from each group that owns relevant assets and give them a tool with which they can collaborate. There are some great stand-alone asset management tools out there that can also be used down the road for semantics sharing and for governance.

The goal of this endeavor, from the infrastructure requirements point of view, is to derive requirements for repurposing existing assets into the SOA. Some things are going to be readily reusable, like existing Web services. Others are going to have value but be less ready to participate in an SOA, like old EAI platform services or business rules trapped in client code. There will be a few pieces that cannot be a part of an SOA, particularly things that are not consistent with your agility vision. There will undoubtedly be wide consensus about things that cannot be included, like that poor-performing DAO code that no one wants to see bog down the new system. As such, it should become clear what is needed to support reuse, e.g., a way to do data access in SOA instead of using that brittle DAO code, and perhaps a way to repurpose EAI platform services that are not currently exposed as compliant Web services. Likewise, a need to reuse an enterprise business rule management system may be elicited. Capture these types of requirements in your infrastructure requirements set. In the design phase we will systematically repurpose all of these elements for use in the SOA.

Derive a Semantic Model for the Enterprise
SOA architects have long recognized that semantic interoperability between service producers and consumers is fundamental for meaningful SOA. Service semantics include message-exchange vocabularies and meaning as well as expressions of service policies, constraints, and quality of service concerns. SOA agility requires that human intervention to translate between data dictionaries, which underlie services at the component level, not be required. Techniques and patterns for deriving SOA semantics are well beyond the scope of this article. But it can be noted that en masse SOA enablement makes the effort more complex by giving it enterprise scale.

In keeping with the premise of divide-and-conquer as a means to deal with complexity, it can be postulated that a federation model, supported in the design phase by transformation and composition capabilities, is a great way to get at enterprise domain data modeling for en masse SOA. If the enterprise has existing data silos with associated data models, there is no harm in respecting these boundaries if vocabularies and associated metadata jibes with the business domain model, expressed as a canonical data format. If the business model for an investment house defines customer as both a customer profile and also includes all financial holdings, don't worry that three systems of record (perhaps managed by different groups) may be needed to fulfill a "getCustomer" component service request. SOA infrastructure components can be used to abstract transformation and composition details from service creators, but you must elicit and capture these infrastructure requirements in the analysis phase.