That's A Crazy Idea!

If the current year was 1965, and you made the statement: “The fastest, least expensive, and most efficient way to send a package from Utica, NY to Boston, MA is to send it through Memphis, TN”, most people who heard or read that statement would tell you that you were crazy! Why on earth would you send a package 965 miles in almost the opposite direction to Memphis, then turn around send it another 1137 miles from Memphis to Boston, when it is only 221 miles from Utica to Boston in the first place! Surely it makes more sense to send your package over the most direct route!

That year, a young Yale University student named Fred Smith conveyed that very same idea, but with a much more plausible and complete explanation. Fred Smith wrote a paper describing an idea for an overnight mail service. His idea was to provide overnight delivery service from almost any point in the country to almost any other point by sending every piece through a single, central hub. In spite of his thorough explanation of his idea, his college professor still thought the idea was crazy. The professor gave him a grade of C for his effort, and returned the paper with the comment that his idea would never work.

Eight years later, Federal Express began operations, and the rest, as they say, is history!. Fred Smith took his revolutionary idea and built it into an extremely successful overnight delivery business: "When it absolutely, positively has to be there!"

Certainly when you think of only one package at a time, without considering any other packages that may also need to be delivered, it only makes sense to send it by the most direct route possible. But if you examine the larger issue of how many packages need to be sent from almost any point in the country to almost any other point, if the number of packages is large enough, it makes sense to invest in some infrastructure that can solve the larger problem. Once you have invested in that infrastructure, and the machinery is in place, it then becomes more cost effective and more efficient to "creatively waste" the resources of airplanes, fuel, pilots, local agents, etc. to send a package over a circuitous route that may be 10 or even 15 times longer than the direct route.

What's That Got To Do With Our Computer System?

Most computer systems in place in larger companies today were built up over the years one program, or one subsystem at a time. The focus was almost always on the one new subsystem that was needed. In cases where I.T. people suggested that investments be made in infrastructure that would help over the long term, more often than not the decision was made to defer long term investment in favor of shorter term investment in the "great new subsystem" that would meet the current need to provide a return "next year". As each new subsystem was added, connections were made directly with other subsystems wherever they were needed. Just like the package delivery problem, the direct route was followed because no one had the time or money to consider a better long term solution that involved long term investment in infrastructure.

The result is what Brian Foote and Joseph Yoder have called the "Big Ball of Mud" or "Shantytown" architecture which is the default standard architecture in most companies. Sean McGrath has referred to this method of integration as "Hairball Integration". Albert Einstein is attributed with the statement that any system should be "as simple as possible, but no simpler." In a system developed with the "Ball of Mud" architecture, "complexity increases rapidly until it reaches a level of complexity just beyond that with which programmers can comfortably cope. At this point, complexity and our abilities to contain it reach an uneasy equilibrium." With this approach, you end up with "the most complicated system that could possibly work."

The same principle behind the success of Federal Express applies to successful computer system integration. The elimination of "tightly coupled" direct connections in favor of "loosely coupled" indirect connections reduces complexity and increases system scalability. While it adds extra work and more cost to require an extra layer of abstraction between any two subsystems that must communicate with each other, the result is more scalability and lower cost communication for the entire system. I think that point is worth repeating: The indirect connection between two points costs more, is more work, and slows communication between the two subsystems by a few milliseconds. However, the much lower cost, greater efficiency and improved scalability of the entire system overwhelmingly favors the indirect, or "loosely coupled" approach.

To I.T. professionals who had to spend many years working with very limited memory and storage capability, the idea of sending large amounts of data from program to program gives rise to an instinctive negative gut reaction. After years of conditioning, wherein a year had to be represented as "82", for example, because "1982" used too much precious, costly memory and storage, it's hard to accept the idea of using the lavish amounts of memory and storage required for a hub and spoke, loosely coupled system based on passing large amounts of data around. Even when such people accept and agree with this "loosely coupled" concept on a conscious, rational level, the negative gut reaction is still there, and it is very strong because of the many years spent carefully conserving the use of memory and storage. A loosely coupled system like the one described here would not have been feasible 30 years ago because the cost of memory and storage was just too high. The economics of computer hardware has changed, but instincts based on years of experience persist.

Let's face it, the GIF file containing the headline banner of our Company's website takes up more space than an XML file containing all the data elements of a complete insurance policy! That file, along with many more graphic files, is routinely sent across the internet to anyone who connects to our site from anywhere in the world without any problem at all! Memory and storage is simply no longer an issue. Given the current state of computer hardware where processing power, memory and storage are abundant and cheap, it is much less costly to install and use a "loosely coupled", wheel and spoke type of infrastructure than it is to throw more human resouces at an immensely complicated "Ball of Mud" in an attempt to adapt to changing business requirements. Such a "loosely coupled" infrastructure, based on passing "packages" of data from process to process where hardware resources are "creatively wasted" makes every bit as much sense as Fred Smith's idea for overnight package delivery.

Robert C. Davidson - July 17, 2003