Mashups and SOA: The Last Mile

May 26, 2008

I love great analogies – they allow us to explain complex things in simple terms. I found a real gem today. Unfortunately I can’t find the blog post in which I read it. In the post, mashups were described as SOAs “last mile” (I’m paraphrasing here because I can’t remember the exact quote). I don’t think I’ve read a more succinct description of the relationship between SOA and mashups.

The “last mile” is a telecommunications term used to describe the section of a telecommunications line that runs from your nearest exchange to your home. I think this is a great analogy for mashups and SOA. Here’s why.

The telecommunications system is a complex thing – exchanges, switches, cables, satellites and who knows what else. Thankfully, all of this complexity is hidden from the end user. We use the telephone network every day through a simple ‘user interface’ – the telephone. Most of us don’t know (or even care) how it all works. For the end user, the true value of the telecommunications network is realised through the ‘last mile’ of wire that connects the telephone to the telecommunications network.

SOAs are similarly complex – just like the telecommunications network, there’s plenty going on under the hood but if implemented properly, all of the SOA complexity is hidden from the users. And mashups may be the tangible thing that allows users to realise the full value of the infrastructure and plumbing that is SOA.

Without the telecommunications network, the telephone is a useless gadget. Conversely, without the telephone that sits on your desk, the telecommunications network is a nebulous concept without a practical use.

I believe that mashups need a proper SOA foundation to realise their full potential. I also believe that building a SOA that doesn’t offer tangible benefits that the business can see and touch is a pointless exercise.

PS: If anyone knows the post I’m talking about please comment below or email me.


Governance and Innovation

May 24, 2008

The word “governance” is getting a pretty bad reputation as bureaucratic red tape getting in the way of people trying to “do stuff”. It’s seen as a barrier to innovation. I take the opposite view. I think that governance promotes innovation. In fact, its essential to innovation.

Thanks to SOA, SaaS, PaaS, WOA, mashups etc there’s a real buzz in IT right now. But have you stopped to think what is the key driver that has allowed these ideas to gain traction? In a word: standards.

Standards define the rules of the game. If everyone plays by the rules, everyone gets along. Standards are a form of governance, but they have clearly encouraged rather than stifled innovation.

Governance is also good in everyday life. Online retailing was an innovation that was made possible by governance: privacy laws mean we don’t mind handing over our personal details and credit card information to online retailers because we know they won’t be divulged to 3rd parties.

Similarly, governance will be a catalyst for innovation within your organisation. Currently the vast majority of data and functionality resides in line of business (LOB) applications. The application and its data is ‘owned’ by a particular department, but the true power of emerging technologies, such as SOA, comes from taking an enterprise-wide approach. LOB managers need to be reassured that opening up their systems to the rest of the enterprise won’t adversely affect them. This is achieved by governance. Without governance, the enterprise will become a ‘wild-west’ free for all and every manager will circle their wagons around ‘their’ data and applications. Obviously this will inhibit innovation.

Governance is also critical when looking outside of the enterprise. When we type in our credit card on a web page we want to know that the site is ‘secure’ – that’s governance. Similarly governance will give people piece of mind that any web based services they use are secure, reliable and stable. No one wants to be responsible for building a mashup around a web-based service which steals customer details.

So how do you go about introducing governance in a way that doesn’t inhibit innovation? Here’s two tips you might find useful:

1. Just enough governance, just in time.
Introducing governance is an incremental and iterative process. Trying to do it all up front just doesn’t work. Provided that your governance maturity level grows in step with your growing IT sophistication, everything will be fine. This lets people try things out and innovate – without unnecessary barriers that may kill enthusiasm. But when something proves successful and is about to move from ‘prototype’ to ‘production’ then it’s time to put of a bit of rigour around it – and introduce a bit more governance.

2. Understand the psychology of governance.
No one likes to “be governed” – but the truth is that everyone likes “governance”. We don’t like to be told what to do but we’re happy that society is governed by laws, and that there are governments in place to provide things like police, schools and hospitals. So when it comes to ‘selling’ the importance of governance, make sure that it doesn’t come across as ‘big brother’. Rather, show people what’s in it for them. Bad: “we need to introduce governance so we know who is doing what with the data”. Good: “with a bit of governance we can make sure that no one pulls the plug on the data you use in your mashup”.

Time To Rethink Data?

May 23, 2008

I was having a quick browse through the Google App Engine documentation. I found one thing in particular very interesting, and that was the App Engine’s Datastore. Data is not stored like a traditional relational database. Data is stored as “entities” which have a key and a set of properties.

Why is this interesting? Because I think it’s a great time re-think the concept of “data” and what it really means in this brave new world of SOA, mashups, services, SaaS, PaaS and Web 2.0.

The vast majority of corporate data is stored on mainframes or in relational databases. I don’t have any statistics to prove it, but from my experience and anecdotal evidence, I would say that ‘object relational databases’ never really took off. I don’t know enough about them to speculate on why that is. The Google App Engine Datastore is an object data store – albeit a very simplistic one.

When thinking about “data” it’s very easy to fall into the trap of thinking “relational data”. I must confess that when I’m talking to the business about a problem domain I’m mentally drawing ER diagrams in my head. It’s years of conditioning I guess. I need to force myself to forget ‘physical’ and stick with ‘logical’.

The reality is – some data isn’t relational. And even if it was, does it necessarily mean that it has to be modelled/stored that way physically in a relational database?

Something to think about.

I often say that you have to challenge your assumptions – to put everything ‘under pressure’. When it comes to enterprise architecture, this isn’t just a “good idea”. It’s an important part of your job.

For those interested here’s the documentation:

Black-Box and White-Box SOA Services

May 21, 2008

Recently I had an interesting conversation with a business process professional, on the subject of SOA services. I started talking about ‘white-box’ and ‘black-box’ services, assuming they were clearly understood terms. It turns out that they weren’t. Here’s how I explained the two terms, and why they’re important.

The services in your SOA need to be ‘black-box’. If ‘white-box’ services creep into your architecture sooner or later you’ll have problems because white-box services increase coupling, thus violating one of the fundamental tenets of SOA (that services should be loosely coupled).

So what’s the difference between white-box and black-box?

Everyone is familiar with airline travel so let’s use that as an example.

Let’s say you want to go on a trip to attend a sporting event.

Now there are two ways of achieving your goal of attending the sporting event.

The Black-Box Way
You find a good travel agent and you tell them “I want to go to ‘such and such’ a sporting event on this date and I want you to arrange everything I need to get me there”.

They’ll ask you to fill out a form with all the information they need to fulfil your request: dates/times, credit card details, name etc, In SOA terms this form is the ‘contract’ through which you interact with the travel agent ‘service’.

You don’t care what they do with that form, who they have to speak to or how they go about arranging everything. It could be all online in real-time or maybe the pick up the phone and make all the bookings manually with each party concerned (event ticket office, airline, airport transfer company etc). You tell them WHAT you want and leave them to it. It’s “black-box”.

The White-Box Way
You find a not-so-good travel agent. They ask you to fill out 5 forms:

  • A form for booking a taxi from your house to airport through Tristate Taxis Co.
  • A form for booking an airline ticket through Qantas.
  • A form for booking a transfer from airport to hotel through Fred’s Airport Transfers
  • A form for booking a hotel room at the Hyatt.
  • A form for booking a event tickets through Ticketek

In this situation you can see inside the travel agent ‘service’. You know exactly HOW they’ll go about getting you to your sporting event. You know step-by-step what is involved and who is involved.

Why is this bad? Because the travel agent ‘service’ wont be able to change anything about its internal implementation without affecting consumers:

  • What if you want to use a different airline?
  • What you need a transfer from the airport straight to the event, but your luggage needs to be sent to the hotel?

Keep your SOA services black-box.
That is a simplistic example. The core concept is: consumers of services should only be concerned with WHAT your service does. Services should not leak any details of HOW they do


May 19, 2008

A few days ago I discussed how Web oriented Architecture (WOA) relates to SOA. Today I’ll touch on EDA. Some aren’t sure whether to go with EDA or SOA. The great news is, that’s not a decision you need to make. It’s like asking yourself a nonsensical question like “should I have a drink or should I quench my thirst?”

Let me cut to the chase: EDA is “event driven SOA”, it’s as simple as that.

If you design your SOA around numerous independent, autonomous services which are unaware of each other and merely publish events which other services subscribe to, then you have event-driven SOA. Or EDA. Take your pick.

So if the two terms (SOA and EDA) are so intertwined where has the confusion come from?

The answer is: EDA is a subset of SOA. It is quite possible that your SOA may use RPC style ‘command messages’. This is a technique that falls outside the scope of EDA – but it is quite legitimate in SOA. When I ‘legitimate’ I mean ‘if used correctly, with implications clearly understood’. It should be the exception rather than the rule.

So. If you have a drink then you’ll quench your thirst. If you create an event-driven SOA then you’ll have EDA. Forget about acronyms – they’re used by vendors and analysts trying to create markets or carve out niches for themselves. Architects should be looking for the substance behind the hype.

Complex Event Processing – In Business Terms

May 15, 2008

If there is one thing that IT people great at, it’s coming up with labels and acronyms that are guaranteed to elicit absolutely zero enthusiasm from business folk. Complex Event Processing (CEP) is yet another TLA (three letter acronym) in that rogue gallery. But just like ‘SOA’, it has immense business potential. If I was cornered, and absolutely positively had to explain CEP to a business person, here’s how I would explain it.

Business person:
Raf, what’s this ‘complex event processing’ thing I keep hearing about?

A lot of stuff happens in a typical business day – a lot of events.

Some events are just noise, and we don’t care about them. For example, Jim’s 2 o’clock meeting is an event but we don’t really care about it from a business standpoint.

Other events are important: an order has been placed, an invoice has been paid, a shipment has been despatched, an account has become overdue. These events are important to the business because when these events occur, follow up actions are triggered: money is posted into accounts, inventory is updated, a reminder letter is produced. These types of events are built into business processes and are very visible.

But there are other events that happen in the business that aren’t so visible. This is where complex event processing comes in. But forget the word ‘complex’. These events are not ‘complex’ in the sense of being difficult to understand. They are complex because they are made up of other, simple, events.

If it helps, forget the phrase ‘complex event processing’ and think of it as ‘business event analysis’.

You’re already familiar with the concept of ‘data mining’ for business intelligence. Complex event processing is essentially ‘event mining’. What we do is capture a huge number of simple events and then analyse them. Typically, we look for patterns in these events, timing, relationships, causality and the like. When we find a pattern deemed important to the business, we generate a new event to notify the people interested so that action can be taken.

Heres an example (albeit contrived):
Say your credit card provider noticed a number of ‘credit card purchase made’ events. Individually, each of these simple events will trigger some action – like adding the purchase to your account. The story could end there.

But say the credit card provider uses complex event processing (or ‘event mining’). They might notice that the first purchase was made in an electronic shop in Hong Kong. The second purchase was made only 10min later – in Shanghai, and a third purchase was made in Sydney a few minutes after that. If you processed each event in isolation you wouldn’t see the bigger picture: irregular purchase patterns. A new (complex) event would be raised: ‘potential fraud detected’ event, and action could be taken in response to the event: cancel credit card and notify card holder.

So that’s it – complex event processing in a nutshell.

Business Driven Mashups: Bottom-Up SOA?

May 14, 2008

When developing a service oriented architecture bottom-up is generally seen as bad. Top-down is generally better, and the pragmatists among us argue for a middle-out approach. If you’re not careful, the rising popularity of user-developed, browser-based, graphical mashups will force you down the road of bottom-up SOA design. And that is not a road you want to be on.

In SOA, the same concept means different things to different people so a few definitions:

  • Top down: focuses on analysing the business, in particular where it is and where it wants to go. The SOA is designed to support the business. Close alignment between business and IT.
  • Bottom up: focuses on service-enabling existing IT assets, often without regard to the business drivers for doing so. There is no strategic plan. IT builds a huge services layer in the hope that what they build may be useful to someone, at some point.

It may be oversimplifying it a bit but you could say that top-down is seen as ‘business-driven’ SOA and bottom-up is seen as ‘IT-driven’ SOA.

But that is now changing. Mashups are gaining popularity within the ranks of business power users. Services oriented architectures are starting to be seen by some as mashup platforms. Strictly speaking there’s nothing wrong with that. Mashups are one tangible, visible result of the behind-the-scenes plumbing that is SOA.

Here’s an interesting question though:

If: Mashup adoption is largely business-driven (by end-users).
And: The top-down (i.e. business driven) approach to SOA is the preferred way
Does that mean: that mashup-driven SOA design is a good thing?

The answer (as usual) is: it depends.

On the one hand, SOAs are supposed to support the business. If you develop services that can’t be used by mashups then your SOA is not supporting the business. Mashups provided a good ‘validation tool’ for checking that you’ve designed your services in a way that is useful to the business. Mashups are one potential consumer of SOA services, and will become an increasingly important one.

On the other hand, you can’t have each business unit submitting requests to IT to develop services specifically for their mashup, without considering what is going on in the rest of the business. This will result in an explosion of services developed in isolation, with no reuse potential and no reference to an enterprise-wide strategy. This is bottom-up design. And it’s bad. Just because it’s business-driven bottom-up rather than IT-driven bottom-up, doesn’t make it any better.

Mashups make bazaar-style innovation possible, but the old principles still apply. You need to take a big picture view and create a SOA strategy that aligns with the needs of the entire business.