Mashups and SOA: The Last Mile

I love great analogies – they allow us to explain complex things in simple terms. I found a real gem today. Unfortunately I can’t find the blog post in which I read it. In the post, mashups were described as SOAs “last mile” (I’m paraphrasing here because I can’t remember the exact quote). I don’t think I’ve read a more succinct description of the relationship between SOA and mashups.

---

The “last mile” is a telecommunications term used to describe the section of a telecommunications line that runs from your nearest exchange to your home. I think this is a great analogy for mashups and SOA. Here’s why.

The telecommunications system is a complex thing – exchanges, switches, cables, satellites and who knows what else. Thankfully, all of this complexity is hidden from the end user. We use the telephone network every day through a simple ‘user interface’ – the telephone. Most of us don’t know (or even care) how it all works. For the end user, the true value of the telecommunications network is realised through the ‘last mile’ of wire that connects the telephone to the telecommunications network.

SOAs are similarly complex – just like the telecommunications network, there’s plenty going on under the hood but if implemented properly, all of the SOA complexity is hidden from the users. And mashups may be the tangible thing that allows users to realise the full value of the infrastructure and plumbing that is SOA.

Without the telecommunications network, the telephone is a useless gadget. Conversely, without the telephone that sits on your desk, the telecommunications network is a nebulous concept without a practical use.

I believe that mashups need a proper SOA foundation to realise their full potential. I also believe that building a SOA that doesn’t offer tangible benefits that the business can see and touch is a pointless exercise.

PS: If anyone knows the post I’m talking about please comment below or email me.

Governance and Innovation

The word “governance” is getting a pretty bad reputation as bureaucratic red tape getting in the way of people trying to “do stuff”. It’s seen as a barrier to innovation. I take the opposite view. I think that governance promotes innovation. In fact, its essential to innovation.

---

Thanks to SOA, SaaS, PaaS, WOA, mashups etc there’s a real buzz in IT right now. But have you stopped to think what is the key driver that has allowed these ideas to gain traction? In a word: standards.

Standards define the rules of the game. If everyone plays by the rules, everyone gets along. Standards are a form of governance, but they have clearly encouraged rather than stifled innovation.

Governance is also good in everyday life. Online retailing was an innovation that was made possible by governance: privacy laws mean we don’t mind handing over our personal details and credit card information to online retailers because we know they won’t be divulged to 3rd parties.

Similarly, governance will be a catalyst for innovation within your organisation. Currently the vast majority of data and functionality resides in line of business (LOB) applications. The application and its data is ‘owned’ by a particular department, but the true power of emerging technologies, such as SOA, comes from taking an enterprise-wide approach. LOB managers need to be reassured that opening up their systems to the rest of the enterprise won’t adversely affect them. This is achieved by governance. Without governance, the enterprise will become a ‘wild-west’ free for all and every manager will circle their wagons around ‘their’ data and applications. Obviously this will inhibit innovation.

Governance is also critical when looking outside of the enterprise. When we type in our credit card on a web page we want to know that the site is ‘secure’ – that’s governance. Similarly governance will give people piece of mind that any web based services they use are secure, reliable and stable. No one wants to be responsible for building a mashup around a web-based service which steals customer details.

So how do you go about introducing governance in a way that doesn’t inhibit innovation? Here’s two tips you might find useful:

1. Just enough governance, just in time.
Introducing governance is an incremental and iterative process. Trying to do it all up front just doesn’t work. Provided that your governance maturity level grows in step with your growing IT sophistication, everything will be fine. This lets people try things out and innovate – without unnecessary barriers that may kill enthusiasm. But when something proves successful and is about to move from ‘prototype’ to ‘production’ then it’s time to put of a bit of rigour around it – and introduce a bit more governance.

2. Understand the psychology of governance.
No one likes to “be governed” – but the truth is that everyone likes “governance”. We don’t like to be told what to do but we’re happy that society is governed by laws, and that there are governments in place to provide things like police, schools and hospitals. So when it comes to ‘selling’ the importance of governance, make sure that it doesn’t come across as ‘big brother’. Rather, show people what’s in it for them. Bad: “we need to introduce governance so we know who is doing what with the data”. Good: “with a bit of governance we can make sure that no one pulls the plug on the data you use in your mashup”.

Time To Rethink Data?

I was having a quick browse through the Google App Engine documentation. I found one thing in particular very interesting, and that was the App Engine’s Datastore. Data is not stored like a traditional relational database. Data is stored as “entities” which have a key and a set of properties.

---

Why is this interesting? Because I think it’s a great time re-think the concept of “data” and what it really means in this brave new world of SOA, mashups, services, SaaS, PaaS and Web 2.0.

The vast majority of corporate data is stored on mainframes or in relational databases. I don’t have any statistics to prove it, but from my experience and anecdotal evidence, I would say that ‘object relational databases’ never really took off. I don’t know enough about them to speculate on why that is. The Google App Engine Datastore is an object data store – albeit a very simplistic one.

When thinking about “data” it’s very easy to fall into the trap of thinking “relational data”. I must confess that when I’m talking to the business about a problem domain I’m mentally drawing ER diagrams in my head. It’s years of conditioning I guess. I need to force myself to forget ‘physical’ and stick with ‘logical’.

The reality is – some data isn’t relational. And even if it was, does it necessarily mean that it has to be modelled/stored that way physically in a relational database?

Something to think about.

I often say that you have to challenge your assumptions – to put everything ‘under pressure’. When it comes to enterprise architecture, this isn’t just a “good idea”. It’s an important part of your job.

For those interested here’s the documentation: http://code.google.com/appengine/docs/datastore/

Black-Box and White-Box SOA Services

Recently I had an interesting conversation with a business process professional, on the subject of SOA services. I started talking about ‘white-box’ and ‘black-box’ services, assuming they were clearly understood terms. It turns out that they weren’t. Here’s how I explained the two terms, and why they’re important.

---

The services in your SOA need to be ‘black-box’. If ‘white-box’ services creep into your architecture sooner or later you’ll have problems because white-box services increase coupling, thus violating one of the fundamental tenets of SOA (that services should be loosely coupled).

So what’s the difference between white-box and black-box?

Everyone is familiar with airline travel so let’s use that as an example.

Let’s say you want to go on a trip to attend a sporting event.

Now there are two ways of achieving your goal of attending the sporting event.

The Black-Box Way
You find a good travel agent and you tell them “I want to go to ‘such and such’ a sporting event on this date and I want you to arrange everything I need to get me there”.

They’ll ask you to fill out a form with all the information they need to fulfil your request: dates/times, credit card details, name etc, In SOA terms this form is the ‘contract’ through which you interact with the travel agent ‘service’.

You don’t care what they do with that form, who they have to speak to or how they go about arranging everything. It could be all online in real-time or maybe the pick up the phone and make all the bookings manually with each party concerned (event ticket office, airline, airport transfer company etc). You tell them WHAT you want and leave them to it. It’s “black-box”.

The White-Box Way
You find a not-so-good travel agent. They ask you to fill out 5 forms:

• A form for booking a taxi from your house to airport through Tristate Taxis Co.
• A form for booking an airline ticket through Qantas.
• A form for booking a transfer from airport to hotel through Fred’s Airport Transfers
• A form for booking a hotel room at the Hyatt.
• A form for booking a event tickets through Ticketek

In this situation you can see inside the travel agent ‘service’. You know exactly HOW they’ll go about getting you to your sporting event. You know step-by-step what is involved and who is involved.

Why is this bad? Because the travel agent ‘service’ wont be able to change anything about its internal implementation without affecting consumers:

• What if you want to use a different airline?
• What you need a transfer from the airport straight to the event, but your luggage needs to be sent to the hotel?

Keep your SOA services black-box.
That is a simplistic example. The core concept is: consumers of services should only be concerned with WHAT your service does. Services should not leak any details of HOW they do

EDA and SOA

A few days ago I discussed how Web oriented Architecture (WOA) relates to SOA. Today I’ll touch on EDA. Some aren’t sure whether to go with EDA or SOA. The great news is, that’s not a decision you need to make. It’s like asking yourself a nonsensical question like “should I have a drink or should I quench my thirst?”

---

Let me cut to the chase: EDA is “event driven SOA”, it’s as simple as that.

If you design your SOA around numerous independent, autonomous services which are unaware of each other and merely publish events which other services subscribe to, then you have event-driven SOA. Or EDA. Take your pick.

So if the two terms (SOA and EDA) are so intertwined where has the confusion come from?

The answer is: EDA is a subset of SOA. It is quite possible that your SOA may use RPC style ‘command messages’. This is a technique that falls outside the scope of EDA – but it is quite legitimate in SOA. When I ‘legitimate’ I mean ‘if used correctly, with implications clearly understood’. It should be the exception rather than the rule.

So. If you have a drink then you’ll quench your thirst. If you create an event-driven SOA then you’ll have EDA. Forget about acronyms – they’re used by vendors and analysts trying to create markets or carve out niches for themselves. Architects should be looking for the substance behind the hype.

Complex Event Processing – In Business Terms

If there is one thing that IT people great at, it’s coming up with labels and acronyms that are guaranteed to elicit absolutely zero enthusiasm from business folk. Complex Event Processing (CEP) is yet another TLA (three letter acronym) in that rogue gallery. But just like ‘SOA’, it has immense business potential. If I was cornered, and absolutely positively had to explain CEP to a business person, here’s how I would explain it.

---

Business person:
Raf, what’s this ‘complex event processing’ thing I keep hearing about?

Raf:
A lot of stuff happens in a typical business day – a lot of events.

Some events are just noise, and we don’t care about them. For example, Jim’s 2 o’clock meeting is an event but we don’t really care about it from a business standpoint.

Other events are important: an order has been placed, an invoice has been paid, a shipment has been despatched, an account has become overdue. These events are important to the business because when these events occur, follow up actions are triggered: money is posted into accounts, inventory is updated, a reminder letter is produced. These types of events are built into business processes and are very visible.

But there are other events that happen in the business that aren’t so visible. This is where complex event processing comes in. But forget the word ‘complex’. These events are not ‘complex’ in the sense of being difficult to understand. They are complex because they are made up of other, simple, events.

If it helps, forget the phrase ‘complex event processing’ and think of it as ‘business event analysis’.

You’re already familiar with the concept of ‘data mining’ for business intelligence. Complex event processing is essentially ‘event mining’. What we do is capture a huge number of simple events and then analyse them. Typically, we look for patterns in these events, timing, relationships, causality and the like. When we find a pattern deemed important to the business, we generate a new event to notify the people interested so that action can be taken.

Heres an example (albeit contrived):
Say your credit card provider noticed a number of ‘credit card purchase made’ events. Individually, each of these simple events will trigger some action – like adding the purchase to your account. The story could end there.

But say the credit card provider uses complex event processing (or ‘event mining’). They might notice that the first purchase was made in an electronic shop in Hong Kong. The second purchase was made only 10min later – in Shanghai, and a third purchase was made in Sydney a few minutes after that. If you processed each event in isolation you wouldn’t see the bigger picture: irregular purchase patterns. A new (complex) event would be raised: ‘potential fraud detected’ event, and action could be taken in response to the event: cancel credit card and notify card holder.

So that’s it – complex event processing in a nutshell.

Business Driven Mashups: Bottom-Up SOA?

When developing a service oriented architecture bottom-up is generally seen as bad. Top-down is generally better, and the pragmatists among us argue for a middle-out approach. If you’re not careful, the rising popularity of user-developed, browser-based, graphical mashups will force you down the road of bottom-up SOA design. And that is not a road you want to be on.

---

In SOA, the same concept means different things to different people so a few definitions:

• Top down: focuses on analysing the business, in particular where it is and where it wants to go. The SOA is designed to support the business. Close alignment between business and IT.
• Bottom up: focuses on service-enabling existing IT assets, often without regard to the business drivers for doing so. There is no strategic plan. IT builds a huge services layer in the hope that what they build may be useful to someone, at some point.

It may be oversimplifying it a bit but you could say that top-down is seen as ‘business-driven’ SOA and bottom-up is seen as ‘IT-driven’ SOA.

But that is now changing. Mashups are gaining popularity within the ranks of business power users. Services oriented architectures are starting to be seen by some as mashup platforms. Strictly speaking there’s nothing wrong with that. Mashups are one tangible, visible result of the behind-the-scenes plumbing that is SOA.

Here’s an interesting question though:

If: Mashup adoption is largely business-driven (by end-users).
And: The top-down (i.e. business driven) approach to SOA is the preferred way
Does that mean: that mashup-driven SOA design is a good thing?

The answer (as usual) is: it depends.

On the one hand, SOAs are supposed to support the business. If you develop services that can’t be used by mashups then your SOA is not supporting the business. Mashups provided a good ‘validation tool’ for checking that you’ve designed your services in a way that is useful to the business. Mashups are one potential consumer of SOA services, and will become an increasingly important one.

On the other hand, you can’t have each business unit submitting requests to IT to develop services specifically for their mashup, without considering what is going on in the rest of the business. This will result in an explosion of services developed in isolation, with no reuse potential and no reference to an enterprise-wide strategy. This is bottom-up design. And it’s bad. Just because it’s business-driven bottom-up rather than IT-driven bottom-up, doesn’t make it any better.

Mashups make bazaar-style innovation possible, but the old principles still apply. You need to take a big picture view and create a SOA strategy that aligns with the needs of the entire business.

Mashups and SOA: Learn from the Past (Part 2)

In my last post I talked about lessons from the past with respect to business driven mashups. In this post I talk about another thing to keep in mind: the danger of designing solutions from the user interface down.

---

A typical custom built software application might have a 3 tier structure: user interface, business logic and data access. The idea was that you could design, develop and deploy each tier independently. If you wanted to replace the database server then you could change the data access layer, leaving the other two alone. If you wanted different user interfaces (e.g. Windows application, web-based, mobile device) you just had to write a new user interface layer and reuse the business logic and data access layers.

Sounds great. Except it never worked like that in practice. In many organisations the business logic and data access layers were tightly coupled to the user interface. For example, when web-based applications were all the rage most projects started with a statement like “we need to build a web site to…” From the first day of the project the focus was on the user interface tier ” and the other two tiers were designed accordingly. It didn’t have to be done this way ” there are plenty of design patterns to help you do it properly. It was done for the sake of expediency.

The problem with this approach: it limited reuse. If you wanted to re-use the business logic you wrote for a web application to create a Windows based composite application, you often couldn’t do it without code changes ” and often substantial ones.

What has this got to do with mashups and SOA?

Some see the rising popularity of user developed, graphic based mashups (where the mashing is done at the browser) as a means to getting SOA’s funded. If this is your strategy then here’s a word of warning. If you sell SOA as a mashup platform there will be pressure to cut corners and deliver ‘just enough’ SOA to support mashups. In fact, there will be pressure to deliver just enough SOA to deliver the particular mashup the business wants to create at the time (limiting the potential of creating other mashups later on).

But graphical, browser based mashups are only ONE potential consumer of SOA services. A true SOA should be built with the greatest number of consumers in mind, including BPM, non-graphical mashups, composite applications, application integration and so on.

If you design your SOA specifically as a mashup platform then you need to manage expectations and make it clear that opportunities for reuse may be limited.

Mashups and SOA: Learn from the Past

My background is software development. I’ve got “hands on” technical skills – I’m certainly not a Visio/PowerPoint architect. The sheer number of IDE’s, middleware products, database servers, toolkits and various other odds and sods I have installed makes my quad core machine run like a Commodore 64. I’m making a point of this because I’m seriously risking getting an ‘anti technology’ reputation. In my last post I gave my opinion on WOA. This post is about mashups.

---

Let me say at the outset that I love the idea of mashups. I’ve done a lot of experimenting over the last few months and the sheer ease with which you can assemble a pretty useful mashup is staggering. I’m sure that a typical non-technical end-user would find it as easy, but the time is coming.

IT is a funny thing – the same patterns repeat themselves over and over again. And I’m having a feeling of déjà vu when I look at mashups.

Some (most notably mashup product vendors) would have you believe that you don’t need SOA to get started with mashups. And this is certainly true. But if you look back I’m sure you can remember other business-driven initiatives which promised to empower end users. We ended up with a proliferation of Excel spreadsheets, Access databases, user developed applications, dashboards and reports. And all of these were used to run the business and make decisions. That’s all well and good – if it worked. But in reality it was a real mess – on two fronts, both business and IT:

• IT wasted enormous amounts of resources on trying to debug and enhance these flaky, undocumented ‘solutions’ when it all got too hard for the original ‘developer’. You couldn’t just get rid of them because they often became critical to the business.
• The business often got a bit deflated when you said something like “did you know that the ‘customer id’ in the finance system is not the same as the ‘customer id’ in the ‘order management system?” or “you’ve misinterpreted that order status code. What it actually means is…”

Now don’t get me wrong – I’m all for empowering users. But is has to managed.

Enterprise-quality mashups can only be built on top of an enterprise-quality SOA.

WOA and SOA

Not sure how it all started, when or why but WOA is getting a lot of press lately. Everyone has an opinion. For what it’s worth, here’s mine.

---

SOA is hard. We’re increasingly hearing about stalled or failed SOA efforts and proponents of WOA are seeing this as opportunity to get some air time. There are many discussions about the relationship between SOA and WOA, and there seems to be three schools of thought emerging. I don’t think any one of them hold up to scrutiny.

First School of Thought: WOA is a stepping stone to SOA.
Some organisations are a bit nervous about SOA and they want to dip a toe in the water. The right way of dipping a toe in the SOA water is to start small, do a pilot project and then deliver additional benefits iteratively. WOA is the wrong way of ‘dipping a toe’ in the water.

Successfully delivering a WOA solution is great – but you haven’t learned anything about SOA.

At the technical level you’ve learned nothing because the technology is different. For example, learning about REST teaches you nothing about developing SOA with WS-*.

At the business level you’ve learned nothing because WOA is all about technology, whereas SOA takes a broader view – it’s about the business – change management, politics, governance, business processes and the like. You can deliver WOA without having to deal with the ‘hard stuff’.

If WOA is all that you need to solve your particular business problem then go with it. Just don’t expect WOA to magically ‘scale up’ to full blown SOA – it’s not a stepping stone. It is a step – but a step in a different direction.

Second School of Thought: WOA is a fallback position for SOA.
Many organisations have started off down the SOA road. A business case was made, approvals given and cheques written. Everyone is excited about the possibilities. The only problem is: things aren’t going well. What now? Some people see WOA as a way to save face – a fall back position.

The problem with this is that SOA is expensive and it is disruptive. When you go to all the trouble and expense of SOA you’re expected to deliver real business results. If, instead, you deliver some WOA eye-candy then the business won’t be impressed. You cannot match SOA ROI through WOA. Sure, delivering something is better than delivering nothing – but WOA is not a credible fallback position for SOA.

Third School of Thought: WOA is a replacement for SOA.
SOA and WOA are two different solutions to different problems. They’re just tools. You need to pick the right tool for a problem. SOA is more complex, expensive and disruptive than WOA. If WOA solves your business problem then use it – there’s no point using a sledge hammer (SOA) to crack a nut. Conversely, if you need to transform an organisation or create rich solution then don’t try and do it with WOA as you’ll soon outstrip its capabilities. Neither one is a replacement for the other.

The Bottom Line
Any discussions about “SOA vs. WOA” are a complete waste of time. Apples and oranges. You can only compare the two at the technical level. But that’s not where the discussions and comparisons should be.

It is imperative that all architects keep their eye on WOA. But don’t get distracted. Nobody said SOA was going to be easy but if you began the journey for the right reasons it’ll be worth it. Stay the course.