November 28, 2009

Service registry (ESB) for scalable web applications.

This blog post is the result of my futile attempts at understanding how others have solved the problem of automatic service discovery.

How do organizations, which have a huge collection of custom applications, design scalable web application without having to hardcode server names and port numbers in the configuration file ?

I believe the terminology I’m hinting at is either called a “Service Registry” or a “Enterprise Service Bus” which is part of the whole SOA (Service oriented architecture) world.

The organization I work for, has a limited multicast based service announcement/discovery infrastructure, but not widely used across all the applications. In addition to the fact that multicast routing can become complicated (ACL management of yet another set of network addresses), its also not a solution where parts of applications could reside on the cloud. Amazon’s EC2, for instance, doesn’t allow multicast traffic between its hosts.

Microsoft Azure .Net services (part of the Azure platform) provides a service registry (or proxy) to which internal and external applications can connect to, to provide and consume services. The design doesn’t allow direct connection between provider and consumer which makes this a massive single point of failure. I agree any kind of registry has this problem, but the fact that you need this service to be up all the time to make every single request makes it extremely risky.

There are at least two open source projects by Apache foundation which touch this topic. One is Service Mix and the other is Synapse. I’ve also spoken to a few commercial entities who do this, and wasn’t really convinced that its worth spending big bucks for this.

The reason why I’m puzzled is that I don’t see a single open source project being widely used in this area. I’ve been a REST kind of a guy and hate the whole SOAP world… and was hoping there would be something simple which could be setup and used without pulling my hair off.

My contacts at some of the larger organizations seem to make me believe that they all use proprietary solutions for their infrastructure.  Is this really such a complex problem ?

If you have an ESB in your network, please do drop in a line about it.