Deprecating a million euro message broker

Navajo ESB In this article we will give an historical review of the rise and fall of a large scale sports message broker platform and how it was replaced by a solution that was orders of magnitude simpler. The intended audience for this article is mixed. The start of this article describes the problem and does not assume any in-depth technical knowledge. The paragraph describing the solution contains technical elements and assumes some technical affinity with the reader. The platform was installed in 2004 and finally “de-plugged” by Dexels in 2010.

Download the PDF or continue reading online.

The original case

In 2002 the Dutch Olympic sport federation (NOC*NSF) commissioned a proposal for realizing a message broker infrastructure for integrating existing club- and sport association environments. The goal of this platform was to create a central hub where all data communication between the clubs and their association would be relayed. The goal of this project was three-fold:

To gain nationwide insight on the exact number of athletes for each association.

Enable easy, plug-and-play, integration between clubs and sport associations.

Standardization by defining an XML message standard for the sport community, the so called “Sport-language”

So far an ambitious and achievable plan, but some factors made it more complex. Firstly not all sport information systems for the associations were continuously online, making the architecture of the broker more complex due to the need for asynchronous communication. Secondly the associations also wanted a very high level of security, resulting in an architecture based on a Public Key Infrastructure, with a third-party authority administration. Thirdly the control over the quality of th e data was commissioned to the broker instead of databases of the associations. The focus was entirely on collecting reliable personal data of all athletes in the Netherlands instead of the integration between the club- and the information systems of the associations.

Clubs and associations were forced to communicate with each other through the message broker, but the message broker only supplied interfaces to communicate personal data and very limited association specific data. The initial phase consisted of registering and de-registering a person with a club, update personal (name, address) data and retrieving a list of club members. A total of four possible interaction patterns between club and association.

The project RFP was granted to a number of suppliers, among which a supplier of a full scale message broker platform, a web security company and a company owning a tool to filter duplicates in the collected personal data. The initial phase was realized in one year time, still with cumbersome manual handling due to the fact that 3% of the messages were rejected. It took an another year to correct the message handling and bringing the message rejection back to an acceptable level. Meanwhile, the platform had already cost many man years for initial development The total system infrastructure demanded a total number of 20 servers that could process a mind-bogglingly total of 60 club – association interactions per minute (mainly due to the fair amount of additional data processing for security and duplicate filtering). The rest of the horsepower was required for maintaining the collected information.

After the system was installed and operational, each and every existing and new club- and association platform had to implement a connector to use the Sports Message Broker System (SMBS). This appeared to be a cumbersome process. One should wonder if there was any positive result from this project and certainly we found one. The software solutions for the sport community were wide spread, each association and club developed their own solution. Due to the SMBS project the mindset of associations and clubs evaluated from, separated solutions to more standardization. It is this standardization which nowadays sets the professional level of software solutions for association apart from the rest of Europe. Sportlink Services, an initiative of the Dutch Football Association, the Davinci Consultancy Group and Dexels developed a standard software solution for multi-sport disciplines. These solutions are nowadays used by a large number of associations and clubs, covering almost 50% of the sport community in the Netherlands.

The challenge

In spring 2009, the “members” (the different sport associations) of the NOC*NSF decided to “de-plug” the SMBS system as of december 2009, due to the high cost of ownership and the fact that the technology of the SMBS system was out-dated. A number of club applications however had become dependent on the broker platform. Killing this platform implied that these applications would not be able to communicate with their sport association system anymore. After some debates between the association and NOC*NSF, finally it was decided to replace the SMBS by a new platform, leaving a mere two months of designing/documenting/developing/testing/implementing left to create an alternative for a platform that took years to implement.

The solution

The project “Dismantle Broker” was started. A meeting was set up between the developers of the three different club applications, the sports association system, and a representative of the “affected” sports associations. This was the only face-to-face meeting that was ever held during the project. It was agreed that we, Dexels, would produce developers documentation for the club applications, detailing how they could connect with the Sportlink Sports Association System[1].

We knew in advance that it could be pulled off, mainly due to the fact that the existing Sportlink system was completely set up as a Service Oriented Application. We were confident that this service oriented nature of the Sportlink system would enable a quick and reliable solution, even under such strict time constraints. The Sportlink system already provided around 1800 different web services compared to the mere four of the SMBS.

The Sportlink system itself had already been connected to the SMBS system, but in such a way that the data quality check of the Sporlink system was leading. The SMBS system used a specific XML language, called “Sporttaal” (Eng. : “sport language”), to exchange messages. Before, we had implemented a hub that translated this Sporttaal format to the native XML format that is supported by the Service Oriented “Navajo framework” of the Sportlink system. We decided to expose our native XML format Service Interface directly to the involved parties, so club applications could 'speak' directly to the association. In addition a uniform web service API was defined for the club applications in order for them to receive remote mutations performed by the association or by other clubs.

The Navajo framework is perfectly suited for these kind of interactions by supporting a web service run-time with additional support for service composition and service orchestration. The required web services were implemented before Christmas 2009 and fully operational with two (out of four) club applications starting from January 2010. The last two club applications followed a couple of weeks later as they had only started implementing the required changes in January. As of now, four months later there have not yet been any issues with the new solution and the system happily processes thousands of club – association interactions per minute. The total investment to implement the changes in the Sportlink system was less then four man weeks, with no additional servers needed to process the extra workload the SMBS interaction required.

The other objective of the original SMBS platform, collecting unique personal data about sport participants, was dropped, but can be realized by frequently exporting the member data to a central database and removing duplicates.

In the remainder of this article we will describe the final solution in more technical detail. The remainder can be skipped if you are not a tech- savvy reader.

Service Oriented Work Flows

Member mutations include registration, de-registration and changes. Besides personal- and address data, so called sports activity data and club function data can be mutated. Also, to make matters a bit more complex, a person can be a member of multiple clubs and each of those clubs can make changes to the same person, unaware of the other clubs.

The following conceptual (web) services can be defined for implementing member mutations for both clubs and associations:

Club: member registration
Club: member de-registration
Club: member update
Club: retrieve member list
Association: member registration
Association: member de-registration
Association: member update

The first four services define an interaction initiated by the club application. The last three define interactions that are initiated by the association application: Sportlink.

Navajo, being a web service run-time, supports flexible deployment of web services. Additional services can be deployed to the run-time without any required down-time for the application.

Furthermore, each Navajo web service is uniquely identified by a name which is typically related to its function. The following web services are made available for club application to communicate with Sportlink:

external/ProcessExternalInsertMember
external/ProcessExternalUpdateMember
external/ProcessExternalDeregisterMember
external/ProcessExternalQueryMembers

The Navajo framework uses an internal XML language called TML[2]. TML messages both contain type information as well as data. Messages are exchanged using the TML format. Invoking a web service involves two TML messages, a request- and a response message.

For member registration and -updating a fragment of the TML request message is shown below:

<message name="MemberData">
    <!-- the club Id property -->
    <property name="ClubIdentifier" direction="in" type="string"/>
    <!-- the person properties -->
    <property name="PersonId" direction="in" type="string"/>
    <property name="LastName" direction="in" type="string"/>
    <property name="Infix" direction="in" type="string"/>
    <property name="FirstInitials" direction="in" type="string"/>
    <property name="FirstName" direction="in" type="string"/>
    <property name="BirthDate" direction="in" type="date"/>
    <property name="BirthPlace" direction="in" type="string"/>
    <property name="Gender" direction="in" type="string"/>
</message>

The TML fragment above is used as a request message for both the external/ProcessExternalInsertMember and the external/ProcessExternalUpdateMember web services.

From the perspective of an association the situation is little bit more complicated. The association uses its own client application, which is a Java Swing desktop application. The association application again uses web services to communicate with the Sportlink system. However, in this case it is the communication between the association application and Sportlink that should be intercepted to check for possible interesting messages for a club. The Navajo framework offers support for aspect oriented service orchestrations called “service work flows” that enable exactly this kind of behavior. A Navajo work flow is basically an Abstract State Machine[3] that contains states and state transitions. A state transitions brings a work flow from state A to state B. Within a state, a work flow can perform useful tasks, like calling web services(!). Again, a state transition itself is triggered by the invocation of a web service. Hence, a work flow can be instantiated by a web service invocation. Basically, work flows offer a perfect loosely coupled (aspect oriented) methodology to interact with web services.

Introducing the work flows

Let us define a work flow that instantiates whenever a web service member/ProcessDeregisterMember is invoked. To be precise, let the work flow instantiate after the web service member/ProcessDeregisterMember is invoked, hence after the web service has succesfully terminated. The subsequent state, deregistermember, should invoke the web service external/clubapps/ProcessSendMutation which in turn calls a web service for all clubs for which the processed member has a club membership. The next transition can be performed immediately and brings the work flow to the finished state.

A work flow is defined using a work flow XML definition file:

<workflow>
    <state id="init">
        <transition trigger="navajo:member/ProcessDeregisterMember" nextstate="deregistermember"/>
    </state>
    <state id=”deregistermember”>
        <task service="external/clubapps/ProcessSendMutation"/>
        <transition trigger="immediate" nextstate="null"/>
    </state>
</workflow>

Each work flow state is defined between <state> tags. Each state has a unique id defined in the “id” attribute of the state tag. The init state is a special “bootstrap” state, which is the only state that can create new work flow instances.

A transition tag defines a trigger, a nextstate and optionally a condition. The condition may address data in either the request- or response XML of the invoked web service. A task tag is used to specify calling a web service within a certain state. Note the immediate trigger and the special null state to define an 'immediate' transition to the ready or finished state of the work flow instance.

Implementing a member mutation work flow

Since the entire Sportlink application is service oriented, we can fully identify a list of web services that should trigger a web service call to external/clubapps/ProcessSendMutation.

member/ProcessUpdateKernMember : updating a member
member/ProcessDeletePlayer : deleting a sports activity of a member
member/ProcessInsertPlayer : adding a sports activity of a member
member/ProcessInsertClubAdminFunction : inserting a club function of a member
member/ProcessDeleteClubAdminFunction : deleting a club function of a member
member/ProcessUpdateClubAdminFunction : updating a club function of a member
member/ProcessInsertClubMembership : adding a club membership
member/ProcessDeregisterMember : de-registering a club membership

For each of these web services we can identify whether a member-registration, -de-registration or -change is in order for a specific club. We can group each of these web services as “deregister member”, “register member” and “update member”.

Deregister member

member/ProcessDeregisterMember; deregisters a member for a specific club, a de-registration message must be sent to the involved club.

member/ProcessUpdateKernMember; in case a diseased date is set for a member, a de-registration message must be sent to all involved clubs.

Register member

member/ProcessInsertClubMembership; the club for which the registration was made needs to be sent a registration message.

Update member

member/ProcessUpdateKernMember; any change to personal data or address data (except for diseased dates) need to trigger a changed member message to all involved clubs.

member/ProcessDeletePlayer, member/ProcessInsertPlayer; if a sports activity is added or deleted to a member the clubs should be informed.

member/ProcessInsertClubAdminFunction, member/ProcessDeleteClubAdminFunction, member/ProcessUpdateClubAdminFunction; any mutation to a club function will result into a changed member message to all involved clubs.

Each club application must provide a mutation web service for receiving messages from Sportlink that in turn is called by the Sportlink web service external/clubapps/ProcessSendMutation.

The request for the club mutation web service will be a member message (see previously shown TML fragment) together with the required operation (deregister, register or update) to every club for which the member has a club membership.

The workflow

To wrap up, the entire member mutation work flow definition is defined as:

<?xml version="1.0"?>
<workflow>
    <state id="init">
        <transition trigger="navajo:member/ProcessUpdateKernMember" navajo="request" condition="!?[/MemberData/MutationIdentifier] AND ( (?[/MemberData/DateOfPassing] AND [/MemberData/DateOfPassing] != null) OR [/MemberData/DeregistrationDate] != null )" nextstate="deregister"/>
        <transition trigger="navajo:member/ProcessUpdateKernMember" navajo="request" condition="!?[/MemberData/MutationIdentifier] AND ( (!?[/MemberData/DateOfPassing] OR [/MemberData/DateOfPassing] == null) AND [/MemberData/DeregistrationDate] == null )" nextstate="changemember"/>
        <transition trigger="navajo:member/ProcessDeletePlayer" navajo="request" condition="!?[/MemberData/MutationIdentifier]" nextstate="changemember"/>
        <transition trigger="navajo:member/ProcessInsertPlayer" navajo="request" condition="!?[/MemberData/MutationIdentifier]" nextstate="changemember"/>
        <transition trigger="navajo:member/ProcessDeregisterMember" nextstate="afmeldenlid"/>
        <transition trigger="navajo:member/ProcessInsertClubMembership" navajo="request" condition="!?[/MemberData/MutationIdentifier]" nextstate="register"/>
        <transition trigger="navajo:member/ProcessInsertClubAdminFunction" nextstate="changemember"/>
        <transition trigger="navajo:member/ProcessDeleteClubAdminFunction" nextstate="changemember"/>
        <transition trigger="navajo:member/ProcessUpdateClubAdminFunction" nextstate="changemember"/>
    </state>
    <state id="deregister">
        <transition trigger="immediate" nextstate="post">
            <param name="Operation">
                <expression value="'DeregisterMember'"/>
            </param>
        </transition>
    </state>
    <state id="changemember">
        <transition trigger="immediate" nextstate="post">
            <param name="Operation">
                <expression value="'ChangeMember'"/>
            </param>
        </transition>
    </state>
    <state id="register">
        <transition trigger="immediate" nextstate="post">
            <param name="Operation">
                <expression value="'RegisterMember'"/>
            </param>
        </transition>
    </state>
    <state id="post">
        <task service="external/clubapps/ProcessSendMutation" navajo="post:request"/>
        <transition trigger="immediate" nextstate="null"/>
    </state>
</workflow>

The additional attribute, condition, that can be found in some transition tags is used for conditionally specifying transitions using original web service request- or response message properties. Furthermore, the attribute navajo specifies whether the request- or response message should be used to evaluate the condition.

The only actual work that has to be done on our side is coding the work flow definition shown above and implementing the web service external/clubapps/ProcessSendMutation. The implementation of the web service is easy: simply loop over all club memberships, find the correct binding address for a specific club application and call a club mutation web service with the MemberData message. The work for each club applications includes: calling a Sportlink web service for relevant member mutations and implementing a mutation web service that can be invoked by the Sportlink association application.