about software programming techniques

JJ van Zon 2023

🌍 Service Oriented Architecture

back

Service Oriented Architecture is about linking systems together. It focuses on software integration.

Contents

The ESB Concept

The term ESB stands for Enterprise Service Bus. It is a pattern for exchanging data between different systems of different organizations in different formats with different protocols. Central components are used, to make integration between these systems more streamlined. One relevant concept is the Canonical model.

Canonical Model

A Canonical model helps us exchange data between systems. Data can be retrieved from multiple systems and converted to a Canonical form, so that the same model can be reused for data, that comes from various systems. The aim for the Canonical model is to be as pure and general as possible, so information of different systems can indeed fit into it.

Less Integration Code

Say you have 4 systems: A, B, C and D and you want to connect all 4 of them together. Theoretically you would have to write 12 different message conversions. See all the arrows in this diagram:

By connecting systems to an ESB, instead of to each other, you would implement only 8 different message conversions. The number of arrows is reduced:

You just saved yourself 33% of the work!

With every added system it gets better. You can see this from the numbers below, indicating the amount of message conversions:

The first integration between 2 systems you actually program more message conversions by using an ESB. But wait: the next system it’s already a tie between ESB and no ESB. And the 4th integration you introduce, you will have saved 33% of the work overall.

It gets better with each system you add to your ESB. When messages from one system are converted to and from a Canonical model, you can automatically connect it to all the other systems.

Clear Integration Code

But it gets better. You save yourself even more work. The code to convert a message to Canonical model, is often easier than converting from one system’s format directly to another system’s format. Instead of converting from one quirky format to another quirky format, which is quite difficult to do, you can convert from one quirky format to a more straightforward format, which is quite a lot easier to program.

In Practice

In practice not every system sends every type of message back and forth. And sometimes the messaging isn’t bidirectional but one-way only. But the benefits of an ESB still hold and systems would be linked with less code and less effort than custom programming every integration.

Memory

An added benefit to the Canonical model, is that it it tends to live in memory. This means that changing it doesn’t require any data migrations. You would only refactor the conversion code. That makes it lower impact and more flexible.

Standard ESB vs Custom ESB

Standard ESB’s can be complex and difficult to learn, requiring specialized training and expertise. But we could also build a custom one. Either way, we would have to program the message conversions anyhow, and design a Canonical model too, which is basically all of the work. The concepts might be easier to implement than you think. Building a custom ESB yourself may certainly be an option.

ESB Model

On top of a Canonical model, we might need more facilities. The ESB could offer a model for administrating Connection settings and registering Enterprises that can log in to our system to get access to our services.

Next: The main entities of the ESB model:

Enterprises

Enterprises participating in this service architecture would be registered in the ESB database. Those who need to log in, will get a User entity with encrypted credentials.

ConnectionTypes

All types of Connections that can be established between systems can be found in the ConnectionTypes table. Each ConnectionType is meant to be a very specific way of integrating with a system, with a specific messaging protocol, message format and implementation.

Connections

Every individual Connection between two Enterprises would be registered in the Connection table with the Connection settings stored with it. Each Connection has an associated ConnectionType to indicate what type of integration it is.

Note that some Connections might not be between Enterprises, but involve only one Enterprise. Not all Connections need to be full-fledged messaging implementations. Sometimes they are simply a database connection or even the path to a network folder.

KeyMappings

Different systems might handle similar sorts of data, like Orders and Customers. However, they are likely to use different Identifiers. To facilitate communication between these systems, it may be necessary to map these Identifiers to each other. An ESB model can have entities and logic to manage those kinds of ReferenceNumbers, which might also be referred to as KeyMappings.

Transmissions

It may be a good idea to log the messages that are transferred over the Connections. This can be helpful for troubleshooting and debugging. But keep in mind, that it also has an impact on performance and storage, so perhaps use this feature with care.

Integrations

The implementation of a service would involve message transformation and transmission. Data is received through some communication protocol, the message format is parsed and then converted to a Canonical model. After that, the Canonical model is converted to another message format and sent over another communication protocol.

Multi-Dispatch

The contents of a Canonical model can determine where it needs to be sent. For example, an Order may indicate a specific Supplier that it should be sent to. One Supplier might use their own unique integration protocol, while another might prefer to receive the Order by email. With this service architecture you can retrieve a message from one system, for instance an Order, and then forward it to an arbitrary other system. That is part of the power of a Canonical model. It enables communication between multiple systems by converting their messages to a common format.

IsSupported

A service environment may hold the same interface for accessing multiple systems. But not every system is able to support the same features. You could solve this by creating lots of different interfaces. But that can make it confusing to know which interface to use. As an alternative, you could use IsSupported booleans in the interface. That makes it possible for an implementation to communicate back if it supports a feature or not:

void PlaceOrder(Order order);
bool PlaceOrderIsSupported { get; }

IList<Product> GetProducts();
bool GetProductsIsSupported { get; }

Then when for instance running price updates, you can simply skip the systems that do not support it. Possibly a different mechanism is used for keeping prices up-to-date, possibly there is another reason why price updates are irrelevant. It does not matter. The IsSupported properties help us keep complexity at bay.

Tag Model

The Canonical model should focus on data, that plays a logical role in your company. But another system may need data that is not relevant to you. To avoid cluttering the Canonical model with unnecessary structure, you could choose to add Tag models. You might use those Tags in domain models too, to add data, that does not apply to your business processes. But this data can still be sent along to another system when it needs it.

Here are some examples of Tag models in pseudo-code:

Order { Tags[] }
Tag { Name, Value }

You might also make the Tags culture-specific:

Tag { Name, Value, CultureName }

Or choose to loosely link Tags to entities, like so:

Tag { Name, Value, EntityTypeName, EntityID }

Canonical KeyMapping

KeyMapping is an idea that maps ReferenceNumbers from one system to another. For example, the same Order could have a different OrderNumber depending on which system or party it is sent to.

If the amount of systems becomes larger, the amount of KeyMappings can go up exponentially.

You might get many IDs in your model:

Order
{
    InternalID,
    CustomerOrderNumber,
    SupplierOrderNumber,
    ManufacturerOrderNumber,
    IntermediaryOrderNumber
}

And the jeopardy of getting many KeyMappings in the ESB database:

KeyA <=> KeyB
KeyA <=> KeyC
KeyA <=> KeyD
KeyB <=> KeyC
KeyB <=> KeyD
KeyC <=> KeyD

This can become difficult to manage. You could make it a bit more generic like this:

Order
{
    Identifiers[] { System, Number }
}

So it becomes an array of Identifiers for different Systems.

But there’s a trick, that requires only 2 key fields in your Canonical models, and no more!

Order
{
    InternalID
    ExternalID
}

What you could do is map ExternalIDs from one system to InternalIDs in the ESB, which can then be mapped to an ID from yet again another system:

{ SystemA, ExternalID } => InternalID
{ SystemB, ExternalID } => InternalID
{ SystemC, ExternalID } => InternalID 
{ SystemD, ExternalID } => InternalID

This way, when a new system is added, only one KeyMapping is needed, to map with all the other systems.

As messages are sent back and forth between systems, the keys in the Canonical model are translated from ExternalID to InternalID. Then, the ExternalID property is overwritten by the ID from the target system.

It all depends on the specific design of your system. But hopefully this demonstrated a few options how to handle KeyMappings, IDs and reference numbers in a Service Oriented Architecture.

Facade

A Facade is a class or interface that sits in front of other classes and interfaces. Its goal is to provide an easier way to access a more complex system.

This concept is used in this architecture to give a service an even simpler interface than the underlying business. It may hide interactions with multiple systems and hide infrastructural setup.

Hidden Infrastructure

When it comes to handling infrastructure setup, there’s a key difference between the application architecture and this service oriented architecture

In the application architecture, the top-level project was responsible for determining the infrastructural context and passing it down to the lower layers, for instance as interfaces on security and data access.

But the service architecture determines the infrastructural context in the bottom-level projects. At least in the case of multi-dispatch this seems necessary. For instance, a bottom-level project like JJ.Services.Ordering.Email would not reveal that there is an SMTP client under the hood. You cannot see that setup from the constructor or the interface. The services would handle that internally.

Namespaces

These namespaces use a hypothetical Ordering system. The main layers can be recognized there, like Data, Business and Services.

   
JJ.Services Root namespace for the (web) services.
JJ.LocalServices Root namespace for Windows services. (Not part of this service architecture, but this is where that other type of service goes.)
JJ.Data.Canonical Where the Canonical entities are modeled.
JJ.Data.Esb Models for Enterprises, Users, ConnectionTypes, Connections, etc. Basically, the configuration settings of this architecture.
JJ.Data.Esb.NHibernate Stores the Esb model using NHibernate.
JJ.Data.Esb.SqlClient SQL queries for working with the Esb database.
JJ.Business.Canonical Shared logic that operates on Canonical models.
JJ.Business.Esb Business logic for managing the Esb model.
JJ.Services.Ordering.Interface Defines interfaces (the C# kind) that abstract the message communication between different Ordering systems, providing guidelines for the message exchange.
JJ.Services.Ordering.Dispatcher Makes sure messages (Orders, Price updates) are received from and sent to the right system depending on message content, settings and other logic.
JJ.Services.Ordering.Email A specific implementation of an Ordering system, in which we send the Order by email.
JJ.Services.Ordering.AwesomeProtocol Implementation of an Ordering interface, behind which we use a hypothetical AwesomeProtocol.
JJ.Services.Ordering.Wcf A WCF service that allows you to communicate with the multi-dispatch Ordering system.
JJ.Services.Ordering.Wcf.Interface Defines the interface of the WCF service. This interface can be used both by server and client.
JJ.Services.Ordering.Wcf.Client Allows a connection to the WCF service using a convenient, strongly typed interface.
JJ.Services.Ordering.JsonRest Exposes the multi-dispatch Ordering service using the Json / Rest protocols.
JJ.Services.Ordering.WebApi There is no reason Web API should not be involved in this service architecture. In fact, the idea of WCF being the default for services, might not be a very long-lived.
JJ.Presentation.Shop.AppService.Wcf A special kind of service in this architecture is an AppService. It exposes presentation logic instead of business logic by returning ViewModels.

back