Setting Up Architectural Boundaries

Dependency Inversion Principle, one of the ground principle to set up architectural boundaries

What does it mean to design and implement a generic, reusable component? Is there an objective measurement of how generic/reusable is the code designed? While I am still looking for a satisfactory answer to this question, I have something to share when I am tasked to implement one.

Recently, I was tasked to implement a API communication component that will form the standard communication client between the HTTP internal services of our systems. It has to address several cross-cutting concerns such as API key lookup, timeout, and retry logic. It was not something particularly difficult, but since it’s the team’s first component that is going to be used in a newly developed microservice, it has to be generic enough so that it can serve as a blueprint to future inter-service communication.

A colleague of mine came up with an initial draft. It looks something like this:

From now on, I will use the interface IFooApiClient to denote the generic API client we were supposed to build. It is later renamed to IFooHttpClient to clarify its intent

For this design, users of this component are supposed to call IFooApiClient.SendRequestAsync<TRequest, TResponse>(TRequest, IRequestDescriptor) to send a request to communicate with internal HTTP RESTful services within the company. Every cross-cutting concern is supposed to be handled by FooApiClient when sending a inter-service request.

My main criticism of this design is the leaky abstraction of IFooApiClient, which exposes a Task<TResponse> SendRequestAsync<TRequest, TResponse>(TRequest, IRequestDescriptor) method. On the surface, it looks like a pretty generic method, and this supposedly separates the client interface with the underlying HTTP request sending mechanism. However, this is not the case.

The main problem with this design is the leaky abstraction provided by the interface IFooApiClient. This interface exposes a SendRequestAsync method with an IRequestDescriptor parameter, and one of the member exposed by IRequestDesciptor is a RequestMethod enum. Can you see what is wrong here?

Since the user must provide an IRequestDescriptor parameter to IFooApiClient, and thus must provide a RequestMethod, the interface actually strong couples itself to the HTTP methods GET, PUT, POST, and DELETE, which defeats its purpose of separating the client interface with the underlying transport mechanism! Not to mention the fact that IRequestDescriptor is an extremely unstable interface, when you foresee different services needing a different combination of IRequestDescriptor members (e.g. some might need an ApiKey, some might not) to construct a valid request.

This design also shifts the burden to construct a valid request to the callers of the component as IRequestDescriptor is listed as one of the parameters, which complicates API usage. In fact, every call to the SendRequestAsync method is burded with details that the caller should not even care.

As a proponent to TDD, I always try to write code that maps to the use case first, and establish a proper architectural boundary to isolate the details/mechanism in the underlying implementation. What do I mean by that? Well, first consider the ultimate question we are trying to answer. Quoting from a few paragraphs above:

…implement an API communication component that will form the communication client between the HTTP internal services of our system.

This should actually boils down to a simple interface method.

Task<TResponse> SendRequestAsync<TRequest, TResponse>(TRequest request)

On the surface this looks no different than the one written by my colleague, but there is one big difference:

There is no coupling to the underlying transport mechanism.

Most readers may counter-argue when they read the statement above:

The underlying implementation is already decided to be HTTP, why do you need the decoupling? What benefits does the decoupling bring?

Let me answer the question from two different perspectives, one through the existing design, and one through proper engineering principles.

The existing design actually tries to achieve the same decoupling (albeit unsuccessfully in my opinion) by introducing the IFetcher interface, which exposes a set of methods that is strong coupled to the HTTP verbs, GET, PUT, POST, and DELETE. This means that there is actually no decoupling at all because it already assumes an HTTP verb naming scheme. What if I have a SqlFetcher that implements IFetcher? SQL’s CRUD do not map directly to HTTP verbs! What if I have a FileSystemFetcher? GET, PUT, POST, DELETE does not mean anything to a file system!

As mentioned above, this design also forces the consumer of the component to provide details it should not be concerned about, due to the IRequestDescriptor parameter. All the consumer cares should only be sending a request.

In my opinion, having a good interface method is really a first step to establish a good code/component design. What Task<TResponse> SendRequestAsync<TRequest, TResponse>(TRequest request) does is building a good architectural boundary between a policy (high-level logic regarding inter-service communication) and an implementation detail (low-level protocol to establish that communication)

Instead of building a generic HTTP client bottom-up, I like to work top-down.

From analysing my colleague’s code, I realised each service should not expose HTTP related operation directly, but should expose the actual business operation it is designed to achieve, so that HTTP becomes solely an implementation detail. This allows the decoupling to switch to another mechanism, e.g. file, database, in the future. And even though this switch might not ever happen, this separation is about relieving the burden of having to construct unnecessary details from the clients’ perspective, which makes an API easier to use.

So I worked on an actual use case that requires inter-service communication, and then hid the actual HTTP communication through a well-defined interface. Imagine IBarClient to be responsible for a business use case to create Bar using a CreateBarRequest, and the actual implementation requiring IFooHttpApiClient for HTTP communication. Here is what I came up with.

The difference is that now every service (one of them being Bar) has their own client, conveniently named as I{ServiceName}Client, which is agnostic to the underlying transport mechanism. But at the same time, if they are HTTP services, they can simply inject a closed generic version of IFooHttpClient<>, perhaps FooHttpClient<Baz> with Baz being a second system, and everything will still work correctly, due to the fact that there will be separate instances of HttpClient as FooHttpClient<Bar> is a different type than FooHttpClient<Baz>, even though FooHttpClient<> is being registered as a singleton through dependency injection.

The added benefit is that I can now inject additional members to FooHttpClient to fine tune the HTTP communication without breaking encapsulation because this class is already outside of my architectural boundary for high-level business logic, which means it will not break BarHttpClient.

Compare this with the old design where there isn’t such boundary, business code depending on IFooApiClient will already be locked down to using HTTP and cannot be changed easily. Even if the interface IBarClient and BarHttpClient were included in the original design, it would still couple BarHttpClient with unncessary details, which originates from the members of IRequestDescriptor.

Since policy and details are being decoupled properly, we are now free to inject additional logic that handles cross-cutting logic into FooHttpClient, let’s take adding an additional HTTP header to the request as an example:

You now have good enough separation to apply decorators cleanly to address cross-cutting concerns, such as the AddApiKeyHttpContentFactory class in the example above. This has two main benefits:

  • Clients such as the BarHttpClient class is not concerned with unnecessary details such as how to build a valid request that requires an API key. As long as the decorators are being applied correctly, every BarBaseRequest routed to the HTTP client is going to have its API key provided
  • More decorators can be applied to address additional cross-cutting concerns, such as caching, retry; and any combination of any decorators can be reused for other derived classes for FooBaseRequest, such that FooHttpClient will still work as long as IHttpContentFactory returns a HttpContent

Yes it requires more upfront design work, but having the correct architectural boundaries and mindset can save you tremendous amount of painful debugging and time to extend a component with new features.

I was criticised before for overengineering things while having no convincing supporting argument to defend myself. Over time, however, I have learnt one thing:

Having an architectural goal is different than overengineering

Yes, a problem can always be solved using the least amount of time and effort if you do not care about long term extensibility and maintainability. Having an architectural goal does not mean writing the most generic code that no one is going to use, it is about being pragmatic in solving a problem, while allowing extensibility points to your work.

And that is how I would differentiate between a junior and a senior software engineer.

Programmer | Watch enthusiast