Learn with Sphere: Modern Backend: The Universal Pipeline

My name is Vladimir Kalinkin and I’m a Multi-Stack Software Developer at Sphere Partners. In this article, I’m going to describe an innovative approach to building modern backend applications using the best practices of software design.

Excluding non-popular frameworks and technologies, the last 15 years was an era of MVC. It doesn’t matter which language you use on backend side – Ruby, Java, C#, JavaScript or even Go – the primary concept of code organization was Model View Controller.

How we usually develop services

The process generally consists of 4 steps software developers are very familiar with. 

1. Generate app from skeleton
2. Write models, controllers, message handlers, views
3. Write tests
4. Fix bugs

Firstly, we generate an application from a skeleton. Secondly, we write models, controllers, message handlers, views. Then, we write tests to ensure that the application behaves as expected. At this stage, we might encounter unexpected results which leads us to our final step. Lastly, we fix bugs.

How we should develop services

The way we normally develop services – is it the right approach? Perhaps there is a more modern approach I’d like to talk about here. 

1. Write pure domain model in TDD (Test-driven development)
2. Generate application from skeleton 
3. Attach domain model to the application 
4. Write controllers, views
5. Write integration tests
6. Fix bugs

For the last 15 years, we have said this is impossible due to…

Delivery Mechanism 

Let’s forget about MVC as an approach and any frameworks we used so far. Let’s turn to theoretical ideas. One of the best principles I learned ever – “The delivery mechanism is an annoying detail” authored by Robert Martin (Uncle Bob). 

This fantastic idea is used by many teams in the world. The issue is, as a certain theory, it is always incorrectly interpreted. Even the author tries to adapt the definition to existing solutions. Let’s take onboard only the key meaning. “The model is the heart of a product, and the delivery mechanism is an annoying detail”. Here are some examples of a delivery mechanism. 

1. Web application
2. Web-socket application 
3. Message consuming application

What do we mean when we say “model” in this instance? 

When I refer to a model here, I mean more than just a set of entities with basic validations. I am talking about a rich domain model. Below you can see a typical backend application structure:

Perhaps we could unify the flow for two variations of delivery mechanism. Let’s compare the flows. These two flows as shown below look very alike and can be unified.  

What do we need to unify the flows? Let’s begin with controllers and handlers. We always decompose controllers into repositories, command, helper etc – and it’s the same for “handler”. Perhaps we have too much noise that does not matter much to a domain model. Why not use “Helpers”? 

• They aggregate various logic against S.O.L.I.D. principles
• They have inculcate procedural programming style: we have many “static” classes with a number of procedures
• It is always difficult to find an appropriate name which eventually does not reflect clear meaning, i.e. “Controller”, “Handler”

Let’s use “Action”. Below you can see how to decompose “Controller” to actions. 

In a simplified approximation, the typical cycle of executing https request should look like this:

… -> Rack -> Router -> Action -> Result -> …

The obvious weak point in the chain is “Action”, because it turns out that you need to load a state of the model, perform an action on it and save its state, generate and send a response. “Action” is essentially the same as a controller, but it doesn’t include a bunch of operations for different routes — it’s just a single action. Ideally, I would like to divide this chain into separate steps for the convenience of testing each of them. What about the chain?

-> Rack -> Router -> Load Model -> Handle Model -> Save Model -> Result ->?

Let’s try to imagine a mechanism which will take a responsibility to build chains from a configuration or specific convention:

I’d like to have a mechanism demonstrated on this schema. Here you can see some real actions and items responsible for accepting messages from a web container or a queue system like Kafka and a specific Dependency State which is used for passing parameters between actions. We need to break the relation between flow and a programming language. Maybe this is not so clear right now because we have another recursive step – we need to define a DSL (Domain Specific Language) for the routing.

Below you can see such a DSL which defines a schema how we handle incoming messages and chain the actions. 

Here we can completely replace a necessity to write chaining and routing scenarios in Ruby. We invented a DSL for that! There’s only one big question: what is the Dependency State and how does it work? It is the power of dependency injection. The schema demonstrates the internals of web request flow.

1. In the beginning we have an Inversion of Control container like dry-container or any other.
2. We handle parameters from the web-app container and put them to the IoC container.
3. We match a route.
4. We execute ‘action’ chain according to instructions from the routing definition.
5. During the execution we automatically create instances of action classes using Dependency Injection. The mechanism automatically resolves any action dependencies and gets them from IoC state. A simple example of dependency is a technical service or HTTP parameters.
6. Any action can create a result in the Dependency State. For example, current_user after action Authorize is required by further actions.
7. [ViewBuilder] An implementation of delivery mechanism can have additional stuff for formatting of the chain execution result.

Example Implementation

Described idea is reflected in a framework Dandy https://github.com/cylon-v/dandy. Dandy is just a router and the chained actions are coordinated by the DI-container Hypo https://github.com/cylon-v/hypo.

Conclusion

The idea I have described here is an early bird of the future of backend development. In the last 15 years, we sought better ideas of backend development: MVC in Ruby on Rails implementation, DSL, Dependency Injection from .NET/Java worlds, functional approach from JavaScript, “pipes” from Go, etc. Now, it is a good time to make the software development process more efficient – the solution I demonstrated is a good example of how to do that.

Interested in innovative software development solutions for your business? Contact us today about your project.