Introduction

In 2020, we built the system for running an energy retail company in Singapore from scratch. When that gradually grew into a succesful operation, I was asked back to lead the project taking that entire solution and customize it so that we could run the same software for Australia, or, in fact, any other market in the future.

In terms of scope, this might have been the largest architectural refactoring that I have been involved with to date, where we continuously had to deal with the tension between market-specific demands and our goal to harmonize as much as possible.

How we handled it

When we started out in Singapore, we knew one thing for sure: the whole system eventually had to become a proper participant in the choreographies defined by the SP Group. Unfortunately, all you have to understand if you are correctly implementing the protocol is a set of documents. So, in order to validate if our solution was correctly implementing the protocols, we created an emulator that behaved as the other market participants, which allowed us if we implemented our own roles correctly.

For Australia, we decided to go with the same approach: created an emulator that emulated the behaviour of various players in the market, and then slowly, over time, adjust the beating heart of our retail system to make sure it behaved according to the needs of the market place.

Also, if we were going to touch the beating heart of our energy retail system, we obviously didn’t want to overload it with if conditions to adjust its behaviour to the local market. We wanted to have a common core, and make sure that customizations were pluggable.

As an example: the Market Emulator

The Market Emulator I created for Singapore was a rules-based system: given a certain input and previous requests, it would reply with certain output. It implemented the SP Groups protocol binding, which is HTTP-based.

So, in line with what we set out to do, I started taking apart the differnt parts of the emulator and generalize them, so that we could compose an Australia based emulator out of the different parts and reuse as much as possible. Even though both Australia and Singapor relied on the exchange of XML messages, the protocol binding was totally different. In Australia, that relies on a custom protocol, involving moving around zip files and ack files over FTP.

So, we separated the machinery for implementing the abstract protocol (a rules engine based on typed JSON documents floating in and out) from the machinery to bind that to a protocol, involving dealing with XML representations, bundling and splitting, zipping and the actual transfer protocol.

By externalizing the protocol binding (the Hokey Pokey library), we could reuse that same logic in a bridge that would implement the bridge between AEMO and Camunda. And we could test that bridge with our emulator as well as the actual test environment they eventually provided us.

Rinse and repeat

The above is an example of just one of the steps we took, but it is exemplary of how we handled the entire process. Take one bit that we knew how to handle. Take our existing code base. Find a way to partition the system in such a way that it would allow us to reuse as much as possible. Implement market-specific plugins and extensions. Move one level up.

One of the things we struggled with for a while is the terminology we would adopt. We did not want it to be Singapore or Australia specific. But if not Singapore- or Australia-specific, what should it be?

In order to postpone that problem for a while, we decided to give each installation of our retailer system a dedicated GraphQL adaptation layer, in which we would be able to adjust our APIs to market-specific terminology. That layer could also implement market-specific business rules and — by the power of federation — integrate other market-specific data sources into the core API.

Team-level harmonization

As you can tell, this whole project introduced some pretty significant architectural challenges. But also some organization challenges. We needed to shift from an organization that assumed everything was done in a certain language, a single timezone and single market into an organization where we all understood that those assumptions no longer held and that changes made for one market might potentially affect the other.

A lot of this was still in progress, but one of the ways in which I tried to address is it is by having a global summit, where we all got together in the same room to discuss the impact of our multi-market strategy on each individual component of our architecture.