Squish Studio - Fogo Island, Newfoundland, Canada Leica SEM 1:3.4/21mm
I was having a beer with a good friend of mine about a year ago trying to get my head around a problem I was running into while designing a new system for a job at a healthcare startup. I knew that we would only have a limited number of clients (initially in the tens) and each client would be interacting with our system on a daily basis. We would be receiving a lot of sensitive data from our clients which had to be encrypted on the wire as well as at rest. Our clients could vary greatly in size and hence would be very different in the amount of resources they would consume. I also wanted to keep things somewhat separate from each other, especially client data, to try to reduce the risk associated with data getting associated with the wrong client.
I had this idea that I could build an event driven, service based architecture across multiple AWS accounts and it was my friend’s job to try to talk me out of it. The idea was to break up all of the services I would need to build so that they could run independently from its own AWS account and then providing an AWS account for each client that would provide access to all of the services they subscribe to as well as be the location for all of their data.
Here are some of the benefits of this architecture in no particular order:
client data is kept separate from other clients which good from a security perspective - no way for other clients to access each other’s data
billing is easier as each account would have its own AWS bill - makes it easier to see who is using resources
don’t need to have all of our clients in the same region
can have some clients multi region while others are not - this is a big deal if you have to mirror a lot of data as you only have to mirror the data for a particular client as opposed to all of them
assuming the accounts are built and managed with cloudformation, it should be trivial to create a client environment
don’t have to write database code to manage clients, just set up an instance of cognito for each client and configure it to meet the needs of each client
outages due to software issues would only affect one client, outages in a region would only affect clients in that region (unless they have a multi-region configuration)
removing a client from our system and deleting all of their data (a requirement under HIPAA) is trivial, just ask AWS to delete the account.
That’s a lot of pluses except for one big minus:
how do you support/manage/maintain this beast?
Fair question. It may not be as hard as you think though, especially if you are starting from scratch.
CI/CD Pipelines
There is no way anyone is going to want to sit around deploying code to each server account. I like my team to work with a very fast cadence which can mean multiple deployments a day. There is no possible way we can manage a bunch of separate client accounts if we don’t have CI/CD pipeline set up to run automated tests, but more importantly, deploy changes to all of the client accounts. Of course, since each client has their own account, it should be easy for use to set up test accounts with fully automated integration testing which can be performed before the release is actually pushed out.
Consistency
Possibly one of the trickiest tasks is build the system so that each client has the exact same software. There is really no easy way to build in customizations for a particular client unless they are purely cosmetic type things which should be easy enough to handle. As soon as you start trying to install different software on each client account though you are going to wind up with a bunch of systems to support that do not all work quite the same way.
Building solid APIs and SDKs makes it easier to keep things consistent - just make sure you never change the behavior of your API or SDK. You can add features but if your API starts returning stuff in a different way you’ll wind up with trying to simultaneous deploys so that nothing breaks.
AWS Organization Accounts
Unfortunately, one bigger problem with using accounts this way is that there is no easy way to just delete an account and start over again. So if you want to rebuild an account from scratch you either have to completely clean it out or have a bunch of deleted accounts sitting around in your AWS Organization dashboard - maybe this is not that big of a deal, just keep in mind that you cannot reuse email addresses when you delete and create a new account. You can use a + sign in the email address though (if your email provider supports it) so your account names might look something like qa+20200516@mydomain.com - not the end of the world