HackIT’s development team have some big ambitions with our API Hub and API Factory (our project which is refactoring of our Hackney APIs into a microservices architecture and delivering future APIs in the most effective and consistent way). The team has been steaming ahead with ideas to make our APIs so good people prefer to use them. But we know that this drive for innovation can’t be done without regular checks against industry standards and best practice. So with that in mind, members of the dev and security team sat with a team from Amazon as the first part of a “Well Architected Review” to sanity check what we were doing and ensure that our approach meets industry standards. Our HackIT manifesto states that we learn more “standing on the shoulders of giants” and Amazon are one of those giants we want to learn from.
We as Hackney team would like to thank AWS team for giving us valuable insights on the tech platform. We are really looking forward to the next session on 17th July.
Once everyone had settled in – with some of the Amazon team (comprised of Katie, Alice, Abir and Mani) very much looking forward to taking part in their first workshop from bean bags, Rashmi set about giving an overview of why are we doing this, where we are currently and where we want to be.
Well Architected Review
We then set about diving into the Well Architected Form which is built in to AWS’ cloud platform – albeit we had to switch our region to Ireland as it’s not available in the London region at the moment – but who doesn’t love Ireland? We began answering a series of questions over various areas such as what our priorities are, how we can understand the state of our application and how we can reduce defects.
It became apparent as we were completing the form that at this point in the process it might be better to have a less structured discussion about what our questions and worries are. We were advised to complete the form at some point in the future but continued the meeting by talking more specifically about what we have in AWS and what we’re trying to achieve.
Amazon estate options
This led to “story time” starting with Alice (solutions architect at Amazon) telling us from her bean-bag that she felt like a teacher at story time. We broke out the white-board and Alice began explaining how many organisations set up their AWS estate (although she was worried as she’d been told that apparently drawing with red pens makes people angry subliminally).
Thankfully the red pen only seemed to engender rather a lot of excited faces amongst the dev team as we thought about the possibilities of building a more secure and easy to manage AWS setup.
We also discussed segregating our current landscape into master account, shared services (such as Active directory federation for example), workspace and multiple developer account model. The multiple developer account model would allow templated “scratchpad” areas for developers to effectively “mess around” and try things out. This would all use a product called Landing zone that would make managing this setup far simpler – providing advice on security, cost optimisation and performance.
Below is the sketch out of how the multiple account model along with Landing zone might look.
The above seems complicated – however this would potentially give us far greater security and a more resilient cloud setup. Specific accounts for specific workloads provides a modularised approach to the cloud infrastructure meaning components can be added and removed with less opportunity for error.
Hackney specific question time
We then moved on to more specific discussions about our platform setup including our use of the API Gateway and how we would manage API Keys, as well as our strategy for migrating and synchronizing data in the cloud.
A lot of time was spent around the current setup within AWS (which consists of a single API Gateway and a single API inside it). Inside that single API all of the rest of our APIs are individual paths. A suggestion for the future was to split each API in to its own API Gateway. This discussion went to and fro for some time with much pointing at screens and hand gestures. Eventually we came to the realisation that we need 1 Gateway and multiple APIs (because this would make access management much easier) but we aren’t sure what that looks like until we’ve had enough time to think it through and design it accordingly.
We also discussed our current ecosystem with the team. One thing we specifically discussed was our current mechanism for logging. They recommended their tool ‘CloudTrail’ however they implied we should be more clear about what we expect from the logs.
We also found out about a tool provided by AWS – ‘Config’. The tool tracks configuration changes and can be customized to include triggers which notify us when a change has occurred. For example, anytime someone changes our root user password, we could set a trigger to get a notification. We will look into the tool as we believe it could increase the security of our platform and applications.
Another key take-away from this session related to a current concern we have around managing access to our APIs and that our current implementation would not scale very well. After much discussion and consultation with our AWS colleagues we came across some really useful ideas that the team were itching to explore where we could really expand on the use of the API Gateway resource.
We discussed how we could we master our current housing dataset on cloud. We spoke about work previously done as a spike. We spoke about the Database Migration Service (DMS) approach we’d taken to extract the schema and data out from our legacy database and described how it had failed due to incompatibility of our legacy system with the tools on offer. The AWS team suggested we send them the details which they could review and provide further advice on it.
Our next steps
The meeting has left us with several areas for research, discussion and food for thought prior to our second conclusive meeting we have scheduled on the 17th of July. To conclude, we have identified several areas for further discussion and review:
- Two new proposals for API access management that we have come up with and that will be reviewed by AWS team.
- The feasibility of replacing our EC2 services (running Docker containers on hosted servers) with Fargate (running Docker containers in a serverless environment) and what would the benefits of that be in terms of performance and costs.
- Possible data migration services provided by AWS to help us get our legacy data into the cloud.