Platform APIs weeknotes, 8 July 2020

This past week has definitely been influenced by our refreshed roadmap, and has brought new challenges for the team to tackle. They’ve responded with new energy; the immediate goal is in sight. 

Work to finish the Housing Resident Information API has been a bit more complex than expected, but we’ve gotten there at last. As that was shaping up, Emma and Mirela turned attention to Cautionary Alerts, which I mentioned last week. These are part of the housing database and alert staff to any health & safety issues associated with a council-owned property, such as “dangerous animals” or “interpreter required”. The API returns no personal information, only the indicator. This information isn’t currently available to Single View users, so it’ll be really valuable to include it. 

Work of course continues on the high-level Resident Information platform API, though at a slightly slower pace. We’ve prioritised the LIST endpoint, which will return information for a search and this first iteration will provide Single View with much of the data required in a consolidated view. There is, of course, more to do here – isn’t there always? But it will be usable for the proof of concept. 

One thing that was unsuccessful – or a success, depending on your view – was the work Matt led into writing back to the source databases. The short version of events is that it’s too hard, so we’re not going to do it right now. This is perfectly fine, and in line with agile principles – demystify the unknowns, don’t waste time going down rabbit holes. He goes into more detail in our Show & Tell, available on YouTube. Do have a watch, we’d appreciate your feedback. 

I’m off for a couple of weeks so there will be no weeknotes. As a preview of what’s coming up:

  • We’ll finish the work on Cautionary Alerts
  • We’ll test and deploy all existing APIs to Production
  • We’ll integrate with Single View
  • We’ll offer the Resident Vulnerability API to the “I Need Help” service, being iterated by that team
  • And we’ll do a Tenancy Information API which can be used by Manage a Tenancy and other services

I expect great things on my return!

Why getting the right name for your APIs is so important

Series 1- Chapter 2

To continue from our previous blog post about our journey of defining platform & service APIs and what we have learned from it, we knew that the next step in this journey without a doubt should be about data taxonomy.

Organizations are often locked down with legacy,mid-age, clunky, etc (;-)) databases. We have the challenge of tackling cases where our on-premises databases don’t have any basic database principles applied. In other cases, we found that people are still reluctant to bid goodbye to their tightly coupled processes and/or are scared of any process changes that might impact them. These databases are typically a developers’ nightmare especially when it comes to integration. I have also been in situations where  we would like to open up the data from the source database and make it more available (of course in a secure manner). So when we’re building an API to unlock data from a legacy application or business process our challenge is: ‘What do we call that API so that it’s clear what data it presents without assuming how it should be used?’

So this is why data taxonomy is so important to the design of our Platform APIs 

Data Taxonomy is a process of classifying your data and building a hierarchical structure which becomes the foundation for the API. And then, across our APIs, the taxonomy helps us explain the relationship between different data models.

We started our journey with identifying our core entities and tried to model different data domains around those core entities. Believe me, the workshop was well executed and we ended up with a nicely drawn spider web of all domains which are related to our core entities – People and Properties. This step was really important to build our landscape of Platform APIs and understand how the usage works between different services as we did not want to have our fingers burnt again. Also, it helped us to identify the development of future platform APIs as well.

We have picked up “People” as the core entity. But in a local government context, that means different things depending on the context. For example, the housing tenant might be different from the council tax payer and the holder of the parking permit.So we then refined that further as Resident Information API (we’ve decided that contact details should sit as a separate layer). I will be honest here, initially I thought this would be easy but slowly reality kicked in. In our discovery session, we realized we have 26 data sources and maybe more that store contact information about residents. The team was gobsmacked.. We immediately asked the question of how would we cope with this mammoth task? The team knew that this was a task that will involve a lot of iterations and improvements, so we decided to start with baby steps and learn from them. 

We started with 6 initial datasets which we thought would be a good starting point –

  •  Adults and Children social care data, 
  • Housing data set,
  • Housing benefits dataset, 
  • LLPG data (holding the address information) , 
  • Flags we’d identified that might indicate someone was vulnerable (eg living alone)  
  • Data about people who asked for help during COVID.

The use case that followed was to provide a platform API that retrieves data from individual lines of business applications APIs in order to provide a consolidated view of all the data we hold about a given person across multiple data sources. We are also considering that as part of future iteration we should tackle storing audit logs of any changes made to the data. 

Hackney has a track record of tackling this issue. Our Citizen Index is more than 10 years old and enables us to match people’s records across business applications and verify their identity. So next, we will build on our understanding of this to determine whether or how we create linkages between different records of people. 

Having a data taxonomy designed for our APIs helps us to think in the right direction in terms of data relationships and building a knowledge model around different types of data domain. In other words, the taxonomy for a given domain should be the foundation step for any API development. Having an idea of the relationship between those domains will help you when designing your microservice architecture, simplifying development and integration. 

What’s Next?

  1. Building the SwaggerHub definitions and best practices
  2. Publishing our domain structure
  3. Securing our data
  4. Data Migration Mechanism

Appendix

Data classification is the process of organizing the dataset into relevant categories which helps efficient data management, improved data security, and streamlined compliance audit when it comes to sensitive data.

Unleashing the data beast: Our Journey so far…

What do we aim to achieve via this blog?

This blog is designed for HackIT and external organisations to evaluate our API-first strategy towards data domain modeling and how we approach our data needs! It will also give us the confidence to explore further in terms of architecture and data taxonomy. We understand the importance of reusability across service needs and designing the core architecture in such a way that it doesn’t add up more technical debt for the future.

Why…How….What…???

  • Why are we building Platform APIs and what value ,in terms of return of investment,will they bring?
  • And, perhaps the biggest question, what are they?

We all wish to have an advanced ecosystem that is led by data and I am sure it is a dream for all to have a scalable, robust, secure and resilient platform to achieve it. Whenever we build any service, these are questions that come to mind:

  1. Is the service highly available and reliable and how secure is our data?
  2. How do we scale to meet the demand in traffic from our applications and ensure they’re reliable?

But most importantly we would like to ensure that we follow these principles:

  1. We want to help residents and solve their problems via technology.
  2. Try not to introduce more problems than what we have currently!
  3. If the problems are huge, trying to find simple solutions is our first priority.

Software engineers follow many principles and they change over time but two in particular have always resonated. Typically within the world of IT they take the form of acronyms.

Firstly “DRY” which stands for “Don’t repeat yourself”. At the micro level that means you shouldn’t have the same line(s) of code doing the same thing more than once within your code base. For example, if you connect to a database then do it ONCE and make it a reusable function. This makes code easier to change, test and maintain as well as reducing inconsistency. At the macro level this still holds true. You do not want the same thing being done by multiple code bases (think multiple APIs that do the same thing). Violations of the DRY principle are sometimes referred to as “WET solutions” (yes another acronym). “WET” can stand for “write every time”, “write everything twice”, “we enjoy typing” or “waste everyone’s time”.

This repetition between APIs isn’t just inefficient but it can lead to inconsistencies and also waste developer’s time by reinventing the wheel. For example, we already have more than one API which pulls out Cautionary contact data — which is information about residents who may be violent or are vulnerable. This data is vital for protecting both our residents and our officers. These APIs pull the Cautionary contact data from multiple data sources. In fact, within just ONE of those data sources (Universal housing) there are multiple different locations (database tables) where that data is held, which means there is a lot of room for error. A potential upshot of this is that an officer could get sent to do a repair at the home of a violent resident without knowing it EVEN though the council does have that information somewhere.

This is where Platform APIs come in. They will ensure that we are only writing code ONCE where that code accesses a particular dataset. If an application or a different API (what we are calling a “Service API”) needs to access the same dataset it will do so via the SAME Platform API. If we need to change the source for that dataset, or add an extension we can do so in ONE location and all of the clients will receive the change. The structure of the response data all of the clients receive will be identical and standardised. From a security perspective we will have ONE gateway into our data which means there is only ONE place we need to secure it.

The second key acronym is “KISS” which stands for “Keep it simple stupid” (more polite versions also exist). The simpler an API is, the less chance there is that something can go wrong with it. The simpler the API is, the easier it is to test. We want to ensure that the APIs we provide are built to the same standards,will be easy to use and so good that people prefer to use them.

We know for a fact that legacy systems tend to be stable but they can be slow-moving, sometimes offline during scheduled system maintenance and unable to cope with technology evolution. This results in services being non-responsive when the backend is busy with applying changes or any massive process is running. What happens, in the end, is that ICT gets questioned about service non-availability, residents are not happy, etc. So how do we solve these problems??

We engineers always think from the dimension of unraveling the data — make it securely open, it’s reusability, the approach taken, catering to similar needs, providing seamless user experience, and make it available for consumption purposes so the services can benefit from it. This allows us to get things going quickly and help us to think in the direction of problem-solving. If the approach becomes sneaky or manipulative then the goal is never achieved. In other words, try to build something which benefits people who use it.

This quote perfectly indicates our scenario.

“Nothing happens unless first a dream ” — Carl Sandburg.

So what is our goal and dream? — Have a series of Platform APIs used at their best!!

We began our journey with APIs to achieve the above goals. We brought the concept of Platform APIs and Service APIs. Identified the benefits for each of the categories along with defining the criteria for each one. Reusability, consistency, quicker access, meeting the user, and data needs (the most important foundation for our approach), and the development of services were the key areas we were looking forward to achieving. The journey was not straightforward.We had to ensure that we are conveying our vision to our colleagues in an understandable, non-technical manner to ensure a collaborative approach. At the same time, we realized there is a mammoth task of understanding the domain model, what needs to be corrected while working with data that is dispersed with no relation whatsoever in the first place. We have got challenges in every direction. (sigh!!)

My point of action was getting the team trained on the latest technologies to build reliable Rest APIs which we all took advantage of and accepted as a challenge to excel. To my surprise, the team smashed it beyond my expectations. By this time we had a suitable platform in place for deploying our APIs and making it securely available to public consumption. We now continue to grow and improve our criteria for a successful framework.

We also have been challenged many times to think about front end services that would use our API. Our answer to this is – if the data is currently being used for a certain internal process why not make it more available? Why don’t we divide the domain into reusable components, which we know are likely to be the foundation to future services? For example, we know that building an application for residents to check their rent balance means that we need to retrieve “people data” (names), transactions, addresses, etc.Those sub domains can be exposed via individual Platform APIs that are then used for any other service that needs the same data. One of the questions we often get is why aren’t our APIs driven by specific user needs related to a front end/service? Because then the APIs would be what we call “Service APIs” — built according to specific needs around one service, which will prompt for duplication in APIs as they will be all very similar due to consuming the same data domains. For quick wins, these service APIs could look very attractive but in the long run, will they be useful? To how many services? Are we duplicating effort? Are we making more room for error? — food for thought!! Following HackIT’s principle of “Fail fast”, we have learned that quick wins often mean more work and they are not viable future solutions.

We strongly feel the data needs should drive API first strategy and not the frontend needs as they tend to change a lot in a given period. These data-driven APIs will shine when we have connected the cloud-based ecosystem using these APIs to their full potential, so why would we make them specific to a service?!! I understand there will be scenarios where we have to develop quick APIs but it shouldn’t always be the case!! However, there is another side of the coin as well. We have data sources that are ‘x’ years old and have no basic database principles applied whatsoever. Building APIs with these data sources as the main entities is a huge challenge because there are no strong principles for any data taxonomy for us to follow.

We have currently partnered with Madetech & AWS Solution Architect -Manish,to achieve the goal of producing an API that will bring resident information from several data sources, achieving a consolidated view.Our aim is not to create a new data source like Citizen Index, it is to provide a consolidated data view in real-time as retrieved by the APIs. This approach will also allow the data to be used for real-time analysis and visualization.

What are we trying to achieve

  1. All services consume Platform APIs, rather than directly speaking to underlying data sources, and thus centralizing the access to data.
  2. Build data pipelines, to make existing data more available from on-premise data sources such as Qlik data etc.
  3. Getting the data normalized and in a sensible format will be a dream to achieve as some of the data sources have no degree of relevant normalization.
  4. Ensure open data standards are followed.
  5. Have a catalogue of all cloud-based platform APIs in the API Hub. The API Hub should be one-stop for all APIs with the required authorization process in place.
  6. Understanding our core objects well and defining data taxonomy around it and further data models to be built and stimulating innovation via our open APIs.
  7. Exposing APIs by establishing data contracts as a step forward.
  8. Engaging different services to understand the vision.
  9. Secure our APIs in a way that does not open possibilities for misuse.

Things to keep in mind or something to explore in the future

  1. How do we keep in mind API first when designing the relevant data pipeline.
  2. As an organisation we have both structured and unstructured data available. I’ve been reading about how a data lake approach enables some organisations to use the data at its best and enables us to build complex business processes powered with advanced analytics. We have several case studies available which could benefit us to learn from. But we want to learn more!

A Virtual Hug, Weeknote 26/06/20

This time last week, I was feeling apprehensive about starting a new role remotely for the very first time. Glady, those feelings were misplaced. Despite this being my first role in the public sector, after just five days, I feel confident and ready to begin the delivery my first project (My Rent Account).

I’ve spent the majority of this week having (virtual) introductions and every single one has left me with the impression that I am joining a community rather than just a job. Thank you!

Discoveries

Relationships Managers

Two key introductions were with Jasmeen and Marian who are both Relationship Managers.

This was useful because one of my first tasks from the ‘Excellent Delivery Manager’ planning meeting is to review the roles and responsibilities between Delivery Managers and Relationship Managers.

These meetings provided me with the opportunity to gain some insight into how DMs and RMs currently work together. We shared ideas as to how we can facilitate a closer working relationship, and it was agreed that there is certainly more scope to increase dialogue and encourage more collaboration.

Further to this, Phillpa has arranged a monthly meet up which is a great place to start. I will be working alongside Emma O’Reilly on this task and look forward to delving into it further.

My Rent Account-

The first project I am responsible for delivering will be the replatforming of My Rent Account. Currently, it’s capped at 5000 users and we have reached full capacity. This means that it is impossible for new tenants to register a new account which must be frustrating.

Under the current climate, finances are likely to be a concern for an increasing number of people so this is likely an unnecessary burden which needs to be fixed ASAP.

The new solution is partially built already and utilises technology from the react component library so it should (fingers crossed) be quite straight forward. We now have a team in place to finish the build and our first planning meeting is taking place next week.

Resources-

The resources availalbe to a new starter at HackIT are endless. I’ve enjoyed the sharing of knowledge, training, show & tells and I’m sure I’ll be exposed to more over the next few weeks.

There are clearly plenty of opportunities to learn and develop. I look forward to improving my skill set and adding value to the lives of residents in the London Borough of Hackney.

Challenges

Despite the readily available resources and support from my colleagues I still expect to face challenges in the coming weeks.

In particular, I have a lot to learn about the inner-working of the public sector. On my first day I was advised to take my time and not take on too much at once because there is an endless amount of information; I agree.

A second challenge I expect is delivering a project with a totally new team. Other than Ibrahim (the Product Owner), we are all new to the MRA project. However, there has been alot of effort already put into this, so I’m more than confident we will deliver!

Up Next

  • My Rent Account- Starting the first sprint and moving forward in helping residents to be able to access their account.

What’s UAT got to do, got to do with it?

Before you start, get comfortable….This is a long one.

The people who build and configure software are, very rarely, the people who will use that software in their day job. No matter how often you’ve spoken to your users and how deeply you understand their needs, there is no substitute for giving a ‘thing’ to users and setting them loose. Never have I ever seen a user acceptance test (UAT) round pass with no amendments which, in my eyes, is the proof of the pudding.

Darth Vader finds your lack of tests disturbing

Recently, I faced users who had either not accepted software before or, if they had, felt it hadn’t gone well at all. This presented me with some challenges; How do you prepare folks to accept software who aren’t familiar with how to accept software or why they’ve been asked to do it at all, especially when their previous experience has been negative? I wanted to make sure we capitalised on the UAT time we had so I knew I needed to address this.

I stripped back to what I consider the absolute bare minimum components of a good UAT and tried to avoid what makes a bad UAT.

hypnotising a user to love the system
This is not how you do UAT (modernanalyst.com)

Here’s what I think you need (at a minimum) for success:

  • Representatives across all the roles expected to use the live system
  • Scenarios that covered the most common types of work the system would be expected to support
  • Clear expectations of what the system should be supporting
  • A way to record what the actual outcome was vs what was expected
  • Enthusiasm & buy in

Colleagues from eight different service areas all needed to work together if UAT was going to succeed. I wanted the team to to know how to do what they’re being asked to do and what good looked like. I wanted them to feel supported to do UAT ‘right’ and safe in the inevitable failures they would have. I wanted them to know that I was on their side 100% because I did not want to end up with this:

I was offered various training courses at high cost to get my colleagues ready for UAT. Spending any amount of money was not really a viable option and the training included from the supplier didn’t addresses what I considered the key points. I could not find any low cost/free ideas that were applicable to our particular situation. I had a bit of a panic so I made a cup of tea (specifically Yorkshire Tea)

Yorkshire tea brass band and giant teapot
Tea can help fix anything

The tea worked.

The team needed upskilling and invigorating to deliver these critical things so I designed a lightweight, engaging way to do that…LEGO.

I engineered a scenario which got 2 teams to UAT LEGO. I tied this into the principals of a good UAT by giving each team different amounts of LEGO and bad instructions so we had the opportunity to learn about what good looks like.

I started with really vague instructions of what I wanted them to do (bad acceptance criteria) and worked with them to identify how that could be improved so the team understood the importance of, and how to create, good acceptance criteria. With vague instructions, these were some of the ‘tests’ we conducted:

As we built better instructions, we found one team couldn’t complete the test successfully because they had the wrong kind of lego to do so. This enabled learning about different roles and why it’s important to not only accommodate them when thinking about what’s being tested but also to actually complete the test in that role.

We also looked at how we could help the team who couldn’t complete their tasks by assessing the feedback we got about what they did and how it went. We worked together to identify what would be useful to know if we’re supposed to be helping them succeed so the team built understanding of why it’s important to record an outcome in a way that allows it to be understood and actioned if it needs to be.

Once the basics were understood, I brought in the industry standard terms to get teams comfortable with the terminology as we needed to communicate effectively and standardised terms generally help with that.

The final part of the session was me facilitating the team to start to identify the roles, scenarios and acceptance criteria for the UAT they would need to run. via the medium of post it notes and enthusiasm. I always related back to what we’d learned in LEGO terms to really reinforce how what they were doing really was the same as the exercise we just followed.

By the end of the session, they had it. They were all confident about how to prepare for UAT and how to run it. We got through 80% of our test cycle before COVID-19 changed the landscape but I am pretty confident we were doing it right due to the amount of issues and queries we were sending back to our supplier. Here’s some figures:

  • Week 1 – 64 issues raised
  • Week 2 – 62 issues raised
  • Week 3 – 52 issues raised

We have yet to get our 4th and final UAT round completed and see just how good our UAT was by releasing the system into the wild but, the team are pretty confident that they have covered all bases to date.

Once the project is back from hiatus and all systems are go again, I’ll update with just how good (or bad) at UAT we turned out to be.