Cloud Engineering weeknotes, 10 December 2021

It passed me by that it was our first anniversary two weeks ago. It’s been quite the year, and although it feels like not a lot has happened over the last week, I’m not sure that any of us would have forecast the progress we have made in the last 54 weeks. I remember a workshop earlier this year, thinking that we had so much to do; it would be interesting to run that again to see how much we’ve moved on.

Although it’s good that we’ve been able to do so much for colleagues across the council, what I’m most proud of is the team we’ve built. Actually agile, focusing on value, flat, and dedicated to learning. We have a camaraderie that has sustained us, a culture that is healthy and welcoming. Long may this last.

Over this last week, the team has got some good stuff done, mostly to support other teams. We’ve built a VPN to connect to Servelec’s back end to enable a data migration; built some EC2s; set up networking for Social Care Finance; and are investigating ways to enable colleagues in HR to receive data from our payroll provider.

Some of the other work is being used as a catalyst, or maybe a test bed, for things we knew we’d need to do in future anyway. For example, we know that external services will need ingress access to our environment, so we have used requests from the Academy application manager and from the Data Platform team to work out how best to do this in a secure way, and how to automate it.

There’s been some progress on account migrations. The GIS apps are talking to each other but we need to set up a connection to the Addresses API as the final step. We’ve agreed a way forward on the Housing accounts with the Housing Finance team as well, and that should be unblocked early next week. Progress on the websites migrations has also come on strongly, with an AMI built in Packer and an ALB configured. We’ll migrate the first site in the staging environment shortly, to make sure it all works as expected.

Modern Tools for Housing – Programme Week Notes 10/12/21

Modern Tools for Housing – Programme Week Notes 10/12/21

Ups and downs this week as the clock continues to run down to year-end. Our workstream updates.

Finance – Silvia – HackIT workstream DM

  • Celebrations:
    • Progress was made on the MAA Letters bug, which has now been fixed
    • Great progress is being made on the MVP and is on track to be delivered before Christmas. PEN testing taking place next week.
  • Challenges:
    • MAA Bugs and issues continue
    • Unknown future of the project and who will be running it is causing uncertainty.

Manage My Home – Yvonne – HackIT workstream DM

  • Celebrations:
    • Repairs showing in MMH now live! 
    • Team have left the work in a good place and well documented to be picked up again in the new year
  • Challenges:
    • With the Amido contract ending we have a very small internal team so ability to produce functionality or progress work is very limited. 

Repairs – Sarah – HackIT workstream DM

  • Celebrations:
    • MMH now displaying repairs data
    • Bug fix for mobile – jobs no longer disappearing when completed
    • Bug fix in DRS – jobs are now being closed to the correct time
  • Challenges:
    • Saturday & Monday there were issues scheduling jobs in DRS due to a release on Friday
    • Dealing with queries regarding the Bonus Calculation app 
    • New phones ready for plasterers, multi traders and carpenters which is great, distributing them however is proving challenging so we’ve asked the managers to work with the planners to book the time

Plenty of good news this week. We’ve resolved our Google Groups issue that was blocking the display of repairs data in Manage My Home which means that information is now available for everyone using that product! We also fixed a number of issues in Managed Arrears including the NOSP (Notice of Seeking Possession) bug.

The paperwork is complete to extend the contracts for two of our vendors until mid-Feb. We’re also busy reviewing the Digital Marketplace submissions for the agency to take over all three workstreams in the new year as we transform into a product team (likely with a number of sub-teams given the size of the programme).

We’re sad to see one of our vendors rolling off this week. The work they have done on Manage My Home has been excellent and they’ve inspired service design and technology improvements (especially around testing) across HackIT.

We held our first Product Owner meeting for a while which made a great start on discussing how we’ll measure the benefits of the programme. Much more on this to do in the new year.

We re-started the conversation around how we’ll define the hierarchy of assets (estates, blocks, dwellings, etc) in our Asset data microservice as this will become important in all workstreams in the near future.

Work continues on how we can ensure that a number of components that HackIT is building (patches, Document Store, notifications, etc) can and are reused by as many of our developed services as possible.

After speaking to the Product Owners I’ve been re-inspired to set up my idea for a regular “blue sky day” to change the narrative for the programme away from replacing Universal Housing to one about what we want the long term vision of the combined service to be. I’m looking forward to discussing who should be involved in that – potentially including a set of resident representatives.

Finally, our contractor job role is still open – if you know anyone who may be interested please email

Rebuilding the Housing Register – weeknotes 07/12/21

Since the previous Housing Register update, significant progress has been made by the development team. Due to time pressure, resident-facing functionality had previously been prioritised over the functionality that would be used by Benefits and Housing Needs Officers (Housing Register) to process cases. Whilst this allowed us to quickly get feedback on the resident-facing functionality, it meant that the Officers were still having to use various manual systems to complete their work.

We are glad to say that multiple features that will assist our Benefits and Housing Needs Officers with their day-to-day work have now been completed and have gone live, including the ability for them to edit the status of a case and a full audit trail of any changes that have been made to a case. As with the resident-facing application, the team has been gathering feedback and iterating on this functionality in order to ensure that any defects are being resolved and that the system matches the needs of that group of users as closely as possible.

On the resident-facing side, we currently have 32 applications to be placed on the Housing Register that have come through the new system and are currently being assessed by the Benefits and Housing Needs Officers. We have also added the ability for residents to be able to sign out and then return to their application when they sign in again, which was previously not possible. And, coming soon, we will roll out a redesign of the “task list” page, based on feedback that we received in order to make it easier for residents to understand and use.

The largest piece of work that has been undertaken on the resident-facing application has been the automated calculation of bedroom need and the automation of the workflow where a household resident has a birthday, as this can affect the bedroom need. This is a complex calculation, so has undergone multiple rounds of testing and is now ready to go live.

Now that the system is up and running, our FutureGov colleagues will be moving on to other projects in the next few weeks. The Housing Register application will continue to be supported in a maintenance capacity by HackIT internally, before being absorbed into one of our larger Product teams in 2022, who will continue to build on the work that has been done up to now and enhance the system even further. We would like to place on record our gratitude to the FutureGov team, who have been a pleasure to work with and who have helped us to deliver a Housing Register that will meet the needs of both those applying for social housing and the staff who are processing their applications in a way that provides an excellent user experience for both groups.

Data Platform Project Weeknotes 14: 08.12.2021

Data Platform Project

For more information about the HackIT Data Platform project please have a look at this weeknote on the HackIT blog.

Enabling MMH to push data to the platform

We have been working on setting up a template for the Manage My Home  team to work with, set up and create their project.  The template  will minimise the amount of repetitive set up tasks developers who are collaborating with the data platform team (and Kafka specifically) would be expected to do. This will also speed up the process of sending data from the MMH app to Kafka so that it can then be ingested into the data platform. Kafka is an open source software which provides a framework for storing, reading and analysing streaming data. 

Moving Qlik to the production environment 


We are in the process of moving the business insight tool Qlik away from the Productions API account  and onto an AWS production Data Platform account. This will allow us to move away from accessing Qlik through AppStream. We hope that it will also make the use of Qlik a lot more straightforward for its users. The process involves the replication of existing connections to data sources. So far we have connected to the Social Care tool which is in the Mosaic Productions AWS account and we are currently looking at connecting up the Repairs Hub data. 

Refining Our Product Roadmap

We continue to further refine our product roadmap. We have now established about 25 user needs for four unique user groups. We  have also worked to find common user needs between these four user groups in order to find out the large scale tasks which we need to prioritise. 

We then worked to find common user needs in order to establish the large scale tasks which we need to prioritise. In the image below, we mapped the user needs in order to establish the complexity/difficulty of tasks. We now  have a clearer understanding of what we have achieved so far and what remains to be done. All of the user needs highlighted in yellow have had some work started on them already.

There is still a lot of work to be done but we are also very proud of the achievements made thus far. We will be going into more detail into the development of the roadmap as well as answering any questions you may have.

 Establishing a process for supporting users

We came together to look at the best way to support our collaborators with requests for help. We have found that it has become somewhat unmanageable to keep track of user support requests and the work that needs to be prioritised or is already in  progress. We have come up with a solution to track requests and will be sharing the proposal with some of our users this week.

Next Show & Tell – Friday 10th of December

Our next Show & Tell is on the 10th of December at 12-12.30pm. Come along to find out more about what we are up to and invite others that may be interested (the calendar invite is open). Email if you need any help. For anyone that can’t make it, don’t worry we will record the session and post on Currents/Slack after. 

Cloud Engineering weeknotes, 3 December 2021

Guest post by one of the team’s engineers

Our second week of the sprint carries on without our delivery manager who is taking some well earned rest. It’s a testament to the trust he has in us that we’ve been left to run our own show and tell and retro without the need for a substitute DM. It’s also a testament to how well the team works that we’re all aware of our roles and what needs to be done.

Account migrations have moved on a little. Advanced e5 was onboarded successfully with assistance from Advanced and the finance team, and GIS apps are being migrated to a dedicated account. The housing staging and production accounts are dependent on some development work being completed. The API accounts are all dependent on resources being migrated to their own accounts and we expect movement on that over the next few weeks.

In Globalprotect we are now looking to set up role-based access so that users can only see “their” applications. So that we can apply additional controls we are looking to implement two entry points (portals) for internal and external applications. As part of the security implementations we have added a set of scripts that enable API application creation, this enables us to restrict access to the management console.

A further security improvement is AWS Federation for GitHub actions, which would allow us to remove some credentials that are currently being stored in the Infrastructure repo. This will also allow for dynamic role assumption based on repo, tag or person running a workflow in the future.

We have started developing Ansible for the various Hackney websites. The idea behind this, is in a similar way to writing Terraform modules, we can start to create Ansible roles that are reusable blocks of configuration. This can then be used to do things like create our custom AMIs, that conform to stricter security standards or have any prerequisite customisations baked in. Additionally, we can use these to actually configure an EC2 after it has been deployed automatically. The intention is to deploy all the various websites and Backstage in this manner, creating a pattern for any other future applications that require hosting on EC2.