Data Platform Project Weeknote 24

About the Data PlatformThe vision for the Data Platform is to rebuild a better data infrastructure to  deliver a secure, scalable and reusable cloud-based platform that brings together the council’s key data assets. Enabling us to democratise access to data (where appropriate), use technology to enable deeper insight, and derive greater value from our data to improve the lives of residents in Hackney. For more information, see our Playbook and our Glossary.    

How we’ve been building the platform this week:

Creating a reusable process to ingest data from a MS SQL database so that we can make Council Tax data available for reporting and reuse: 

We now have a fully automated and re-usable process of ingesting data from databases onto the Platform. We have successfully ingested parking data from Geolive, making this data available to our Parking analysts. Geolive is the corporate GIS database held in Postgres/PostGIS. It contains data from different departments, including Parking orders (these are managed in Parkmap and exported nightly as GIS flat files loaded into Geolive).

We’ve also significantly improved the cost and time efficiency of the entire ingestion process of Academy data as it previously took 15 hours + to ingest all of the 291 tables. This particular ingestion process splits up the tables into several Glue jobs which run concurrently taking no more than a few hours each to complete. The data is then programmatically split into the raw zone areas of two new departments: Benefits & Housing Needs and Revenues where relevant analysts can access, query (via Athena/ Redshift) and prepare the data for dashboards in BI tools such as Qlik. The work is currently being reviewed by our team and should be live very soon.

Creating a reusable process to ingest data from our in-house applications through event-streaming so that we can make housing data available for reporting: 

We now have a working setup for the Kafka implementation in Amazon MSK. We have been working to productionize the infrastructure and move the setup fully into our Terraform repository so that we can deploy the setup to Pre-Production and continue testing with Manage My Home. At the same time we’ve continued talking with the AWS team to evaluate if there is any potential to get the AWS Glue Schema Registry implementation working as a better alternative to the self-managed Schema Registry that we have working. With the current self-managed Schema Registry we will be required to make some changes to the Lamdba function implemented by the Manage My Home team, however with the AWS Glue Schema Registry implementation this additional work wouldn’t be required.

Migrating Qlik into the platform infrastructure and moving away from AppStream so that we can improve the user experience for staff:

 Following the Technical Design Authority’s decision to proceed with Qlik on the web with a WAF we were also asked to look at the user needs that result in downloading of the data and explore if this can be met through other means. Last week a number of users from Children and Families Service and Housing explained how important the feature is for them to carry out some key functions. However there are also opportunities to reduce the need if we can automate saving the data into google sheets for them to use. We are working with analysts in the service areas to deliver these whilst the Cloud Engineering Team configures the WAF.

How the platform has been used this week:

Academy data in Qlik so that analysts can use it : 

Now Academy data is in the relevant department’s raw zones, we are able to query the data in Redshift from Qlik and recreate the majority of the data pipelines that were in use prior to the Cyber attack. One focus has been to recreate the “Housing Benefit and Council Tax Support profiler” in Qlik which provided the service with an overview of the current claim caseload. We have been able to reproduce the data and refresh the dashboard and will spruce up the dashboard before sharing with the service. Housing Benefit data is also an integral source of the Supporting Families team (FKA Troubled Families) who use the data to evidence that cases have moved back into employment and claim payments from central government. We shall imminently be reinstating the reports for the team so they can restart the Payment by Results process. 

Steve Farr has also been working with Dave Ellen, an analyst in Revenues, to recreate commonly used queries relating to Council Tax and Business rates. Steve has successfully recreated a number of Dave’s queries in Athena and will be carrying on this work and exploring how to make “trusted” data sets available to managers in the service and onboarding Dave to use Athena to query the data on the Platform. 

We created a reusable training module to help onboard users so that more staff can use the platform to complete their work: 

This week we continued our training program, and ran module 3 training for our team members. We’ve used the feedback from the training to refine our processes. We’ve also been putting together a form which we will be rolling out to people who would benefit from being trained in the use of the data platform for the purpose of data analysis.

Up Next:

  • Continue to develop trusted datasets for Council Tax and recover previous data products dependant on Housing Benefit data
  • Continue testing our event streaming process with the Manage My Home team
  • Support the Cloud Engineering team to implement Qlik over the web, and draft another paper for the Technical Design Authority about the downloading of data
  • Investigate how we make Alloy data available to analysts in Environmental Services through the Alloy API

Modern Tools for Housing – Programme week notes 11/3/22

Modern Tools for Housing – Programme week notes 11/3/22

The Modern Tools for Housing programme is the development of a suite of software products developed in collaboration between HackIT and Hackney Council’s Housing Department to support our staff and our tenant and leasehold residents. It currently covers housing repairs, home management, housing finance and support for our arrears collection team.

Highlights from our workstreams this week.

Managed Arrears – Silvia – workstream DM

  • Celebrations:
    • To start to build a new application, and follow the agile methodology
    • Clearer overview of tasks and assignments
    • Finalised and built the User Interface(UI) for the user stories
    • Started building the use cases to send email with rent statement attachment via gov notify
    • Had User Acceptance Testing (UAT) sign off for letter issue date bugs fix
    • A bug in the MAA system was brought to our attention on Friday morning which was immediately resolved by Felipe.
    • Being able to wire up the Frontend with Income Application Programming Interface (API).
    • Being able to integrate MMA into Finance Application.
  • Challenges:
    • Understanding the existing data structure and integration.
    • Understanding the requirements for wiring up the frontend with the Arrears and Patches API.
    • Using the Search API for find data between MMA and Finance correctl.
    • Prepare backlog items properly in terms of Acceptance Criteria, description – thanks to Elaine & Burak’s hard work we’ll be in a better place with this in the next sprint.
    • The flow of data wasn’t as clear as it could be so we’ve completed a component/data mapping exercise.
    • We still have some bugs in the product and these are creating some problems for our users. We need to make sure we devote time to fixing the most important ones.

Finance – Kate – workstream DM

  • Celebrations:
    • The data migration and Leasehold estimates on staging for UAT with good initial feedback
  • Challenges:
    • Some team members being off due to sickness caused a slight delay in some progress.

Manage My Home – Yvonne – workstream DM

  • Celebrate:
    • First S&T :partying_face:
    • Ways of Working workshops and changing how work organised
  • Concerns:
    • Lack of FE Capacity – it’s a single point of failure that needs addressing
    • Yet to release with new team (mainly due to capacity issues and upskilling)

Repairs – Sarah – HackIT workstream DM

  • Celebrations:
    • Released value to users 3 times this week!
    • Enabled multi trade variations
    • Resident safety meeting regarding follow on’s risk assessments was really interesting and there are areas of opportunity to explore
    • We’ve found a solution to make the Purdy onboarding stories smaller
  • Challenges
    • We weren’t able to UAT legal disrepair flags – Rolled over another week
    • The contact centre reporting bugs in the UI when raising repair
    • Fitting in supporting Made Tech for Online Repairs

Lots of great stuff is happening in our individual workstreams as ever. I highly recommend that you read the individual weeknotes listed above.

Not much newsworth at the programme level this week. The latest Repairs statement of work was approved at the steering group meeting on Friday while the ones for the Finance and Manage My Home workstreams have been carried over for further discussion.

We’re looking forward to both our two new QA engineers and the MadeTech Repairs Online team starting work on Monday.

The Product Manage internal recruiting for the new Repairs product team closes this week and we’ll be continuing to recruit for other roles across the programme with a particular focus on the 2nd front-end developer for Manage My Home.

Modern Tools for Housing – Programme week notes 4/3/22

The Modern Tools for Housing programme is the development of a suite of software products developed in collaboration between HackIT and Hackney Council’s Housing Department to support our staff and our tenant and leasehold residents. It currently covers housing repairs, home management, housing finance and support for our arrears collection team.

Highlights from our workstreams this week.

Managed Arrears – Katie – workstream DM

  • Celebrations:
    • Large parts of the NoSP work already completed
    • Work on Rules Engine and Components Library started
  • Challenges:
    • The team had a few challenging moments with the Ruby on Rails framework, which have now been resolved

Finance – Kate – workstream DM

  • Celebrations:
    • The data migration update on staging’ this is a big step forward, as it includes millions of rows of data that can now be tested by users
  • Challenges:
    • Waiting for ADR sign off to enable progression on a number of tasks

Manage My Home – Yvonne – workstream DM

  • Celebrate:
    • Really getting into the swing of working in sprint formats
    • Great workshops for the team to understand how they’re working, their pinch points and what they want to do differently.
    • Julian returned this week – welcome back Julian!
  • Challenges:
    • Only one FE Dev is meaning a much slower pace.
    • One service designer/Ux designer is also meaning a slower pace

Repairs – Sarah – HackIT workstream DM

  • Celebrations:
    • Removed Wick village TMO flags
    • Deployed: New priorities surveying and planned maintenance, syncing to the new contact details API, bulk cancelled over 600 out of date jobs, jobs can be closed without paying operative bonus
    • Introducing a simple but structured release strategy
  • Challenges
    • We weren’t able to UAT legal disrepair flags
    • Lots of questions about online repairs and how we need to ensure it fits with our needs
    • Bugs reported by operatives on Monday

The biggest news this week came from the Manage My Home team where they were able to confirm that the product is now being used by over 500 staff members across more than 40 Hackney Service Area teams. This is a great achievement and a testament of all the hard work put in by both the current team and our previous partner agency for that service, Amido.

Speaking of great teams, the Product Manager role for the new Repairs product team, a key pilot part of HackIT 3.0, was advertised internally on Currents on Friday. If you’d like to apply for the role you have until 17/3/22 to do so. We’ll be finalising the discussions about potential internal moves for other roles in that team this week. Following that we’ll be doing external recruitment starting as soon as next week!

In addition to the new roles in Repairs we should be finishing staffing up the new Manage My Home team soon to get them back up to full speed as well as hopefully bringing in both a Service Design and Business Analyst at the programme level. The BA in particular is going to focus on finding paths to enable us to start removing the remaining interim spreadsheets introduced after the cyber-attack and also looking into clearer ways for the programme to be able to express the value to users that’s provided by each of our newly released features.

Even that’s not everything related to new people coming into the programme. On 14/3/22 we have two new QA engineering specialists joining to help all four workstreams ensure that they deliver the highest quality code.

Meanwhile, the must be an auspicious date as it’s the same day the our new partners in the programme, MadeTech, will be starting the delivery of Repairs Online. This is our implementation of a free pre-existing open source system that will enable residents to raise repairs directly without needing to call the call centre.

Finally, it’s our steering group on 11/3/22 so we’ll be prepping for that this week including make sure that the new SoWs for MMH, Repairs and Finance are circulated in advance so we can have them approved at the meeting (they’re all in good shape and should be sharable on Monday).

Cloud Engineering weeknotes, 25 February 2022

It feels like a lot has been done this week, which is good. It helps that all team members have been available this week, not that having time off is a bad thing!

We started the week with our old friend WordPress. We found out late on Sunday night that two Hackney sites had fallen over. Emergency action was taken to restore both sites and to amend the EC2s they run on, so that this shouldn’t happen again. The real solution, though, is to complete the move to the new infrastructure. This has been going slowly due to an issue in the load balancers, but we think we’ve resolved that now. 

Migration work has picked up again; the first grants application is nearly finished after being paused for a few days for urgent support work. We will review whether to move the other two now or later, in view of this week’s government announcements. At account level, we are almost in a position to move the DevelopmentAPIs account, but have discovered that it has only one subnet – which the move process tries to destroy. We’re investigating a workaround. 

We’ve been supporting the Mosaic project a good bit this week. The EC2 box they need has been built and the client installed, and we are just waiting for Servelec to amend the VPN configuration on their side. We’ve also now obtained a trial licence for Host Access to test with Plus5, and need to set up a VPN for that. Finally on the networking front, we had a really constructive discussion with Data Platform about making access to Qlik as secure as possible, given the desire to take it out of AppStream but in the knowledge that it’s not compatible with Globalprotect. A paper will be discussed at TDA next week. 

We reviewed our roadmap this week. Most of the changes were to break existing items down into smaller chunks, but we have added new tasks to create new Terraform modules (as a result of the Backstage paper mentioned last week) and to iterate existing ones. One iteration pulled forward is to change the default setting in the Overnight Shutdown module to TRUE, so that all EC2s will shut down overnight unless explicitly changed. This change will be merged soon and will apply when you next update your Terraform, so be aware and take necessary action to ensure that any EC2s needed overnight remain powered up. 

Modern Tools for Housing – Programme week notes 25/2/22

The Modern Tools for Housing programme is the development of a suite of software products developed in collaboration between HackIT and Hackney Council’s Housing Department to support our staff and our tenant and leasehold residents. It currently covers housing repairs, home management, housing finance and support for our arrears collection team.

Highlights from our workstreams this week.

Managed Arrears – Katie – workstream DM

  • Next Show and Tell: 22nd February 2022
  • Celebrations:
    • Large parts of the NoSP work already completed
    • Work on Rules Engine and Components Library started
  • Challenges:
    • The team had a few challenging moments with the Ruby on Rails framework, which have now been resolved

Finance – Kate – workstream DM

  • Next Show and Tell: 8th March 2022
  • Celebrations:
    • Establishing Jira processes to allow us to start using it for our next sprint.
  • Challenges:
    • ADR sign off pushed into next week due to key Senior Technical Lead being on annual leave.

Manage My Home – Yvonne – workstream DM

  • Celebrations:
    • Had a really good workshop where we explored our roles and responsibilities, identified gaps and now looking at how we close them. 
  • Challenges:
    • Learning and doing means pace is slower
    • Not enough BA, Front end, service/UX design or QA capacity

Repairs – Sarah – HackIT workstream DM

  • Celebrations:
    • Adding legal disrepair flags is almost ready
    • Really positive meeting with the trade union
    • Making good progress mapping out risk assessments for operatives
  • Challenges
    • We weren’t able to release new features this week as planned
    • Teodore, one of our software engineers, is leaving

There’s a lot of other work always going on in the programme that isn’t directly related to the delivery being done by our four workstreams. This week that’s included the following.

  • Lining up conversations with our new partner Mobysoft to ensure that we’ll have all the technical integrations in place for when our Arrears team starts to use their product RentSense in April
  • A number of conversations with the new team in Hackney working on reviving the pre-cyber-attack service called Single View. We’ve been explaining to them some of the reasons why previous decisions related to their product were made and showing some of the reusable components and APIs that they’ll be able to integrate with.
  • We’re set to have our first technical integration meeting with MadeTech to help line up their team which will be joining us shortly to introduce the resident-facing service Repairs Online

Meanwhile, other work that our teams are doing that’s not mentioned above includes the following.

  • It’s “statement of work” (SoW) time again. This is how we chunk-up the work we do in order to gain spending approval through our governance process. This time around it’s unusual that all four teams will be writing SoWs at the same time so we’re going to have to coordinate carefully to enable them all to be approved before the next steering group meeting on 11/3/22.
  • We’re now fully lined up with approved job descriptions so will begin the process for recruiting into the long-term HackIT 3.0 Repairs product team next week
  • The Delivery Manager community at HackIT is going to be spending a bit of time observing each other’s teams – potentially leading to an “Agile health check” document that could be used by teams in retrospectives. We’re lining up someone to watch some of our sessions.
  • We’re just about to confirm the introduction of two QA engineers who will be joining the programme soon – working across all four workstreams.

Finally, speaking of quality assurance – we’re going to be reviewing our programme QA requirements (accessibility, privacy impact, data retention, etc) to make sure they’re all up to date. At the same time we’re going to review our regular meetings and documents to make sure everyone understands the purpose of each one and that we’re not spending time on anything we don’t definitely need.