Data Platform Weeknotes: 17 24.01.2021

For more information about the HackIT Data Platform project please have a look at this weeknote on the HackIT blog.

Help us improve how and where we communicate

Thanks to those of you who have taken the time to fill in our feedback form. There’s still time if you’d like to voice your opinions about how we can best improve the way that we communicate with everyone who is interested in the work we are doing. 

Update on how Planning data in the platform is being used

This week we had some brilliant feedback come through from our collaborators in planning. Previously on the data platform project we had worked hard to ingest planning data from the Tascomi API so that planning analysts can produce the reports they need. It has been a challenging process at times but we heard this week that our collaborators are extremely happy with the data dashboards that have been created using the data platform ingested data.

We hope that managers will be able to use the information to inform their planning and decisions. We hope that in turn residents will be able to benefit from the  insight being gained about the way they use the council’s service.

Whilst we’ve been able to recreate the vast majority of planning’s KPIs through the data we’ve already ingested, there were still some tables we couldn’t access via the Tascomi API. This was unblocked this week, so we’re using it as an opportunity to test out our process and documentation by onboarding two D&I analysts who are now able to add new tables to the pipeline. This not only means we have more data in the platform, but we’ve also increased the people who are able to get data in too.

Making housing data available for reporting

We are working with developers from the Manage My Home team to set up an event streaming process to put data from the platform APIs (e.g. tenure API, person API) into the data platform. We hope that this will allow the housing services to see what data can be used within dashboards to provide important information to service managers and product owners, for example how many tenures are created or amended within Manage My Home. 

For a more detailed breakdown of the process involved in this, please have a look at weeknote 16

Housing data challenges: This week we have been testing running a lambda function in the development scratch account that gets a tenure from the development tenure API then pushing a message to a kafka cluster in the data platform account. We’ve had to make some small changes as we’ve been doing this. We are nearly there but are still working out the correct networking settings for the Kafka cluster which is proving a bit tricky. We’ve also had some staff absence due to covid which has slowed us down a little.

Future proposal for ingesting historical data 

We presented at the HackIt Technical Architecture meetup and discussed our proposal for a way to stream historical data. Currently the event streaming will only supply us with data from new events that are streamed (e.g. a new person is created, or a tenure is updated) but won’t provide any historical data. Our proposal includes some changes being made to the platform APIs which will require some coordination with our colleagues in the dev team so that the work carefully considers all of the affected dependencies.

Next: Meet with the dev team to refine our proposal for streaming historical events, and once we’re all on the same page we’ll likely take this to the Technical Design Authority as it’s a fairly wide-reaching proposal.

Backfilling data from our production environment to pre-production

We have completed a lot of work which will enable our move to a production environment. For a more detailed breakdown of this work, please refer to weeknote 16.


Making Council Tax data available for reporting

We’ve been exploring the database created by Capita’s Insight tool as a means to get Revs & Bens data into the platform. Last  week our initial investigations suggested it had most of the key tables that were used in previous analytical outputs relating to Council tax, Housing Benefit and Business Rates.

This week we’ve reviewed our findings with our collaborator in Council Tax, Dave Ellen. We still think the Insight database gives us approximately 90% of what we need, but it’s frustrating that the other 10% available in the live Ingres database isn’t there. We’ve been working with Chris Lynham to get further access to see if these tables could be hiding elsewhere within Insight or whether we really do need access to the full database.

Next : Decide whether the Insight database is sufficient (or at least good enough for now) and run a tech spike on how to ingest this into the platform.

Developing a community of practice

We are keen to connect with more analysts across the council and get more key  people engaged with our work. We have come up with an idea for a superuser group of analysts who we hope will build the foundation of a thriving data platform community at Hackney. We will be planning some meetups and workshops in the near future. We’ve also been asking ourselves, ‘what are all the things we need to improve in order to onboard new users easily?’

+ posts

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.