Data Platform Project Weeknotes 13: 23.11.2021

For more information about the HackIT Data Platform project please have a look at this weeknote on the HackIT blog.

Improving the Data Platform playbook

We have spent some more time restructuring our Data Platform playbook. After thinking about the particular needs of data scientists, analysts, engineers and managers and mapping the common and unique parts of the playbook that they may need to access, we have started the process of restructuring the playbook so that it uses a much clearer navigation menu for these users. We see this as a two part process involving restructuring the menu and refining and adding necessary content. 

One example of additions made to content in the playbook is the process  around simplifying creating Glue jobs in code (see weeknote 12 for more information). After writing the content (in this case instructions) for the playbook we have spent some time making sure that they are user friendly to analysts with a broad range of familiarity with such tools as Terraform and GitHub.

This testing is set to continue with our colleagues in the Parking team but has already given us a lot of insight into how we can make the playbook more user friendly and accessible.


Collaborating with Manage My Home

Last week members of the data platform team presented a proposal  on the benefits of using Kafka over an AWS S3 bucket for the event streaming process to the HackIt Technical Architecture meetup. 


Kafka is an open source software which provides a framework for storing, reading and analysing streaming data. We looked at the positives and negatives of introducing Kafka over the current SNS/SQS solution and believe that Kafka provides a more reliable and scalable solution to meet the needs of the data platform. After some lively discussion and debate, we are waiting to find out the next steps for this and are keen to start working with the Manage My Home team on event streaming as soon as possible. Ultimately this will enable us to get the data into a BI tool like Qlik so that the Manage My Home team understands how the tool is being used.

Exploring our Roadmap and Ways of Working

We held another workshop to further refine our product roadmap. As a team we identified all the possible user needs that various analysts, engineers and managers might have when using the platform. We then looked for commonalities between these user groups and mapped the needs on  an affinity map. This process enabled us to refine user needs by priority, complexity and difficulty.

We have also held a workshop to reflect on our current team pattern of agile working and ceremonies. There was a lot of debate about Scrum vs Kanban as agile processes and the best approach when it comes to estimating the complexity and time to complete a task when planning. We have come up with some changes which we hope will make the planning process more efficient. However, we acknowledge that this is an evolving process and one that we will reflect on the success of in the near future.  

Ingestion and use of Tascomi planning data

Work is still ongoing to change the Tascomi ingestion process in a way that stores daily data snapshots in the platform. This sprint, we are also attaching data quality checks to the process.  This is an opportunity to test and refine an entry in our Playbook.

Adam Burnett from the Data and Insight team is deconstructing previous Qlikview reports to understand the business logic behind key KPIs, sourcing the relevant data in the Tascomi tables and recreating them for the Planning team to review. In some cases this means identifying new datasets that need to be added into our daily loads. 

Email ana.bebic@hackney.gov.uk if you have any question about that Data Platform.

+ posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.