Housing Data in the Cloud: Weeknote, w/c 17.02.2020

Man playing invisible drums.
Drum roll please….

Chelsea and Simon from AWS have been working on a proof of concept sync-back – a key part of the hypothesis we’ve been trying to validate over the last couple of months. It’s been our biggest unknown and hence our biggest risk. 

Drum roll please….

The proof of concept has demonstrated that we can sync data from a cloud Postgres database back to our on premises SQL Server database. This is good news. The basic concept involves capturing the queries on our cloud database and queuing them to be replayed in our on prem database on a frequency of our choosing. The design is modular so we’re not relying on a single native tool to do all the heavy lifting.

So, we have taken a big leap forward in our understanding of what’s possible. The input from the AWS professional services team has been great. To move from the realm of the theoretical to the actual, we have to ask is it practicable? 

Steve, our data engineer, pointed me in the direction of Richard Feynman’s famous quote: “The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test.”

With this in mind, our next step is to get the proof of concept cloud template set up in our AWS environment so our in-house team can run some tests with it. We need to explore:

  • How will our on prem SQL database behave when the sync process is applied?
  • Does this solution lend itself better to some types of data better than others?
  • What’s the best way to handle conflict resolution?
  • What happens when we increase the number of simultaneous queries?
  • How do we capture information generated after a record insert or update? (Triggers are proving fun)

In a (not so) small personal triumph, I’ve finally managed to coordinate team diaries and we have a show and tell booked in on 10th March. We’ve not been great at doing these in these phase of the project. This will be an overdue opportunity to share our progress with a wider group of colleagues.

Automated Testing weeknotes w/c 17.02.20

Background

This week we have kicked off a new project as part of the DevOps programme looking at automated testing. The idea came from some of the work we did on strengthening support on the website before Christmas, where several members of the team were exposed to Cypress as a tool for writing automated tests. 

These team members identified this type of tool as something that could be used in their current work, which involves both them and colleagues in services manually testing our web based applications. This is an area where we can use modern automated testing tools to save people considerable time and allow HackIT to arrange deployments more regularly. 

This project will last 8 weeks, the idea being that we give the team a broad introduction to what tools are available and get them to start writing automated tests in that time. This will allow us to get a sense of where automated tests can quickly deliver value for HackIT and where we might need to carry out further work. 

The team for this automated testing project is: 

  • Rasit Arif
  • Richard Austin
  • Erdem Cankiran
  • Sharon Comery
  • Hamid Hazrati

All of the team except for Erdem and Hamid are in Applications teams. Erdem is an infrastructure apprentice. Hamid is a front-end developer from Digi2al who will be supporting us to develop our automated testing skills. 

What have we been doing so far?

  • The first week has been a bit of a sprint 0, designed to get everyone up to speed on the key concepts and start to get to know each other as a team. It has also been a chance for us to build out the product backlog. 
  • Hamid gave the team an overview of various automated testing options: Selenium IDE, Cucumber and Cypress. These tools all have different levels of complexity and abstraction. Selenium IDE is the most straightforward to get started with and install on our machines. It is what we are planning to start off with to build the team’s understanding of automated testing and its possibilities. 
  • We also had an introductory session on agile working. Most of the team have not worked on agile projects before, however working in HackIT they have picked up most of the key concepts and terminology. We went through the key ideas and gave the team a chance to ask any questions they wanted. 

What next?

  • Now that everyone has their feet under the table, an idea of the different types of tooling and we have a backlog we are ready to plan our first sprint, which we kick off this week. 
  • In the first week we also identified that the team’s current machines are unable to install certain tools so this week we will get the team access to machines that they need to install the right tools.

Report a Problem Weeknotes : week ending: 2020-02-21

A picture of a man spinning lots of plates.

GOOD THINGS

We’ve had the initial response from Martin at mySociety following our workshop about Noise and Anti-social behaviour (ASB) workshop last week. We’ve got a great first iteration of some user journey maps, personas and user stories. Next mySociety is going to create an initial prototype extension of FixMyStreet for us and other interested councils to provide feedback on.

There’s a lot of other good things progressing well.

  • The contract invoice is sorted
  • We discussing a number of technical issues such as sending automated emails from Hackney’s domain, automated cybersecurity scanning
  • Our respective experts are chatting about the FixMyStreet Privacy Statement

Our first system to be integrated with FixMyStreet will be Alloy (for Highways) which is progressing well. The Alloy team are due to go live in a couple of weeks and are already working hard on integrating into FixMyStreet.

Conversations are continuing with Hackney’s Customer Contact Centre. We’re lining up the first “champion” users and working out who will attend the free mySociety FixMyStreet training.

We’re planning our user research and learning from FixMyStreet related research done at other Councils. Hertfordshire Council have kindly shared their research with us. We’re also looking into how to test the end-to-end journey with Hackney residents – especially around the issue categories.

We’re continuing to discuss how RaP will be supported by the Apps Support team. From a discussion with the Support team this week it was great to hear that this will include some continued iterative improvement as well as “keeping the lights on”.

LEARNED THINGS

It’s really hard to line things up. There’s a lot going on in the project and working out the dependencies can be tricky (see below). As Cate suggested the best way forward may be to move quickly to introduce something we know isn’t ideal but with firm plans on how to support active improvement iterations.

DIFFICULTIES

I’ve defaulted to old-school non-Agile ways of working. As indicated above there’s lots of teams involved in the overall introduction of RaP. All of the people associated with the project are extremely busy with other things – often checking in for only 15 mins a week or less. It would be very difficult to ask them to commit to regular stand-ups or spend time learning how to work in an Agile way so, for now, that’s no high on my agenda. When the timeline is better understood (see below) I’ll see if we have enough time to think about changing this.

We’re still lacking an overall master timeline. This is my current key priority.

We’re short of a Service Designer and I haven’t had any time to work on the Hackney website redesign. Ideally we want to put these changes in place ASAP but I just haven’t has any time to work on them this week. I’ve blocked out some time with a kind Service Designer from another time next week to get things going.

We still need to speak to speak to the Service Areas in the Council that will need to update their ways of working to close tickets in FixMyStreet. I’ll be speaking to the Service Owner about this when they get back next week.

ACKNOWLEDGEMENTS

Our widely distributed team continues to do great work on this. Thanks to Matthew, Louise H., Louise W., Martin, Sam, Struan, David, Rasit, Wayne, William, Ade and Nic.

WHAT’S NEXT

In the next week we will do the following.

  • Complete a first version of the master timeline
  • Do a first cut of the service design for the website update
  • Run a session to define our measure of success
  • Book in a first show and tell!

Hackney Re-Platforming Weeknotes Week ending: 2020-02-21

Weeknotes are a way for us to keep people informed about progress on the project. Given the technical nature of the re-platforming work we will use them to explain technical choices that we are making, including the benefit and impact of these choices. 

Project goals

  • Enable housing officers to use MaT offline when there is poor mobile coverage
  • Enable Hackney to decommission Outsystems, saving £80,000 per year
  • Enable Hackney to support, develop and deploy future improvements to MaT more quickly, at lower cost

Good things

The top news this week is that we have a way forward to address our pressing issue of lack of front-end developers! Starting from next week Ana is going to move from supporting Bukky on fixing BAU issues to working with F to upskill herself on the data-driven front-end (DDFE). The even better news is that two days a week she will be joined by Wayne from Apps Support. This is part of our devops initiative to bring the work of our developer and infrastructure teams closer together.

F’s back and is joined by George from DXW. This means the team can start focusing again on sprinting to deliver the first iteration of T&HC as soon as possible so we can start getting feedback from Housing Officers. 

We held the first of two significant backlog reviews. This went very well, enabling us to remove a number of old tickets that had been completed or were no-longer relevant. Once we’ve held the second meeting to go through the rest of the issues we will also have grouped the remaining items into sets of related tasks which will make it easier to both address similar issues at the same time and prioritise our work going forward.

Decisions

The main decision the team needs to make at the moment is what the main focus for F will be during their remaining time at HackIT. Given the move to have Ana and Wayne start on their journey to learn the DDFE, this means that F may not be able to complete all the work currently lined up for them in the backlog. The team needs to decide how to balance the need for short term delivery vs. longer term capability after DXW have rolled off the project.  

Difficulties

We’re still somewhat uncertain what happens after this round of re-platforming is complete. There’s a lot of things the team could be doing after we complete the re-platforming work such as addressing some of our systemic technical issues, improving the new processes and ETRA and implementing the new Change of Tenancy process. However, at the moment we’re not sure how long the team will remain together after the re-platforming is complete. We have a meeting about this next week where I’m confident we’ll get a clear view of the way forward.

Acknowledgements

Thanks to everyone who worked on the project this week – Ana, Bukky, David, Emma, F, Gill, George, Lorraine, Richard and Tuomo.  

What’s next

  1. The team will work out the best way to start upskilling Ana and Wayne on the DDFE while attempting to minimise the impact on F’s time
  2. Ana and Wayne will start focusing on delivering a re-platformed version of ITV
  3. F and George will continue to complete the front-end delivery and T&HC and HC
  4. Tuomo will continue to focus on producing the back-end code to support the delivery or the replatforming
  5. Bukky will continue to finish off our remaining issues for ETRA and known incorrect data in our databases
  6. Gill will be speaking to Housing Officers about how they are finding working with ETRA and lining up people to test the replatformed T&HC as soon as it is available
  7. David will continue to work with Apps Support to bring MaT into support for non-project work

The following people will be working on the project next week – Ana, Bukky, David, Emma, F, Gill, George, Lorraine, Richard and Tuomo.

The HackIT Excellent Delivery Management project : Sprint-note 18/2/20

Like Voltron the HackIT Delivery Management team has come together over the last 6 weeks to work on a number of excellent tasks designed to either improve how we work or to benefit the rest of HackIT. The team has always been buzzing with great ideas for this kind of thing and people often had iterative-improvement side-hustles going on. What we lacked previously was organisation and coordination. Ian came up with the solution to these problems – organise our own improvement tasks as an Agile project. Hence Excellent Delivery Management was born!

So far Cate, as Product Owner, has suggested that the team use 10% of their time to work on this kind of non-project related work. It’s often hard for people to feel they can set aside their time for such things when important projects are pressing, but we’ve achieved quite a lot in this first sprint.

Speaking of sprints, we’re running the full set of Agile ceremonies over a six-week sprint period. These include weekly stand-ups and sprintly planning and retrospective sessions.

In the last six weeks the team has completed the following.

  • Started building a list of the live digital services we have in Hackney
  • Started building a contractor “bench”
  • Started looking into what tools we can use to construct better product roadmaps
  • Created a list of external Service Standard assessors

We have 18 owned tasks in our current sprint which include the following.

  • Ensure all DMs know how to do a set of “basic” tasks (like raise an PO)
  • Improving HackIT’s capacity planning
  • Create an Agile community of practice
  • Start to improve our overall Product Ownership

If you’re interested you can view our Excellent Delivery Management Trello board.

It’s really great to be part of an organisation where so many enthusiastic people naturally want to be continuously self-improving. We’ll keep you updated on what we do next!