Housing Data in the Cloud: weeknotes, w/c 20.01.2020

The Starship Enterprise – going to places no human has been before.

Sync – the final frontier

This sprint is all about moving data. We are testing our preferred migration tool and running some manual tests to see how our existing system* reacts when we try and introduce new data. 

We played with LEGO on Monday. Part team building, part therapy, part creativity. We split the team into two. Both groups built a bridge without seeing what the other group was building. Then they had to work out how to attach the bridges together. 

Mind the gap: getting our bridges to connect

Our LEGO bridge has become an analogy for our sync deliberations. We realised that we have to come at it from two directions: understanding what’s possible in our current reality (hence our manual tests) and articulating set a of high level principles for a future state. The latter we were able to break down further into “must haves” and “nice to haves”. 

This sets us up nicely for our collaboration with AWS starting next month. We will be working together on building setting up the migration, cleansing and transforming our sample data and syncing this back our existing database. 

In people news, Omar from infrastructure is joining our core team. This is great as he has expertise we’ve been missing. A crucial element of this project is about shared learning and making things repeatable. The knowledge we are rapidly accumulating about creating and maintaining cloud environments needs to sit across multiple teams. 

On that point, I’m thinking about how to visualise the learning curve of the team. I’m looking for ideas, feel free to point me towards resources or blogs in the comments section. 

*The system being the old house from my previous weeknote: “an old house which has had many occupants. Each occupant has added data, moved that data around, put it in different rooms, called it different things and used it to prop up the fabric of the building. The old house is weighed down with data, nobody can find what they are looking for and removing data risks a structural collapse.”

Housing data in the cloud: weeknote, w/c 13.01.2020

A starry tunnel.
“Love tunnels, hate walls”

What happened last sprint

“Love tunnels, hate walls.” Profound words from Steve our data engineer, which neatly summarise our last sprint. Getting the VPN tunnel working was our biggest challenge and the majority of our work in the sprint depended on an operational VPN. 

On the positive side, our developers and infrastructure engineers did some excellent problem solving and we did get it working. We learnt a lot, shared knowledge between two teams and we are able to document what we’ve done to make it repeatable. We’re chipping away at professional silos and demonstrating that collaboration really does make the world a better place (cue some suitably cheesy ear worm). 

On the flipside, three quarters of our stories were blocked at the end of the sprint because of our prickly VPN problem. We understood that the VPN was a dependency when we planned our work, but we didn’t anticipate the degree to which it would hinder us. 

Our star of the sprint is Isaiah who worked long and late last Thursday to get the VPN tunnel working. 

What we’ve changed as a result

Omar, our colleague from the infrastructure team will be coming along to daily standups and sprint planning. We would love to steal him and bring him onto the project team, but that’s not possible at the moment.  

What’s happening this sprint

We are facing our biggest unknown – how to sync data back to our on prem database. We’ll be doing some in depth work on this with AWS in February. Ahead of time, we need to do our homework and make a decent stab of understanding the art of the possible. 

Even Doctor Who doesn’t have all the answers all the time.

This really plays to the strengths of the team. Collectively, we are comfortable with a high level of detail and enjoy tackling complexity. There’s a sense of anticipation about the sync bit of the puzzle. It’s the make or break of the hypothesis we’re testing. There’s a flurry of activity going on as I write. The team is preparing for a workshop on Monday afternoon. I’ll share the outcomes in my next weeknote.

Housing data in the cloud: Weeknote, w/c 16.12.2019

This is the last weeknote of 2019. The team has worked hard to keep things moving over the last couple of weeks. I’m grateful for their continued enthusiasm in the face of winter germs, train strikes and general end-of-year weariness. 

I’m iterating my weeknotes based on some valuable feedback; I’m attempting to be clearer about progress and next steps. Here goes…

Bold ambition meets messy reality
When we started out four weeks ago, our ambition was to have (some) data migrated into the cloud by Christmas. Spoiler alert: we didn’t achieve this. 

However, we’ve made solid progress by:

  • Building the cloud platform 
  • Building and testing a VPN tunnel
  • Identifying a use case (solving a problem for users)
  • Agreeing on a database (postgreSQL)

Our workshop with AWS last week highlighted the need for a more indepth look at how we migrate data into the cloud. We will bring together infrastructure, data and developer colleagues from Hackney and AWS in January to tackle this problem. 

What we plan to do in the New Year
We took a pause from sprinting this week. Instead, we focused on tying up some loose ends. Getting the VPN tunnel up and running has been a blocker for a couple of weeks and we wanted to crack this before Christmas. 

On 6th January, we are going to re-group and kick off the New Year with a sprint refinement session. With our stakeholders, we will:

  • Remind ourselves of the goal for the prototype and what “done” looks like
  • Review progress against the roadmap
  • Brainstorm backlog items
  • Roughly prioritise these into 2-3 sprints

Things on my mind
Despite pace* being an agile buzzword, I’m less concerned about this and more interested in team cohesion. As we learn more, do we:

  • Have the right skills mix in the team?
  • Are we sharing knowledge effectively or still working in silos?
  • Have the right balance between doing “enough” for the prototype and the bigger picture
  • Manage dissenting views whilst maintaining progress?
  • Stakeholder engagement: what’s our responsibility vs theirs?

There’s much to ponder here. In the meantime, we wish you all a peaceful Christmas and a joyous New Year.

*I have a whole other ranty blog post about this word in my head.

Housing data in the cloud: Weeknote, w/c 09.21.2019

Cartoon character Scooby-Doo wrapped in a blanket and sneezing
We’ve got germs, they are multiplying

Three things you need to know this week:

“It’s the most wonderful time of the year…” and full of cold and flu germs. We’re falling by the wayside, brought low by various seasonal ills. I’m pleased with what we’ve managed to get done but I’m concerned for the health and well-being of the team. Some enforced relaxation over Christmas can’t come soon enough. 

A change is as good as a rest. We spent the day with AWS on Friday testing our thinking around database choice and migration options. It feels like we are on the right track with our initial decision to go with a PostgreSQL database for the prototype. We identified that we need to do a bit more exploration on how we might migrate data in New Year. It  was great to have colleagues from the infrastructure team with us, but I’m also aware that our data and insight team need eyes on this too. 

What’s next? We parked the idea of doing a sprint next week. Instead, we are using the time to refine our backlog (long “to-do” list) and prioritise that into manageable chunks. There’s a lot of moving parts to manage and we need to get more of a handle on what these are and what’s in our gift to control or influence. To this end, I’ve asked Father Christmas for a crystal ball this year.

Housing data in the cloud: Weeknote, w/c 02.12.2019

Housing data in the cloud. It does what it says on the tin. 

We’ve got data, lots of data. 

This data lives in an old house which has had many occupants. Each occupant has added data, moved that data around, put it in different rooms, called it different things and used it to prop up the fabric of the building. The old house is weighed down with data, nobody can find what they are looking for and removing data risks a structural collapse. 

We are changing this. 

That’s a bold statement, let me put some context around it. We are taking our first experimental steps to see if we can move a tiny piece of housing data into a cloud platform. 

Back in October we did a week long discovery to identify a data candidate for prototype and to think about what success might look like. Thankfully, we’re not starting from scratch. Other fine minds have looked at our old, overstuffed house – we’ve valiantly attempted renovation and even extension. Colleagues from MadeTech have pulled all this learning together and made a set of recommendations, which we are testing in the prototype we are building over the next few weeks.  

Introducing the dream team

The avengers getting ready for battle.
This is what we look like in my imagination. I’m Natasha Romanoff obviously 🙂

We are working with support from AWS and MadeTech along with our award winning team of Hackney developers. With got expertise from our data and insight team and three technical architects (at my last count). I’m terrified, the team is absolutely buzzing. We’re finally staring into the eyes of our nemesis – let the battle commence. 

We are working in 5 day sprints. I love the drive of the team. They want to work hard and fast. We’ve covering new ground everyday. The team are absorbing new ideas, skills and ways of working like a sponge. We don’t all see things in the same way, but the team are embracing this too. 

This week, we’ve identified our use case. Right from the outset we want to demonstrate how our work can bring tangible value to the Council services that rely on this data. We’ve got to keep this grounded in business need and ultimately the needs of Hackney residents. We’ve also set up our cloud platform in AWS this week. Next up: is deciding what database we need for this prototype. There is A LOT of debate about this in the team. I’m expecting a few fireworks. We’ve got a spike early next week to try and crack this.