Improving how we manage user research data

Managing user research data and participant details correctly is a challenge that many user researchers face. This ranges from organising your research data clearly, to briefing colleagues on their responsibilities for protecting participant privacy during research. 

When you’re busy working across different teams or pieces of research, there is a risk that these tasks slip to the bottom of your to-do list. But they’re essential in ensuring that participants have confidence in their privacy rights being respected and so that we follow the law.

Why we wanted to improve our process

One of our new user research values involves embedding research operations within our team so that we can do our best work. Richard mentions how we created our values in his recent blog post.

Hackney’s Digital Design Team is a mixture of researchers, designers and apprentices. We all carry out user research as part of our roles and we’ve completed research on a wide variety of council services now. 

Over time we’ve developed different ways of managing research data within the team. As a result we found that participant details were sometimes being stored in people’s personal drives, emails and in product team drives. It was also becoming difficult to remember to delete all participant data as soon as it was no longer needed. 

Making sure we store research data securely

When we talked to other user researchers in local government we discovered that other teams experienced similar problems. There is helpful guidance from the Government Digital Service (GDS) on managing user research data and participant privacy but we needed to formalise our process for storing research data securely. 

We approached GDS to find out how they do this and then worked with information management colleagues to create our guidance on how we store user research data.

The guide includes how we:

  • store research outputs, such as consent forms and research reports
  • organise participant details 
  • delete data when it is no longer needed 

We now store participant data in a shared drive which only researchers have access to. As well as minimising the number of people who have access to the data, this allows us to easily manage participant details as per our consent form. It also reduces the likelihood of adding any raw data to the user research library

We’ve all started following the new process now. It’s a work in progress but feedback from team members shows we’re heading in the right direction:

 “Thanks to the guide, I was reminded that data can end up in unexpected and forgotten places such as email. That data has now been deleted and I’m confident that our raw participants data is now being stored in a secure place that we’ll be regularly monitoring.”

How we are avoiding research debt

As part of our new process, we recently held our team’s first data hygiene day. We had the idea for this after reading Caroline Jarrett’s blog post on how to avoid research debt

Everyone reviewed, organised and deleted research data from shared drives, Google forms, third party tools, emails and elsewhere. We are going to do this every six months to help ensure we keep on top of organising research and looking after participant data correctly. 

Next steps 

We’ve published this guide alongside other guidance we follow when conducting research with Hackney residents and council staff

We now need to improve our user research operations in related areas, including reviewing how we get informed consent from participants. 

We also want to consider how we can best support colleagues to manage user research data correctly. This is because projects do not always have a Digital Design Team member to lead on user research or they might hire a contractor to do this.

DevOps Practices – where have we come from and where next?

Where have we come from?

Our DevOps practices programme has been running for over a year now. During this time we have supported delivering value to users in many areas. In each case sought to guide the team towards making the product automated, reusable and maintainable. As a result of our work users are seeing new products delivered faster, the website is quicker and more sustainable, and teams across HackIT have been exposed to new digital tools for hosting, deployment and testing.

We have demonstrated the financial benefits of a primary cloud supplier, created standards and consistent approaches to development, deployment and automated testing which means we deliver value more quickly to users. We’ve worked across boundaries to do this between infrastructure, apps management, support and development.

We did this deliberately through the medium of specific projects but now we want to take what we’ve done and make it core to everything we do. So far we have been testing and learning and delivering value along the way. We have introduced people to new skills around automation, monitoring, and cloud architecture. We now want to focus on increasing the amount of value we’re delivering to our users and to ourselves. 

Where should we go next?

We held a retrospective with people who have been involved in the projects in the last few years to inform our next steps. There were a number of key themes that came out of it. 

We had worked well in collaboration across teams, we had helped people to learn new skills and try them out. However we needed to provide more structure, a narrower focus and dedicated time for people to really give people the opportunity to embed these new practices as part of their day to day work.

As a result of this we identified some key principles that would help us to find an area where we could narrow, focus and structure our future DevOps work. 

The principles agreed were that future work would: 

  • solve a whole problem
  • intentionally involve people from across teams
  • contribute to work we would need to do anyway
  • be an exemplar of what we mean by DevOps, 
  • incorporate key DevOps functions / principles / ways of working
  • help us to signpost or decide what happens next

What will we do?

We will use these principles to work on an area of technology that a project team of around 10 people could take ownership of. This team will be made up of people from across HackIT. It will be supported by Digi2al and will provide the expertise and structure people need to develop. 

We will be spending the next couple of weeks working out the details of how this will work with the existing HackIT teams. We will write something shortly about the types of people that will be involved in this project. If you have any questions please get in touch.

Testers needed: designing inclusive retros

The What

In January I ran a workshop at UKGovCamp. I think it was called “How to love your introverts” – I’m struggling to remember that far back. Attended mostly by introverts (and a couple of brave extroverts), we explored participants’ views on what introverts need to thrive in collaborative environments.* 

Borrowing from the fantastic workshop contributions, I’ve pulled together** a slide deck on designing and delivering introvert-friendly retrospectives. The focus is on ways to foster inclusivity as the norm, not the exception. Why retros? I needed a place to start. My hypothesis is that the principles translate to any type of facilitation. 

The Why

I am an evangelist for collaboration and I am a card carrying introvert. Sometimes it feels like the two things don’t sit well together. Open plan working (although clearly not a thing at the moment), lots of visual stimulation, conversation-led work could be a recipe for disaster for introverts who generally prefer space for thinking and reflection and low stimulus settings. On the flip side, working in “pizza sized” focused teams is fertile ground for introverts, who tend to blossom in smaller groups. 

Despite the connotations of the word, collaboration isn’t necessarily inclusive. Reflective of our extrovert-centric culture, facilitation techniques often reward the attributes and qualities of quick-thinking, talkative types. Figures vary, but up to 40 per cent of the population are introverts. Designing our agile practices to bring out the best in introverts is a necessity, rather than a nice to have. 

The Ask

I’m looking for feedback on the ideas in the slide deck. What works? What doesn’t? What have I missed? My contact details in the deck or use the comments section below.  Thank you for helping me to make this piece of work better.

*As an aside, introversion and extroversion are not binary concepts. Personality traits exist on a continuum. People who exhibit both introvert and extrovert tendencies are dubbed ambiverts.  
**What took me so long, right?

DfE Technology Support Scheme – project note

life fail GIF
This didn’t happen. Honest.

Back in April, the government announced it would be providing free laptops and internet to school children. HLT were leading on this work and took care of the ordering and liaising with schools on the numbers needed. 

The TLDR version is we smashed it! We made some mistakes, and we learned a lot and we adapted and changed approaches as we needed to as none of us really knew what to expect or how it would go. We were thinking on our feet and timeframes didn’t allow us the luxury of a lot of things. 

A lot of what we did on this project can certainty help us in other projects, such as the Future Homeworking project, in designing how we deliver a thing and collate data. 

Also, we all laughed a lot and Christian, Anwar and Shakti (and Bruno) were a brilliant bunch to do this with and worked so well together to solve problems and get things done. 

My aversion to public transport meant I was walking in and home most days. By the end of it I’d walked a total of 71 miles.  

I also went deaf in one ear again due to an ear infection. This caused a lot of amusement. 

What we did: 

To be honest, my first thought on discussing this was ‘I have absolutely no idea how on earth to do this’. 

We had the task of designing and organising the end to end process of accepting the delivery of 1600 devices and getting them to each school as quickly as we could. 

So, no pressure. 

On the HLT side we worked with Emma Claridge who handled the ordering and liaising with the schools on the numbers they needed and also arranged the drivers and vans for delivery from Environmental Services. 

We worked at an incredibly fast pace on this. We knew delivery was due on 30/06/20 so had just less than a week to find a team and work out:

  • Where were we going to put 1500 devices?
  • Who would we need on the team?
  • How would we get them in?
  • How would we log them all?
  • How would we assign them to the destination? 
  • How would we batch them? 
  • How many deliveries could we get on the van? 

Where on earth were we going to put 1500 devices?

We met with Paul Hornsey who kindly lent us Robert House until 10th July. This meant we finally had a hard deadline to get these out by. 

Who would we need on the team?

Sandeep was onboard from the start with his ability to understand how to log assets and ensure we track them properly. These weren’t devices that were going to be managed by Hackney but we did need to ensure we logged them as received and delivered property and onto the asset register. 

How would we get them inside?

We were told the delivery would arrive on pallets and they would put all of them onto the pavement and we’d have to get them inside. The doors to the room we were using and storing weren’t wide enough to get a whole pallet through. This sent my anxiety levels sky high as we needed enough people to help get them in quickly as possible and others to help stand guard to ensure they stayed safe. Oh, and hope it didn’t chuck it down.

Joss from HLT came down to help, Sandeep drove in from Kent and Paul came on site for the day.

The delivery was, alas, 5 hours late and some had to get home as they couldn’t stay that late. Zaf and Colin arrived with the key to solving our delivery headache – there was apparently a big allen key that opened a partition wall which allowed us to access the wide doors to get full pallets in! Just had to find the big allen key. Which we did! Hurrah! And with more help from HLT staff and facilities team we got the pallets in all ok and my anxiety subsided. 

How would we log them all? 

Sandeep had worked on a google sheet that easily allowed the team to log the serial number of each device, the date received and date dispatched. We’d been working on delivery receipts for the schools to sign so were able to log in that these had been signed as received. But still – 1600 devices to type into a sheet is still quite tedious and time consuming. Enter three barcode scanners that just made ife super easy in this dept! Beep, beep, beep!

Lesson learned early on in this – best to scan small batches and check to ensure that all logged on the spreadsheet. The team had got to number 456 and realised that there weren’t the correct number of populated rows in the sheet. Without a way of knowing which had been missed, it was a case of learning the hard way and having to re-scan them all. Going forward, they scanned in 10s and checked the number of populated rows correlated. It turned out there were connectivity issue with the internet dropping out while scanning so the team was now alert to this and managed it better. 

How would we assign them to the destination 

Once we got the names of the schools they were going to and the numbers going to each school, as well as the type of device (Chrome or Windows) we got super high tech and I sat with a bunch of post it notes and wrote out stickers for them – one post it with the name of the school on for each batch of 5 or less

  • Eg: one school had 53 devices
  • I wrote out 11 post it notes with the name of the school on. 
  • The team would scan 5 devices, tape them up and stick one post it note on it 10 times. 
  • The final 3 devices were scanned and batch as a 3.
  • For that school, there would be 10 bundles of 5 and 1 bundle of 3.  
  • There’s probably a much better way of doing that, but in the time we had, we couldn’t think of it. 

How we would we batch them for delivery? 

We received the names of the schools and Sandrine and Marta were able to help in pulling these into a map. 

We knew the van had a min capacity of 100 devices so Marta did a fabulous job of grouping them into local area of no more than 100. This gave us the number of runs that would need to be made and these were what decided went into each batch for each run. We now knew we had a total of 7 batches to get out. 

How many deliveries could we do in a day? 

We weren’t clear on the number of vans or drivers per day so the worst case scenario was 7 working days, which would take us up to 10th. We had to hope all were delivered and no returns! 

As it turned out, we had two drivers and one van had a capacity of 200 devices. We immediately were ahead of schedule and all went home well smug. 

However, in the following days, a couple of schools weren’t open for delivery and one driver was off sick, so we fell behind but then raced ahead with re-delivering failed first attempts and ended up finishing ahead of schedule! But, we had some last minute additions to deliver on the last day. We had to keep everything crossed there was no hiccups and they all got delivered ok. And that they did.

We had one school (with over 100 devices) that we weren’t able to get hold of so these had to be safely stored in HLT for them to deliver at a later date so we could clear Robert House on the 10th as agreed. 

This was a short and fast paced project to deliver and despite so many unknowns and variables, the team performed amazingly well to get this over the line ahead of schedule and keep on track despite set backs.

Oh, and the Mayor popped by with Cllr Bramble. Which was nice. 

Lies, damn lies and YouTube analytics – Part 1

Or the boring but effective title: What YouTube analytics are telling us about virtual council meetings

TL;DR

  • Online quizzes with your family are no longer fun, more like a war zone
  • Statutory council meetings are taking place online and available in real time to citizens
  • YouTube analytics are revealing things about viewer engagement that suggest we need to rethink the regulations and experiment with how council meetings are led and managed
  • Meetings are viewed asynchronously, rather than in real time
  • Viewers are watching short sections, rather than an entire meeting
  • Engagement across different types of meeting vary significantly
  • Don’t be a rookie analyst – large viewing numbers don’t equate to a successful meeting, its effectiveness or value. 

Back in April – when doing a virtual quiz with our relatives still felt like a novelty not a chore – Parliament passed regulations making provision for statutory council meetings to take place remotely. Councillors – for the first time – could meet online and continue the virtual role of governance and scrutiny. To ensure access and accountability to local citizens, the regulations also stipulated that meetings should be available online and in real time. 

For many, the prospect of delivering council meetings virtually sparked ambition for increased participation. At Hackney, we’ve been live streaming our statutory meetings via YouTube. Early indications show that more people are watching the live stream than would have attended a meeting in the town hall prior to COVID-19. Whether this trend will continue is something we need to assess over the coming weeks. 

In June and July we’re collecting baseline data for all our statutory meetings at Hackney. We are using a combination of qualitative feedback and YouTube analytics. The analytics are revealing things about viewer engagement and behaviour that suggest assumptions in regulations are not reflected in online behaviour, and that the structure and delivery of meetings may need to change to increase participation levels.  

Here’s the health warning. The analytics tell us very little about the inherent value of a meeting or how effectively it was managed. These things need to be measured in other ways. To be clear, a low number of views doesn’t mean a meeting has little value or is ineffective, and popularity should not be taken as a proxy for quality. 

What do we know so far… 

Engagement varies hugely between different types of meeting

This makes it difficult to draw conclusions that are relevant for all meeting types, but over time we expect to see trends emerge for specific committees and commissions. In June scrutiny commissions attracted a higher number of views, both concurrent (in real time) and asynchronous (watching afterwards), as did our planning sub committee. 

People are viewing in their own time, rather than real time

Concurrent views measure the number of people watching in real time. This is small in comparison to the number of people who are watching asynchronously across all meetings. The regulations assume that people would want to watch the meeting as it was taking place, but the analytics we have don’t bear this out. 

People watch short sections, rather than the whole thing

People aren’t watching the entirety of a meeting. This chimes with online behaviour. We tend to skim and skip through online content. We have grown used to bite-sized pieces of information or entertainment. In 2020 it’s an anathema to give your undivided attention to something for three hours. 

Long meetings, agendas, copious reports

Long meetings traditionally have been a bit of a badge of honour – a reflection of their seriousness and gravitas. Lovingly prepared, lengthy reports by Council officers are also held in similar esteem. I’m not seeking to decry or undermine the Herculean efforts of councillors or officers (I’ve written and presented those reports myself). My point is that neither translate well into an online environment. The implications of the previous sentence are huge, more than a blog post can convey or one Council can solve trying to tackle it alone.

Conclusions, thus far

The new regulations are a game changer in some ways and in others not. No-one anticipates a wholesale rollback to face-to-face meetings – virtual is here to stay. The big question is how can it’s value be sustained and meaningful. 

Lifting and shifting a face-to-face process online isn’t the answer. The emergency regulations replicate a process that is over 40 years old – something that wasn’t designed for internet-era culture, unresponsive to short feedback loops, out of step with people’s expectation of service provision and a 24 hour news cycle.

How might we: 

  • Improve the experience for viewers watching asynchronously (watching after the live stream has taken place)?
  • Improve the experience for viewers who skim content and watch short sections?
  • Experiment with structure and delivery of meetings?

We’re continuing to measure our meeting analytics in July and I’ll blog again about the results.