Chelsea and Simon from AWS have been working on a proof of concept sync-back – a key part of the hypothesis we’ve been trying to validate over the last couple of months. It’s been our biggest unknown and hence our biggest risk.
Drum roll please….
The proof of concept has demonstrated that we can sync data from a cloud Postgres database back to our on premises SQL Server database. This is good news. The basic concept involves capturing the queries on our cloud database and queuing them to be replayed in our on prem database on a frequency of our choosing. The design is modular so we’re not relying on a single native tool to do all the heavy lifting.
So, we have taken a big leap forward in our understanding of what’s possible. The input from the AWS professional services team has been great. To move from the realm of the theoretical to the actual, we have to ask is it practicable?
Steve, our data engineer, pointed me in the direction of Richard Feynman’s famous quote: “The rate of the development of science is not the rate at which you make observations alone but, much more important, the rate at which you create new things to test.”
With this in mind, our next step is to get the proof of concept cloud template set up in our AWS environment so our in-house team can run some tests with it. We need to explore:
- How will our on prem SQL database behave when the sync process is applied?
- Does this solution lend itself better to some types of data better than others?
- What’s the best way to handle conflict resolution?
- What happens when we increase the number of simultaneous queries?
- How do we capture information generated after a record insert or update? (Triggers are proving fun)
In a (not so) small personal triumph, I’ve finally managed to coordinate team diaries and we have a show and tell booked in on 10th March. We’ve not been great at doing these in these phase of the project. This will be an overdue opportunity to share our progress with a wider group of colleagues.