This is a story of how we built a working BI tool in one sprint.
It shows how easy it is to screw up if you don't ask questions (even when building for yourself). It also shows that you don't always need to solve all problems. Creating a tool that simplifies the process can do the trick (and then some). Assisting the human brain rather than replacing it was key to the success of this project
Early in 2018, we were a couple of months into a presidential campaign. We had opened 20 or so regional offices with over 60 to go. The main goal of the early stages of the campaign was to register at least 350,000 people (but no more than 7,500 per region) to support the nomination.
We needed a way to track how many supporters we had registered, their locations, and if we had enough time.
Create an easy-to-use BI tool that will allow campaign management to track campaign KPIs, identify trends, and make data-driven decisions.
This is a happy story about how we almost screwed up (bigly). As a team, we had experience of working together on a bunch of smaller political campaigns. And for a long time, we’d been discussing the need for a proper campaign CRM down the line.
Ideas were circulating, we routinely discussed random features we definitely knew we wanted to build. Wireframes were occasionally passed around, everyone seemed to have some kind of shared vision of a system. The scary part was that the system looked like an extremely complex product to build.
Email and SMS campaign management tool, supporting time and action-based chains of emails and texts
Audience segmentation based on geography, user actions, previous engagement and communication
Campaign performance tracking
As a team, we were sold on this vision. But purely out of curiosity, (and as part of my UX-research self-training) I decided to do a set of formal user interviews.
To better understand our needs and to be able to prioritize possible features, I set up a series of user interviews with people who would potentially use our analytic tool. Because only a few specific roles would have access, I was able to interview almost every single user.
in charge of vision and strategy
At any given moment I want to see how the campaign is performing overall. What are our current KPIs. What regions are performing particularly well or poorly. What cities look the most promising for the next campaign office. I’m ok with ugly tables, just show me the numbers.
I want us to track all interactions and any engagement with supporters and volunteers so that we can have targeted campaigns later.’
responsible for finance and operations
I need one place where campaign management can see the current overall state of a campaign at a glance.
If there is a crisis, I want to know about it. If some of the regional offices are failing, I want to know.
I want to see how many real signatures we can expect to get in each city. It would be nice to track the budget and its conversion to our key metrics (registrations and signatures). But I doubt we can build it quickly enough.
I’m ok with raw data and numbers.
works in HQ, responsible for 10-15 regional campaign offices
I don’t know. I probably want to know how many people signed up in each region, how many volunteers we have and if we can contact them
works on the ground, in charge of a regional campaign office
I want to know how many people signed up so that we can contact them and convert them to volunteers or donors
in charge of outreach and engagement
I want to know how our outreach campaigns affect the key metrics.
If we do local context campaigns, do they perform better than federal for specific regions.
Can regional campaigns affect the federal context?
yep, the candidate
I want to understand what works and what doesn't. What actions increase the amount of website registrations and what actions don’t. I want you to be able to tell me this. This is the only thing I care about
As much as we wanted to have a mighty CRM, it was more useful and urgent to have a dashboard with key metrics and a system that could track campaign activities and how they affect the number of registered supporters.
One-stop place to track campaign performance
Understand how various campaign activities affect key metrics (number of people registered on the website, number of volunteers and donors)
After we all agreed on the updated product vision and set of questions, we had one more problem to figure out — scheduling. Political campaigns tend to be messy to manage at the best of times. Being in the very beginning of a very aggressive political campaign in a very aggressive environment while juggling a bunch of projects can be overwhelming.
We knew that the task at hand called for focused work and ideally some headspace, but it was hard to imagine any of that happening at HQ.
Track overall campaign state → Track the key up-to-date metrics
Track performance of campaign offices → Ability to segment key metrics by region (and or groups of regions), ability to easily see if we have a working campaign office in the selected location or not
Predict total amount of expected signatures from each region → Show the number of confirmed and unconfirmed registrations, come up with some prediction model
It was a time-restricted project for an internal product with only a handful of potential users. We had only one chance to make it happen. We all agreed that our main goal was to deliver a usable product by the end of the sprint.
The dashboard showing the metrics would be our minimum, and any ‘analytics’ on top would be a great success. For the sake of speed, we also agreed on two principles: ad-hoc development, and 0 additional investments in UI polish.
Jumping ahead, these wireframes were our final design mocks. The rest was built ad-hoc; we were sitting in the same room with our CTO and pairing on styling and product issues. Below I'll briefly describe our solutions for each requested feature.
We had 3 main metrics to track: the number of fully verified registrations on the campaign website, the number of registered volunteers, and the number of emails collected.
Showing the chart tracking our key metrics over time was more or less easy (though we had to overcome some technical challenges).
We went through some back and forth trying to understand what visualisation would be the most useful and settled on combining two charts:
A cumulative line graph showing metric growth over time
Daily real-time registration charts
With these charts we were able to see the general state of the campaign (the number of registrations) and the overall trend (if the registration rate is slowing down or speeding up) as well as make rough predictions if we had enough time. It was also easy to spot exact days/weeks when something happened that caused an unusually high or low number of registrations.
As you can see, we also added a bonus feature: the ability to select a specific interval to see the exact amount of registrations during that time (to estimate the impact of certain events).
To track the performance of campaign regional offices we added the ability to filter all data by region. This helped us to rate their performance, and to identify the best locations for future offices.
Just showing the number of registered supporters would have drawn a far too optimistic picture. For a more realistic prediction, we showed
Total number of registrations
Total number of registrations per region
Number of fully verified supporters (email, phone number and address confirmed) who lived in a city where we had a campaign office (Russia is huge and we had a lot of supporters who lived far away from campaign offices)
To rank suggested locations we simply sorted the regions without campaign offices by the number of registered supporters. To push it a little further, we added lists of best and worst-performing offices. Based on this data, we decided when and where to close or open new campaign offices.
Figuring out the relationship between campaign events and the key metrics was the most challenging part of the system to build (or even to comprehend).
Every single day A LOT was happening. Even when we discussed possible ways of creating this feature, it sounded overwhelming: it was extremely hard to track all relevant events (online and offline), not to mention trying to analyze their possible effect on specific metrics.
Finally, we decided that we would leave two major actions up to humans:
inputting relevant events
making conclusions
Our product would simply show the timeline with key metrics and the list of events. To make it easier to draw conclusions, we added the ability to change the timeline scale and the ability to select a specific time period. This allowed users to view only relevant events happening at a specific time + corresponding changes in metrics.
Screencast of the real product in action (sorry for the low image quality)
To make it easier to tag events, I came up with a way to categorise all types of events that might be interesting to us and created a set of tags and instructions.
The key was not to build a ‘smart’ analytic tool that could track and connect every event to its measurable outcome. Even if such technology existed we had no time to implement and fine-tune it to make it useful. Luckily we had access to something better — human brains. Not trying to substitute the human brain with technology, and allowing human intelligence to be part of our product became our silver bullet.