I know the customer-facing dashboard you want to build. I know that your customers have asked for it. It's been on your roadmap for at least a quarter or more. The dashboard you want to build contains four KPIs in four boxes across the top of the screen. I know that below those boxes is a time-series graph and, to the right, a leaderboard. If you bring this to your engineering team, I know how they will want to build it.
Usually, this goes one of three ways.
We can surely deliver in-product dashboards to our customers with our costly business intelligence software that we're using to provide internal reports. Davis Treybig, from Innovation Endeavours, wrote a great blog post on why this is a woefully inadequate solution, so I'll defer to his prose and encourage you all to head there to read more.
We can query the production database!
All that beautiful data is just sitting there in all its normalized glory. The backend team can build a new service to communicate directly with the database. The front-end team can write a few SQL statements with a smattering of well-intentioned JOINs. They'll send those to the nifty new service, and presto! You'll be rendering counters and graphs to all your heart's content.
I'm oversimplifying here because any team worth its operational mettle would ensure the service has observability, can page when things go wrong, and is well-tested and hardened for the rigors of production. They would, of course, ensure that any SQL received from the frontend in support of the dashboards wouldn't take down the production database if, by chance, that query took too long or locked a table or two. Better yet, the backend team would ensure only a small set of tested, performant queries are allowed to execute.
Introducing analytical workloads to your production databases - which were designed specifically for fast transactional writes - tends to inflict pain upon your end users, especially when those long analytical queries start hogging up database resources. If you enjoy the 2 AM cacophony of “somethings broken, somethings broken,” then, by all means, ship it.
We have a data pipeline (e.g., Kafka or Kinesis)!
Your data is already in motion, traversing the inner pathways of your cloud provider. The backend team just needs to land it in a customer-facing dashboard.
However, in all likelihood, your existing team isn’t running an analytical data store like Apache Kudu or ClickHouse (we used Elasticsearch at Twilio). If you have the headcount, you could hire backend engineers to spin up your analytics cluster; otherwise, you’ll pull from your existing humans to stand this up and carry the operational burden. Once your pipeline is set up, depending on your volume of data, you’ll most likely want to pre-aggregate the data to achieve predictable read performance.
So let’s build out another set of services to run Spark or an equivalent technology that will generate the buckets of aggregated data. Now, we’ll need an API to front the newfound insights stored in our services mentioned above. Node, Java, Go, take your pick: design your API and connect it up! Bring in the frontend folks to create those pretty analytical visuals that your UX designer vetted with the customer focus group and get those dashboards some data.
Reflect
Let's pause for a second and quickly summarize the last few months of your efforts. There is a new team in the org chart, two or three new backend services, a couple of new on-call rotations, an API, new SLOs/SLIs, QA cycles, UX/UI, and with that, you’ve shipped the analytics, and dashboards your customers have been asking for!
Is everyone excited to start building now? It’s not hard to understand why so many Product Managers we’ve talked to over the last two years have had to continually postpone building the in-product dashboards their customers so desperately want. Engineering leadership is abhorrent in taking on the additional backlog ceremonies and operational burdens. Organizationally, it’s a costly endeavor of both time and money, and when the only option is to “build”, the budget item to move forward will likely be at the bottom of the list.
Repeat
In multiple senior engineering leadership roles throughout my career, I’ve been part of building and shipping the approaches described above. I’ve written the job descriptions for hiring the headcount and slogged through the toil that came with delivering these critical roadmap items. During my tenure at Twilio, my team delivered the first “insights” data product that allowed our customers to understand how their WebRTC voice clients were performing. Internally we’d been using all of that data for network performance debugging and to provide answers to our customers when they opened tickets with our customer support team. As minutes increased and more critical use cases were built on the voice platform, our customers demanded that the same data be available through the console so they could self-diagnose call quality issues.
Shortly after launching, a number of the other product lines within Twilio, like Chat, Messaging, and Video, all wanted to ship their own version of insights. However, there wasn’t a platform they could buy or leverage internally; just like the teams in my organization, they had to build their own services similar to what was described above. The teams had to opt into the complexity and expense of building and operating the services to deliver insights to their customers. This was not the core competency for many teams, yet here they were: hiring and “drawing the owl”, as we liked to say. This “forced acceptance” never sat well with me.
Build Different
With Propel, the toil described above no longer needs to be suffered; the proverbial can no longer needs to be kicked down the road. With Propel, you can sleep well, knowing there won’t be any new services to operate in production and no expensive new employees to compete for. The data you’ve painstakingly landed in your warehouses and your existing team is all you need, freeing you up to get on with the rest of your roadmap.
Welcome to the world of serverless analytics - one that simplifies your architecture, where you can build faster, and deploy in days instead of months.
Like an extension of your team, you can do more with less, and your customers get the self-service magic they’ve been looking for. One Head of Product we spoke to said it well - “Propel is making the impossible possible.” It’s time to change how teams build and deliver customer-facing analytics. Your customers have been asking for it - it’s time to build different.
