Data and product teams always work smoothly; yeah, sure.
Why is it so hard for data and product teams to work together?
Their conversations go something like this:
Product team: We need to build this insights dashboard in our console and mobile app for customers to see the outcome of XYZ by the hour and be able to slice and dice by A, B, C, D, and E.
Data team: We only have that data monthly, and you can only slice and dice by C, D, or E. How important is this? How fast do you need to query it?
Product team: We need to be able to query super-fast (sub-second), and it is crucial to meeting our goal, so it is super important.
Data team: We can't do it. File a ticket with the requirements, and we can see where it falls in the priorities.
The product team is frustrated because they can't build the product they envisioned.
The data team is frustrated because they are constantly bombarded with high-priority tasks that are impossible to prioritize against each other.
It is not that the product team is unreasonable or the data team is incapable. The product team is looking to build what's best for their customer. The data team seeks to inform decisions as effectively as possible across the organization.
To understand the root of the problem, we must dig deeper. There is a fundamental tradeoff in data between flexibility, performance, and cost-effectiveness.
You can have two data characteristics above at the same time, but not all three.
You can pre-aggregate data which will give you performance and cost-effectiveness, but you will have tragic flexibility limitations.
You can beef up compute to query as fast as possible in order to keep the performance and flexibility, but it will come at a steep cost.
You can have all the flexibility in the world and be cost-effective, but it will likely get very slow queries, even on a small scale.
The root of the problem is not the tradeoff itself but who makes the tradeoff.
In the previous conversation, the data team made a tradeoff that sacrificed flexibility. At the time, it was the right tradeoff. They are also the team empowered and equipped to make such a tradeoff.
Unfortunately, that tradeoff did not work for what the product team needed. It would be surprising if it did, as it was made a long time ago with other requirements in mind.
Now imagine the product team has a huge budget. Their big budget can’t get them the performant and flexible product they want because it is blocked by the data team's ability to prioritize it. The data team's resources are very constrained, and the organization has many other priorities which make the product’s team ask fall below the line every quarter and therefore it never gets done.
Unlike the data team, the product team is not empowered or equipped to make this tradeoff.
The root of the problem is that the team that is closest to the customer is not empowered to make and execute the tradeoff.
The data team gets pulled in a million different directions and is not in a position to prioritize what's best for an individual product team.
The solution is for product teams, who are closer to the customer experience, to be empowered and equipped to make the tradeoff. How does an organization get there without each product development team having data engineering resources?
Our goal at Propel is to empower product development teams to build in-product analytics quickly and easily. We want product teams to be able to make the tradeoff they need to build their product. Propel does this by:
Our new File Upload Data Source. The new functionality lets you upload data files directly to Propel to power your user-facing analytics.
You can now power customer-facing analytics use cases such as insights dashboards, product usage reporting, or analytics APIs with Parquet f