Audiense evolves its social media timeline with Tinybird

Audiense is a one-stop shop for audience research needs powered by social data, giving marketers and consumer researchers intelligence for audience and marketing strategies by capturing high-value, high-granularity social data and connecting it with any tech stack.

Audiense maintains a suite of social data providers to power their consumer insights. One of these providers is X, as Audiense is an enterprise customer of the well-known platform's data division. This relationship with X provides Audiense with access to massive amounts of data, including billions of posts and interactions from X users over the years.

Audiense goes the extra mile to capture this data in a compliant, privacy-first manner, leveraging it to extract meaningful, anonymized consumer behavior insights for its customers.

Audiense stores its X data in a proprietary database containing trillions of data points. They have maintained this database since 2014, updating it daily with new information about almost every single X user in existence. This massive database sits at the heart of their analytics service, giving their users access to high-granularity consumer trends on their target audiences and communities, a feature that Audiense calls "X Evolution."

Up until recently, Audiense stored this data in a massive MySQL cluster with trillions of rows sharded across 8 tables totaling over 20 terabytes, and they were adding new rows every day.

While the company and its users benefited from this treasure trove of X data, they started to struggle with performance and cost associated with it. The MySQL cluster itself cost them $3,000 a month just to maintain, and performance began to suffer under the on-demand queries that Audiense's growing user base was making against the data.

Audiense had a choice to make. They could either assess alternative technologies and make a migration, or they could reduce the load by deleting old data points and reducing the granularity of their dataset.

They considered moving the data to Snowflake, but it was overkill for what they needed and would be difficult to engineer for the low-latency, high-concurrency requirements of user-facing analytics.They also considered some cloud time series databases, but the costs for storage and compute were too high, making a migration unjustifiable.

Left with few technological options, all of which were undesirable, Audiense considered deprecating X Evolution. They understood the value of this carefully curated data set and the opportunities that it could create in the future, but they were unsure of how to cost-effectively maintain it.

Audiense needed to discover a new angle; a fresh solution that would allow them to retain their prized dataset, develop new functionality on top of it, and fit within their budget.

After running a proof of concept with Tinybird, the data team at Audiense had the "aha" moment they were looking for.

They could move the data from MySQL to Tinybird, completely offloading their massive MySQL costs while shifting the analytical workloads to a platform purpose-built for serving the user-facing analytics they wanted to build.

During the proof of concept, Audiense's data team discovered how easy it was to write SQL queries on top of the dataset, and then instantly publish those queries as low-latency, scalable APIs.

The Audiense data team chose to move the data from their MySQL cluster over to Tinybird to support their X Evolution feature. Migrating a 20 TB dataset is no easy task, however, so the team at Audiense turned to Tinybird's Customer Support team, which is made up of experienced, professional data engineers.

The team at Tinybird assisted Audiense in exporting the data from their MySQL cluster as a series of Parquet files, which they were able to then import into Tinybird.

Audiense's adoption of Tinybird is part of a broader initiative within the company to shift to a streaming data platform. The data team and the company at large recognize the importance of real-time, and they're committed to creating internal infrastructure and systems that will capture the data they need and make it available to end users much more quickly.

Tinybird sits at the center of the platform, ingesting X updates as they stream in and exposing SQL-based metrics as real-time APIs that power the Audiense user experience.

Audiense uses Tinybird's Events API, a simple but scalable HTTP streaming endpoint, to stream change data from their X sources into Tinybird.

Once the data lands in Tinybird, they use a Materialized View to build pre-aggregated metrics that get calculated incrementally, as new data is ingested. On top of that Materialized View, they've built a parameterized API Endpoint using SQL to power their user-facing features.

They can pass time- and user-based filters as query parameters within their frontend code and return specific X trends based on user activity in the platform.

After making the transition to Tinybird, they've not only reduced overall storage from 20 TB to less than a terabyte, but they're maintaining static storage amounts by discarding unnecessary stale data, instead maintaining aggregate states in a much smaller Materialized View.

And with Tinybird's API layer, they can easily handle fluctuations in customer usage while still maintaining the sub-second API latency that promotes a positive user experience.

For almost a decade, Audiense has been sitting on an incredibly valuable data asset: detailed consumer insights about all X users and how they've evolved over the last 10 years.

Before Tinybird, they suffered under an intractable MySQL cluster that had grown too big to manage. They considered deleting data or hiring more team members to set up and maintain new infrastructure.

But after switching to Tinybird, they have fully tapped into the value of this asset, discovering new ways to leverage their dataset to explore, test, and deploy revenue-generating use cases.