How Celtra uses Spark and Databricks for event analytics and ad-hoc queries - Seguridad Mania.com - España y América Latina

Celtra provides agencies, media suppliers and brand leaders with cross-screen HTML5 technology for brand display advertising. We bring simplicity to the increasingly complex world of digital advertising, allowing customers to deliver more relevant and engaging data-driven dynamic creatives with ease, great reliability, and deeper insights. The data we collect isn't just large in size, but also very complex and diverse. We collect and track a large number of events that happen to an ad during an impression, but we go beyond simple counting and group events into sequences (also known as sessionization), which allows us to do more complex types of analytics (e.g. “funnel analytics”). Analyzing the data is instrumental to our mission of helping customers power the creative side of their campaigns, so it's key that we can dig in easily and quickly. To do that, we use Databricks Cloud, because it simplifies our Spark deployments, increases collaboration via results sharing and allows for faster turnaround. It means our data scientists and analysts can focus on drilling down into the raw events, quickly verify hypotheses and visualize results, all of which would be very difficult if not impossible one year ago. We devote significant attention to making sure our ads work and perform well. As we operate in a complex ecosystem where a lot of data can be construed as dirty, a lot of our time is also spent on debugging and investigation. The ability to easily connect disparate data sources such as ELB access logs, raw tracking events, operational data and reportable metrics, all of which reside in different databases, on different systems and are queried using different protocols is an important factor. We will explain: - Why we use Spark to power our production analytics pipeline - How we use Databricks Cloud as a "logical data warehouse" to connect to diverse data sources - Some ex. of using Databricks Cloud to do event analytics and best practices from our experience