At Statsig, we recently transitioned to using structured streaming for our ETL. As the number of streaming queries grew, we wanted a centralized place where we could quickly view a snapshot of all our pipelines.
When it comes to monitoring our queries, we are primarily interested in answering three questions:
Structured Streaming queries on Databricks already come with a UI attached to them when they are started, which helps monitor the input rate and processing rate (#1 and #2) at the job level:
These graphs help look at the performance of a specific job, but when you have a lot of queries, it is not practical to individually check each one. Additionally, this UI does not answer question number three: What is the age of the freshest data being processed?
Before monitoring the query, we first need to install the Datadog Agent on the cluster. You can follow the ‘Driver only’ instructions on the Datadog Databricks integration guide. Before running the script provided by Datadog, you need to modify it to set ‘enable_query_name_tag’ to true. This is placed under ‘instances’ like this:
- spark_url: http://\$DB_DRIVER_IP:\$DB_DRIVER_PORT
enable_query_name_tag: true <----
This will tag your metrics with the QueryName you provide, allowing you to view them individually even if there is more than one query per cluster. You also need to enable `dogstatsd` by adding the following two lines to the script:
# Enable dogstatsd
echo "use_dogstatsd: true" >> /etc/datadog-agent/datadog.yaml
echo "dogstatsd_port: 8125" >> /etc/datadog-agent/datadog.yaml
To run the Datadog agent on your cluster, you need to have the install script you generated run as an init script as well as enable streaming metrics and pass in your Datadog API key:
In order to select a specific query when making a dashboard, we need to provide a QueryName for the query to use. This can be done when writing the query:
Input rate and processing rate are automatically tracked for you. However, if you want to know your data freshness, you need to track it using foreachBatch. First, you need to pip install datadog. Then you can track the freshness like this:
from datadog import statsd
def record_freshness(df, epoch_id):
timestamp = df.limit(1).collect()['enqueued_time']
freshness = (datetime.now() - timestamp).total_seconds()
If your query does not use ‘foreachBatch’, you can create a second query that reads the updates from your first query and records metrics:
Now that we have all the metrics we care about being tracked, we can build a dashboard on Datadog. We can track all of these metrics using time-series graphs. Here is a quick guide on setting up a graph.
This is what the dashboard could look like when all the charts are set up:
You can quickly swap to a different $query_name to view graphs for different queries. These dashboards will allow you to ensure that your queries are keeping up with incoming data and track how changes are affecting performance.
Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.
💡 How to decide between leaning on data vs. research when diagnosing and solving product problems Four heuristics I’ve found helpful when deciding between data vs. research to diagnose + solve a problem. Earth image credit of Moncast Drawing. As a PM, data...
Have you ever sent an email to the wrong person? Well I have. At work. From a generic support email address. To a group of our top customers. Facepalm. In March of 2018, I was working on the games team at Facebook. You may remember that month as a tumultuous...
Run experiments with more speed and accuracy We’re pleased to announce the rollout of CUPED for all our customers. Statsig will now automatically use CUPED to reduce variance and bias on experiments’ key metrics. This gives you access to a powerful experiment...
You Can’t Invent Without Experimenting When Amazon launched Home Services, the team was convinced that most people want to schedule home installations in the mornings, evenings, or weekends. This naturally constrained the number of available time slots, and...
Training your team to make independent decisions Image Courtesy: The New Yorker “It was like the debate of a group of savages as to how to extract a screw from a piece of wood. Accustomed only to nails, they had made one effort to pull out the screw by main...
Photo by Joshua Hoehne on Unsplash By now, most people realize that when they open Facebook or Instagram on their phone, their experience is very different than the person next to them. It goes deeper than just the content that you see, and the ranking...