A Statsig tradition, Hackathons are quarterly, two-day parties wherein employees have free reign to work on whatever they want.
To some, this means fixing and improving existing stuff. To others, it means building new features that might otherwise be outside of scope.
đď¸ Fun fact: Statsig Sidecar began as a hackathon project.
Letâs take a look back on some of the projects and hacks that Statsig built during our Q3 2024 Hackathon which ended this week.
Note: Unless otherwise stated, please assume all Hackathon projects are simply concepts and have not yet been shipped to end users, and may not be shipped at all.
If you really want us to prioritize the development of a certain project listed in this article, please let us know!
Finally, this is also not a comprehensive list, as itâs only comprised of the hacks weâre allowed to talk about. đ
Presenters: Brock Lumbard, Sedona Duggal, Minhye Kim, Matt Garnes
Summary: Experiment ideas generator is an aptly-named tool that generates ideas for experiments for your website.
It is currently open for all users to try, and was shipped during our hackathon.
To get started, simply enter any website into the search bar. The AI will analyze the website, including scraping source code and taking screenshots to garner a better âunderstandingâ of the site, its purpose, its CTAs and prospective user flows, and so on.
Its output includes a summary, a hypothesis, and a control and test for each experiment.
You can open these experiments directly within your Statsig account, or (coming soon) use Statsig Sidecar to make them happen in real time.
Presenters: Pablo Beltran
Summary: Pabloâs hack allows for Statsig users to automatically generate Statsig projects and data using AI.
For Pabloâs demonstration, he prompted to create âa tech company made by evil villains whose clients are also villains.â
On the backend, this calls out to OpenAIâs SDKs and describes Statsig console API, and ensures that all response conforms to the console API.
From there, it creates some âdummy dataâ to experiment with including experiments, metrics, and other sample data.
The example data was hilarious. Some highlights:
Blackmail metrics
Disaster simulations
Lair upgrades
Presenters: Lior Barak, Jairo Garciga
Summary: AI console search enhances the consoleâs existing search functionality with the power of AI. â¨
You now have the option to ask AI to generate page results to navigate to based on whatever you typed into the search bar.
It will generate a few results that it âthinksâ are appropriate, rather than requiring users to know the exact name of what theyâre searching for.
Presenters: Chong Xie
Summary: âEveryone wants extra coding help so they can build their own kingdom,â jokes Chong.
Codemonkey is meant to be a coding AI that takes open-ended queries and translates them into impactful changes, leveraging the power of LLMs to perform ultimately simple tasks.
In his example, Chong says âMake our codebase better.â Codemonkey then goes through the Statsig directory and feeds each file to the LLM, presenting it with options on what to do with the code.
This is an iterative process. After each iteration, the changes are compiled and run in a test environment. If the changes donât work, they are sent back to the LLM for a redo.
Summary: A lot of information is gained along a userâs experimentation journey that can be helpful to others.
To address this, we created an experiment knowledge bank that sends experiment data to OpenAI with an assistant behind it, which allows users to query for very specific criteria using regular language.
Questions like âShow me an experiment where we did a certain thingâ will now yield intelligent answers.
Presenters: Yuzheng Sun
Summary: Do you ever wish you had more Craig? Same.
Craig is a data scientist and experimentation expert here at Statsig. He's largely responsible for the creation of Statsig Warehouse Native and is generally regarded as an all-around helpful person.
âWhat would Craig sayâ is an AI chatbot trained on our knowledge base, Statsig docs, and anonymized sales call transcripts.
âThe ideal situation is that weâd have a Craig in every conversation,â muses YZ.
For now, YZâs Craigbot is only available for Statsig employees to communicate with on a 1:1 basis, but who knows what integration opportunities will await it in the future?
Presenters: Andrew Huang
Summary: MEx assistant is probably what youâd imagine; Statbot wrapped into Metrics Explorer.
This allows users to ask questions like âWhat exactly does this funnel show me?â directly in the console and receive clear answers.
As of writing, this has not been shipped.
Presenters: Craig Sexauer, Daniel West
Summary: CUPAC is a way to use pre-experiment data to predict what a user will do, but instead of using one variable, it can use N variables.
Still running in SQL, this new calculation method cuts off ~20% of variance, and allegedly uses âreally complicated math,â as evidenced by Craig and Danielâs really complicated math slides.
In the console, this will show live data about how much time was saved on an experiment using CUPAC when users hover over CUPED.
Presenters: Liz Obermaier and Nicole Smith
Summary: âOh no, what happenedâ is a new feature that will allow users to generate tables as a starting point when investigating something that went wrong.
Especially useful for looking at different properties where a metric drilldown is required. If a value changed, you can look at how it changed for specific percentages of your sample population.
User attributes, experiments, and cohorts can also be plugged in for more accurate tracking.
As of right now, Oh no, what happened? works for count, sum, and mean metrics.
Presenters: Brent Echols
Summary: Brent Echols created a unique solution to a common infra problem.
âRight now, Statsig gets results from Mongo and âshoves themâ into Redis, which creates a situation where Redis has to be checked,â Brent explains. âThis isnât good for caching efficiency, as you can only cache things per query. Without Mongo connections, we would have to do a huge offline cycle of caching results into Redis.â
Sigma is a read/write cache that acts as a reimplementation of MongoDB, but is done in memory which grants it greater speed.
The way this workflow works is that Mongo acts as storage, but CRUD operations and listings can be done in Sigma without the need to validate queries.
Presenters: Jay Kodeswaran, Jason Wang
Summary: Say hello to one-shot infra rollouts.
The way the current model works is app owners interface with SDKs, which use Statsig to get results.
App > SDK > Statsig > Results
The Kubernetes operator is a new workflow that adds a second pathway: Kubernetes operator > Statsig > Results.
Jay demonstrated this by using it to change the color of a shape. Directly from Statsig, he added which parameters needed to be changed and started a simple rollout.
Kubernetes operator takes the current deployment, copies it, changes the fields, and then rolls it out.
Jay reckons it will be a âdelightâ for infra people.
Presenters: Cooper Reid
Summary: Cooperâs Slack integration is a way to post experiments into customer conversations via Slack Connect.
Essentially, each experiment can have a webhook URL, which, when dropped into a project, can serve as a changelog which pings Slack every time the project is updated.
Presenters: Alex Coleman
Summary: Using Prophet, an open-source forecasting library, Alex has brought forecasting capabilities directly to Metrics Explorer.
With a little bit of manual tuning, Metrics Explorer can now create forecasts with built-in holiday detection and anomaly detection, and can even be used with a group-by applied, like âDAU by country.â
Presenters: Pierre Estephan
Summary: Pierreâs hack, MEx maps, brings a new visualization type to Metrics Explorer: Maps.
Have you ever wanted to see pageloads grouped by country? With MEx maps, you can view this data as a geomap, pan, zoom, and all the other things youâd expect to be able to do with a map visualization.
Presenters: Eric Liu
Summary: CPU profiling will allow users to see which of their processes are taking the most time.
Traditionally, this is a tiring process for engineers and involves digging around through snapshots, hoping to find one that helps.
With Ericâs changes, Statsig SDKs can now capture profiles. A Statsig agent that runs within our clusters parses and processes and sends the output to Statsig as events, logging all stat traces.
To demonstrate, Eric created a feature called âdo a lot of work,â which was meant to use tons of computing time.
Presenters: Nick Jiang
Summary: Statsig Warehouse Native runs a lot of queries a day. Naturally, some are slower than othersâand then this happens, users want to know why.
Nick added a post-processing step that uploads all job data to SWN, shows some visualizations, and can identify which experiments are the most performance expensive or which ones cost the most time per job, for instance.
This also enables users to see which metrics are contributing to overall loading times, and even es a sample view for all the jobs running in a warehouse native environment.
Presenters: Cat Lee
Summary: The aptly named Statsig Dog Calendar is a calendar that features a different Statsig dog each month.
Designed by our internal brand design team, the calendar will hopefully be a wonderful and welcomed addition to just about every household in the world.
Presenters: Jess Barkley
Summary: Remember Myspace? Jess created an internal tool that looks eerily similar.
On our internal tools interface, users now have their own accounts which include their anniversary date, accomplishments, badges, and other internal tracking information.
First feature request: Let me add emo music to my profile.
Presenters: Katie Braden, Jessie Ong, Maria McCulley
Summary: Remember Spotify Wrapped? Imagine that but for experimentation!
Statsig Wrapped is a way to entertain users by showing them historical experimentation wins and metrics, all beautifully wrapped up into a shareable post. (If youâre okay with sharing your internal experimentation metrics.)
Statsig Wrapped includes different types of awards that users can earn, and has been tested on a handful of beta customers to much delight.
Presenters: Yuzheng Sun
Summary: When asked to recap this hackathon project, YZ told me:
There is an office gaming PC
It is by the window
Thanks, YZ!
Presenters: Pushpenda Nagtode
Summary: Costs by jobs is a handy way of being able to see how expensive specific jobs are.
This is internal-only, but it allows us to more easily track by time and by job all in one dashboard.
Presenters: Brian Do
Summary: Brian made it easier for Statsig developers to make Statsig look great!
Our new design system introduced lots of new visual treats from colors and icons to layouts and themes.
With Brianâs hack, Figma files in a certain format are automatically added to our codebase for use by developers. It also works on images, and is filterable, bringing more pleasing aesthetics to our end users!
This article was meant to give a peek behind the scenes at what our Hackathons actually entail. If youâre a Statsig fan, you probably see the end result when we ship features or post product updates.
This is a special glimpse at how many such features are born.
Thanks for reading!
A short list of reasons why a great experimentation tool is a horrible idea.
How we optimized Pod Disruption Budgets in Kubernetes to reduce resource waste and improve rolling updates for service deployments handling live traffic.
Statsig's AI Prompt Experiments allow you to run experiments for AI-powered products and gain real-time insights into what's working and what's not.
Master data-driven product development with Statsig. Simplify experimentation, make informed decisions, and accelerate your product's growthâall without complex coding.
Debunk the myth that you can never accept the null hypothesis and learn when you should by exploring the key differences between Fisherâs and Neyman-Pearsonâs frameworks.
Use our customizable, detailed cost comparison tool and flexible pricing assumptions to find out which platform reigns supreme.