Multi-armed Bandit Experimentation

What is a Multi-armed Bandit Experiment?

Problem Intuition

Let’s imagine you’ve decided to hit up the casino and try your luck at the slots. You don’t know which slots are best, but you have reason to believe that some of the slot machines are better than others. You only have a limited amount of money to play with, so you want to play each of the machines enough to learn which is going to be best (explore), and then as you learn, play the best machines most frequently to maximize your return (exploit). Choosing which machine to play is referred to as the exploration / exploitation dilemma.

Read More

Tags: experiments, ruby, engineering, kafka, dynamodb

Turning Designers Into Data Nerds

At ezCater, we believe that designers should care deeply about using data at all stages of the design process. Whether qualitative or quantitative, it helps to inform and empower our design decisions.

Data is a powerful tool that allows us to:
  1. Validate that we are solving the right problem
  2. Test small assumptions to get to the right solution with quantifiable impact
  3. Understand what the business cares - and should care - about, which influences design strategy

Product manager Lindsay Matthews and I gave a talk at the recent UXPA Boston conference about all the ways that designers can use data to be more effective. Learn more in the slides below:

Read More

Tags: experiments, women_in_tech, ezcater, design

Measuring AB Tests: SQL for p-value graphs in Redshift

So, in deciding when an AB test is actionable using p-value graphs we saw some cool charts that looked like:

Read More

Tags: experiments, data warehousing, redshift, sql

Deciding When to Call an AB Test: p-value Graphs

So, you've read about how to get your event data in Redshift, you've read about the experiment exposure event, you've got your AB test ready to run, now... how do you decide if you've got a winner?

Read More

Tags: experiments

AB Testing at ezCater Part 2: Tracking Experiments & The Exposure Event

What events should I fire to track an experiment?

A fabulous question and one with a number of different answers. During my travels I've found that there are 4 basic ways to try and tackle the problem

Read More

Tags: experiments

AB Testing at ezCater Part 1: Event Taxonomy & Warehousing

Experiments! AB tests! On one hand it seems like everybody's doing it and on the other hand I regularly get the impression that a lot of people have found it to be more cumbersome than it’s worth. Worse, some blog posts I see imply that it’s leading folks down a path of micro-optimizations based on totally insignificant amounts of data. So what does it look like to have an AB testing framework that actually works?

Read More

Tags: experiments, data warehousing

About ezCater

We're the #1 online – and the only nationwide – marketplace for business catering in the United States. We make it easy to order food online for your office. From routine office lunches to offsite client meetings, from 5 to 2,000 people, we have a solution for you. ezCater connects business people with over 50,000 reliable local caterers and restaurants across the U.S.

ezCater is hiring!

We’re always looking for highly skilled full stack and iOS engineers to help execute our technology goals while riding this rocket ship of growth. Our people are terrifically talented straight-shooters who love what they do. And everyone is generous and kind, too — we have a strict no jerk policy.

View ezCater Opportunities

Recent Posts

Subscribe to Email Updates