Group Sequential Testing
Group Sequential Testing
Group Sequential Testing
Group Sequential Testing (GST) allows us to stop or ship an A/B test early when there is a large effect, while also retaining the ability to detect smaller effects within a reasonable timeframe.
Group Sequential Testing (GST) allows us to stop or ship an A/B test early when there is a large effect, while also retaining the ability to detect smaller effects within a reasonable timeframe.
Group Sequential Testing (GST) allows us to stop or ship an A/B test early when there is a large effect, while also retaining the ability to detect smaller effects within a reasonable timeframe.
This approach combines elements from the modified Sequential Probability Ratio Test (mSPRT) and the classical fixed-horizon testing into a best-of-both-worlds method which is ideally suited for online controlled experiments at scale.
This approach combines elements from the modified Sequential Probability Ratio Test (mSPRT) and the classical fixed-horizon testing into a best-of-both-worlds method which is ideally suited for online controlled experiments at scale.
This approach combines elements from the modified Sequential Probability Ratio Test (mSPRT) and the classical fixed-horizon testing into a best-of-both-worlds method which is ideally suited for online controlled experiments at scale.
Unlike mSPRT, Group Sequential Testing does not require a drastic increase in sample size to be able to detect small effects. And unlike fixed-horizon A/B testing, Group Sequential Testing does not require one to wait until the end of the experiment to make a decision. An A/B test using GST requires pre-determining the maximum sample size with a power calculation based on the minimum effect of interest, as well as the number of analyses to be conducted throughout the duration of the experiment. Our intuitive user interface guides users through this process in an easy way.
Unlike mSPRT, Group Sequential Testing does not require a drastic increase in sample size to be able to detect small effects. And unlike fixed-horizon A/B testing, Group Sequential Testing does not require one to wait until the end of the experiment to make a decision. An A/B test using GST requires pre-determining the maximum sample size with a power calculation based on the minimum effect of interest, as well as the number of analyses to be conducted throughout the duration of the experiment. Our intuitive user interface guides users through this process in an easy way.
Unlike mSPRT, Group Sequential Testing does not require a drastic increase in sample size to be able to detect small effects. And unlike fixed-horizon A/B testing, Group Sequential Testing does not require one to wait until the end of the experiment to make a decision. An A/B test using GST requires pre-determining the maximum sample size with a power calculation based on the minimum effect of interest, as well as the number of analyses to be conducted throughout the duration of the experiment. Our intuitive user interface guides users through this process in an easy way.
In the case of a 30-day experiment where daily data review for decision-making flexibility is desired, the test can be configured with 30 analysis points. While this may slightly reduce statistical power compared to fixed-horizon testing, it greatly speeds up decision-making, leading to 20% to 80% faster decisions on average, as significance is commonly reached before the full sample is collected.
In the case of a 30-day experiment where daily data review for decision-making flexibility is desired, the test can be configured with 30 analysis points. While this may slightly reduce statistical power compared to fixed-horizon testing, it greatly speeds up decision-making, leading to 20% to 80% faster decisions on average, as significance is commonly reached before the full sample is collected.
In the case of a 30-day experiment where daily data review for decision-making flexibility is desired, the test can be configured with 30 analysis points. While this may slightly reduce statistical power compared to fixed-horizon testing, it greatly speeds up decision-making, leading to 20% to 80% faster decisions on average, as significance is commonly reached before the full sample is collected.
Importantly, Group Sequential Testing does not increase the false positive rate, and it can accelerate experiments by up to two times compared to the traditional mSPRT approach used in other A/B testing tools on the market.
Importantly, Group Sequential Testing does not increase the false positive rate, and it can accelerate experiments by up to two times compared to the traditional mSPRT approach used in other A/B testing tools on the market.
Importantly, Group Sequential Testing does not increase the false positive rate, and it can accelerate experiments by up to two times compared to the traditional mSPRT approach used in other A/B testing tools on the market.
Try ABsmartly on A PoV
You can acquire a 60-day PoV (Proof of Value) to explore the features and experiment with the platform before committing to an annual subscription.
With ABsmartly, teams embrace experimentation
A/B Smartly helped clean up our AB testing process in many ways. It reduced the number of tools we used and gave us a simple SDK to consume within our applications.
Kyle P.
Senior Software Engineer
"It's a game changer for us"
We are very serious experimenters and come from backgrounds where tens of billions of dollars of value were created through enterprise-grade UX experimentation programs.
Scott Lake
President Pharmacy
It scales for many experiments and goals/metrics, in isolated environments. The SDK is easy to work with (frontend and backend), and the Full-on mode saves time for winning experiments.
David R.
Software Engineer
"We love A/B Smartly"
We moved from Google Optimize to A/B Smartly so that we could have full access and control of our data. We also want the ability to do server side/backend tests. We went with the onprem option and set up the service in our AWS account, the system has been running smoothly ever since.
Mike Rizzo
Principal Architect
A/B Smartly is easy to configure; It has a great dashboard that offers valuable insights at a glance. It helps me quickly set A/B experiments and get timely feedback on our hypothesis.
Vince V.
UX Designer
There's a lot to like about AB smartly; if I had to name it, it is optimized for high throughput: many experiments, many goals, multi-variant. The product covers everything I need to do currently without needing to solve anything myself or find ways of getting around it. It doesn't take a lot to get going and the team is extremely helpful.
Peter C.
Sr. Director of Engineering
Running A/B Smartly in our Google Cloud infrostructure side-by-side with our self-built data solutions gives us the best flexibility and we foresee AB Smartly as a platform that will scale with us for years to come.
Sander Olislagers
Head of Product Development & Marketing
The A/B Smartly platform is really easy to integrate within your current product/codebase and the support from the team is unsurpassed.
Stefan B.
CTO
With ABsmartly, teams embrace experimentation
A/B Smartly helped clean up our AB testing process in many ways. It reduced the number of tools we used and gave us a simple SDK to consume within our applications.
Kyle P.
Senior Software Engineer
"It's a game changer for us"
We are very serious experimenters and come from backgrounds where tens of billions of dollars of value were created through enterprise-grade UX experimentation programs.
Scott Lake
President Pharmacy
It scales for many experiments and goals/metrics, in isolated environments. The SDK is easy to work with (frontend and backend), and the Full-on mode saves time for winning experiments.
David R.
Software Engineer
"We love A/B Smartly"
We moved from Google Optimize to A/B Smartly so that we could have full access and control of our data. We also want the ability to do server side/backend tests. We went with the onprem option and set up the service in our AWS account, the system has been running smoothly ever since.
Mike Rizzo
Principal Architect
A/B Smartly is easy to configure; It has a great dashboard that offers valuable insights at a glance. It helps me quickly set A/B experiments and get timely feedback on our hypothesis.
Vince V.
UX Designer
There's a lot to like about AB smartly; if I had to name it, it is optimized for high throughput: many experiments, many goals, multi-variant. The product covers everything I need to do currently without needing to solve anything myself or find ways of getting around it. It doesn't take a lot to get going and the team is extremely helpful.
Peter C.
Sr. Director of Engineering
Running A/B Smartly in our Google Cloud infrostructure side-by-side with our self-built data solutions gives us the best flexibility and we foresee AB Smartly as a platform that will scale with us for years to come.
Sander Olislagers
Head of Product Development & Marketing
The A/B Smartly platform is really easy to integrate within your current product/codebase and the support from the team is unsurpassed.
Stefan B.
CTO
With ABsmartly, teams embrace experimentation
A/B Smartly helped clean up our AB testing process in many ways. It reduced the number of tools we used and gave us a simple SDK to consume within our applications.
Kyle P.
Senior Software Engineer
"It's a game changer for us"
We are very serious experimenters and come from backgrounds where tens of billions of dollars of value were created through enterprise-grade UX experimentation programs.
Scott Lake
President Pharmacy
It scales for many experiments and goals/metrics, in isolated environments. The SDK is easy to work with (frontend and backend), and the Full-on mode saves time for winning experiments.
David R.
Software Engineer
"We love A/B Smartly"
We moved from Google Optimize to A/B Smartly so that we could have full access and control of our data. We also want the ability to do server side/backend tests. We went with the onprem option and set up the service in our AWS account, the system has been running smoothly ever since.
Mike Rizzo
Principal Architect
A/B Smartly is easy to configure; It has a great dashboard that offers valuable insights at a glance. It helps me quickly set A/B experiments and get timely feedback on our hypothesis.
Vince V.
UX Designer
There's a lot to like about AB smartly; if I had to name it, it is optimized for high throughput: many experiments, many goals, multi-variant. The product covers everything I need to do currently without needing to solve anything myself or find ways of getting around it. It doesn't take a lot to get going and the team is extremely helpful.
Peter C.
Sr. Director of Engineering
Running A/B Smartly in our Google Cloud infrostructure side-by-side with our self-built data solutions gives us the best flexibility and we foresee AB Smartly as a platform that will scale with us for years to come.
Sander Olislagers
Head of Product Development & Marketing
The A/B Smartly platform is really easy to integrate within your current product/codebase and the support from the team is unsurpassed.
Stefan B.
CTO
Run and discuss experiments at scale.