Home / Blog / Article
Pipeline Automation · Aug 12, 2026
A/B Testing Infrastructure for Mobile Game Features
Most mobile studios run their first A/B test manually: two build variants, a spreadsheet tracking which players got which version, and a lot of hoping the sample split was actually random. It works exactly once, for exactly one test, and then the team either builds real experimentation infrastructure or quietly stops testing because manual variant management does not scale past a handful of concurrent experiments. Here is the infrastructure that lets a LiveOps team run tests continuously instead of as one-off projects.
Server-Side Bucketing, Not Build Variants
Shipping separate app builds per test variant is the mistake that kills experimentation velocity before it starts, because every new test then requires a new build, a new store submission, and a new review cycle, the exact bottleneck LiveOps content pipelines exist to avoid. The fix is server-side bucketing: a single client build that queries a configuration service on launch, receives a variant assignment for every active experiment, and adjusts behavior based on remotely delivered config values rather than compiled-in branches. This is what makes it possible to launch, monitor, and conclude a test entirely without a store resubmission.
Assignment Consistency: One Player, One Variant, Always
A player bouncing between experiment variants across sessions invalidates the test entirely, because you can no longer attribute a behavior change to the variant they experienced. Assignment has to be deterministic and sticky: we hash a stable player identifier together with the experiment ID to produce a consistent bucket assignment that survives app restarts, device changes tied to the same account, and does not shift if the experiment’s traffic allocation percentage changes mid-test. Consistent hashing rather than a database lookup per session also means bucketing works correctly even during a brief connectivity gap, since the client can compute its own assignment locally once it has the experiment’s configuration.
Sample Size And Statistical Power Before Launch, Not After
The question we ask before any test ships is not “is this interesting to test” but “do we have enough daily active users in the target segment to reach statistical significance in a reasonable timeframe.” A test on a feature only 2 percent of the player base ever encounters, run at a standard significance threshold, can take months to produce a conclusive result on a mid-size player base, by which point the test is no longer answering a question anyone still cares about. We calculate required sample size and expected test duration before allocating any traffic, and we kill or redesign tests that cannot reach significance within a useful timeframe rather than letting them run indefinitely as background noise.
Guardrail Metrics That Auto-Halt A Bad Variant
An experiment testing a monetization change can accidentally tank retention, and an experiment testing a retention change can accidentally tank revenue, and neither should be allowed to run to full statistical significance if it is actively damaging the metrics it was not designed to measure. Every test we configure includes guardrail metrics, typically day-one retention and revenue per user, monitored independently of the test’s primary success metric, with automatic traffic reduction or full halt if a guardrail crosses a predefined danger threshold. This turns a potential week-long incident into an automated same-day correction, without requiring someone to be actively watching the dashboard the moment a bad variant starts hurting real players.
Experiment Configuration As Versioned, Auditable Data
Experiment configs, which variants exist, what traffic percentage each gets, what config values each variant sends to the client, need the same rigor as any other production configuration: version-controlled, auditable, and rollback-capable. We store experiment definitions in the same versioned configuration system that drives LiveOps content, not in a one-off admin panel disconnected from the rest of the pipeline, specifically so a bad experiment config can be rolled back with the same tooling and the same speed as any other production content mistake, rather than requiring an emergency engineering fix.
Avoiding Interaction Effects Between Concurrent Tests
A studio running real experimentation infrastructure quickly wants to run more than one test at a time, and that introduces the risk of interaction effects, a UI change test and a pricing test both touching the same purchase screen, for instance, where the combined effect of both variants is not simply additive. We tag every experiment with the game surfaces and systems it touches, and the bucketing layer checks for surface overlap before allowing two experiments to run concurrently on the same segment, forcing either sequential scheduling or an explicit interaction study rather than silently letting two tests contaminate each other’s results.
An A/B Testing Infrastructure Checklist
Before treating an experimentation system as production-ready, confirm: variant assignment happens server-side via remote configuration, not separate app store builds; assignment is deterministic and sticky per player across sessions and reconnects; required sample size and expected test duration are calculated before any test launches, not discovered mid-test; guardrail metrics are monitored independently of each test’s primary metric with automatic halt capability; experiment configuration is versioned and rollback-capable through the same pipeline as other LiveOps content; and concurrent experiments are checked for surface overlap before being allowed to run simultaneously. Infrastructure that passes all six lets a LiveOps team run experimentation as a continuous practice. Without it, most studios run one ambitious test, learn the hard way why one of these six points mattered, and quietly go back to shipping changes on instinct.
Keep Reading
More from the blog
Designing LiveOps Events That Do Not Burn Players Out: Cadence, Offers, and Segmentation
Want these insights on your build?
The engineers writing these articles are the ones who would work on your game.