A Comparative Study of Static and Adaptive Traffic Allocation in A/B Testing Systems by IRJET Journal

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

A Comparative Study of Static and Adaptive Traffic Allocation in A/B Testing Systems

Abhishek

Kumar Shrivastava1 , Dr. Punit Kumar Johari2

1Student, Madhav Institute of Technology & Science, Gwalior, M.P, India

2Associate Professor, Madhav Institute of Technology & Science, Gwalior, M.P, India

Abstract - A/B testing is widely used for comparing feature variants based on user interaction data such as clicks and conversions. In most conventional systems, traffic is distributed equally among variants, which is referred to as static traffic allocation. While this approach is simple and easy to analyze, it does not utilize intermediate results during the experiment Because of this, it may miss opportunities to adapt earlier. In this work, we propose a methodology using a custom A/B Feature Testing application to simulate and analyze both static and adaptive traffic allocation behaviors. The simulated experiment evaluates 100,000 hypothetical users across three variants to observe system efficiency and bias. Interestingly, the static method identifies Variant 1 as the winner, whereas the adaptive method favors Variant 2. This difference suggests a possible trade-off. Adaptive allocation may improve efficiency by reducing regret, but it also appears to be sensitive to early statistical fluctuations. This paper explores that balance in more detail.

Key Words: A/B Testing, Static Allocation, Adaptive Allocation, Epsilon-Greedy, Traffic Allocation, Experimentation Systems

1.INTRODUCTION

A/Btestinghasbecomeastandardapproachinmodernproductdevelopment,allowingteamstoevaluatefeaturevariations basedonrealuserbehavior.Asystematicliteraturereviewindicatesthatmostimplementationsstillrelyontraditionalsingle A/B test setups [1]. In these setups, traffic is distributed equally between variants, and results are evaluated only after sufficientdatahasbeencollected.Whilethismethodisstraightforwardandreliable,itisalsorigid.Oncetheexperimentbegins, allocationdoesnotchange,evenifearlyobservationssuggestthatsomevariantsareperformingsignificantlyworsethan others.Thiscanleadtoinefficiencies,particularlyintermsofwastedtrafficanddelayeddecision-making.Theconceptofregret highlightsthisissue.Trafficassignedtounderperformingvariantsrepresentsalostopportunity,andthiscostaccumulates overtime[2].Additionally,real-worlddataoftendeviatesfromidealassumptions.Non-Gaussiandistributionsandintra-user correlationscanaffectthereliabilityofresults[3],[4],andstaticallocationdoesnotaccountforthesevariations.Toaddress theseconcerns,adaptiveallocationstrategieshavebeenproposed.Thesemethodsadjusttrafficdynamicallybasedonobserved performance.However,thisraisesanotherquestion:canearlyadaptationleadtoprematureconclusions?Thisworkdoesnot attempttodeclareonemethodsuperior.Instead,itinvestigateshowbothapproachesbehaveunderthesameconditions.A customsimulationenvironmentisusedtocomparestaticallocationwithanepsilon-greedyadaptivestrategy.Thegoalisto understandtheirpracticaldifferencesandidentifypotentialtrade-offs.

2. RELATED WORK

The challenges associated with A/B testing have been widely studied, particularly in the context of statistical reliability. Traditional methodsoftenstruggle whendata deviatesfromnormality,leadingtounreliableestimates[3].Similarly,ratio metricsintroduceadditionalcomplexityandcanbehaveunpredictablyundercertainconditions[4].Anotherimportantissueis theassumptionofindependentusers.Inmanyreal-worldsystems,usersinteractwitheachother,leadingtoviolationsofthe StableUnitTreatmentValueAssumption(SUTVA)[5],[6].Thiscanintroducebiasintoexperimentalresults.Techniquessuchas cluster-basedrandomizationandlink-awaretestinghavebeendevelopedtoaddresstheseconcerns.Specializeddomainsalso requiretailoredapproaches.Forexample,auction-basedsystemsinvolvestrategicinteractionsthatcomplicatetraditionalA/B testing methods [7]. In parallel, automated experimentation frameworks like SEAByTE aim to integrate A/B testing into continuousdeploymentpipelines[8].Recentworkhasalsoexploredsimulation-basedevaluationusingAI-drivenagents[9]. Despitethesedevelopments,theproblemoftrafficallocationduringanexperimentremainsanactiveareaofinterest.Thiswork focusesspecificallyonthataspect.

Research

2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

3. CONCLUSIONS

Theproposedapproachisbasedonasimulationframeworkthatmodelsusertrafficandallocationstrategies.Incomingusers areassignedtodifferentvariantsthrougharoutingmechanism,andtwoallocationstrategiesareevaluatedsimultaneously: staticallocationandadaptiveallocation.Staticallocationdistributestrafficuniformlyacrossallvariants,whereasadaptive allocationfollowsanepsilongreedystrategy,whereaportionoftrafficisusedforexplorationandtheremainderisdirected toward the currently best-performing variant. User behavior is modeled probabilistically, with clicks and conversions generatedusingBernoullidistributionstocapturevariabilityinuserinteractions.Thebackendsimulationlogicintegrates severaldesignchoicestoreflectreal-worldbehaviorwhilemaintainingcomputationalefficiency.Usereventsaregenerated probabilistically, specifically modeled as Click Bernoulli(p click) and Conversion Bernoulli(p conversion). To balance responsivenesswithstability,updatesarenotperformedaftereveryuser;instead,usersareprocessedinbatchesof100,which helpsreducenoiseandensuremorestableupdates.Ateachbatchinterval,thesystemrecalculatesperformancemetricsand adjuststheallocationstrategyaccordingly,whilecontinuouslytrackingkeymetricssuchasimpressions,conversions,and cumulativeregret.Thesimulationprocessesatotalof100,000usersacrossthreevariants(V1,V2,V3),andtheworkflow includesinitialization,userrouting,eventsimulation,metricupdates,andfinalevaluation.Theresultsofadaptiveallocationfor 10,000usersarepresentedinTable1,whilethecorrespondingstaticallocationresultsareshowninTable2.Forthelargerscaleexperimentwith100,000users,adaptiveallocationresultsareprovidedinTable3,andstaticallocationresultsareshown inTable4.TheoverallcomparativetrendinregretacrosstheseexperimentsisillustratedinFigure1.

4. MATHEMATICAL FORMULATIONS

CR=Conversions/Users

CTR=Clicks/Users

Regret=∑(Usersassignedtonon-optimalvariants)

P(explore)=ϵ, P(exploit)=1−ϵ

5. EXPERIMENTAL RESULTS

A. Experiment1(10000users) 1) AdaptiveAllocation

Table 1

2) StaticAllocation

Table 2

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

B. Experiment2(100000users)

1) AdaptiveAllocation

Table 3

2) StaticAllocation

6. GRAPHICAL ANALYSIS

7. EXTENDED ANALYSIS

Adaptiveallocationshowslowerregretinsmallerexperimentsbutbecomesunstableatscale.Staticallocation,whileinefficient early,providesmorereliableoutcomesovertime.

Table 4

Figure 1

2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

8. DISCUSSION

The results show a noticeable difference between the two approaches. Under static allocation, traffic remains evenly distributed,andVariant1achievesthehighestconversionrateat10.11%.Theothervariantsfollowcloselybehind.Incontrast, adaptive allocation gradually shifts traffic toward better-performing variants. Over time, Variant 2 emerges as the top performer with a slightly higher conversion rate than Variant 1. While adaptive allocation reduces regret by minimizing exposuretoweakervariants,italsoappearstobesensitivetoearlyperformancesignals.Ifthesesignalsareinfluencedby randomness,thesystemmayconvergetoasuboptimalchoice.

Thereareadditionalconsiderations.Adaptivemethodsassumearelativelystableenvironment,butreal-worldconditionscan change over time. User behavior may shift due to external factors, which can affect the validity of results. Furthermore, adaptiveallocationdoesnotensureequalexposureacrossvariants,whichmayraiseconcernsincertainapplications.Looking morecloselyattheexperimentalresults,especiallyacrossthetwodifferentscales(10,000vs100,000users),amorenuanced patternbeginstoemerge.Inthesmallerexperiment,adaptiveallocationshowsclearadvantages.Itreducesregretsignificantly andslightlyimprovesconversionrate.Thissuggeststhatinlow-trafficorshort-durationexperiments,reactingquicklytoearly signalsmayprovidepracticalbenefits.

However,thelargerexperimenttellsadifferentstory.With100,000users,staticallocationperformsbetterintermsofboth conversionrateandregret.Thisisimportant.Itindicatesthatadaptivesystemsmaystruggletorecoveronce4theycommittoo early.Thebiasintroducedintheinitialphasecontinuestoinfluenceallocationdecisionsthroughouttheexperiment.Another subtleobservationishowtrafficgetsdistributedinadaptivesystems.InExperiment2,alargeportionoftrafficisallocatedto Variant B, even though Variant A eventually turns out to be the best performer. This suggests that the system may have overreactedtoearlyperformancedifferences.Oncethathappens,explorationbecomeslimited,andcorrectingthedecision becomesharder.Fromatheoreticalstandpoint,thisbehavioralignswiththeknownlimitationsofepsilon-greedystrategies. While they maintain a balance between exploration and exploitation, the balance is not always sufficient when early observationsarenoisy.

Theexploitationcomponenttendstodominate,especiallyasmoredataisaccumulated.Ontheotherhand,staticallocation appearsinefficientintheshortterm.Itcontinuessendinguserstoweakervariants.Butovertime,thisuniformsampling providesamorereliableestimateoftrueperformance.Inthatsense,staticallocationactsasasafeguardagainstpremature convergence.Thisleadstoaclearerinterpretationofthetrade-off:Adaptiveallocationprioritizesefficiency,butrisksbias. Staticallocationprioritizesreliabilitybutsacrificesshort-termgains.Neitherapproachfullydominatestheother. Thechoicedependsheavilyonthecontextoftheexperiment trafficvolume,duration,andtoleranceforrisk. Ahybridapproachnaturallyfollowsfromthisobservation.Startingwithstaticallocationallowsthesystemtogatherstableand unbiaseddata.

Oncesufficientconfidenceisachieved,switchingtoadaptiveallocationcanimproveefficiencywithoutbeingoverlysensitiveto earlynoise.ThisbalancebetweenstabilityandresponsivenessappearstobethekeychallengeindesigningpracticalA/B testingsystems.

9. CONCLUSION

ThispaperpresentsacomparativeanalysisofstaticandadaptivetrafficallocationinA/Btestingsystems.Throughcontrolled simulationexperimentsatdifferentscales,thestudyhighlightsthattheperformanceofthesemethodsisnotabsolutebut contextdependent.Staticallocationdemonstratesstrongreliability.Bydistributingtrafficuniformly,itensuresunbiaseddata collectionandstableestimatesofvariantperformance.However,thiscomesatthecostofefficiency,asasignificantportionof traffic may be allocated to underperforming variants. Adaptive allocation, on the other hand, improves efficiency by dynamicallyshiftingtraffictowardbetter-performingvariants.

Thisleadstoreducedregret,particularlyinsmallerexperiments.However,theresultsalsoshowthatadaptivemethodsare sensitivetoearly-stagenoise.Onceabiasisintroduced,thesystemmaycontinuereinforcingit,makingrecoverydifficult.The experimentalresultsreinforcethistrade-off.Insmaller-scaleexperiments,adaptiveallocationperformsbetter.Inlarger-scale experiments,staticallocationprovidesmoreconsistentandreliableoutcomes.Thissuggeststhatadaptivemethodsmaynotbe scaledaseffectivelywhenearlyuncertaintyplaysasignificantrole.Akeyinsightfromthisworkisthatadaptiveallocationis notuniversallysuperior.Whileitofferspracticaladvantages,italsointroducesrisksthatmustbecarefullymanaged.

Similarly, static allocation, although conservative, provides a level of robustness that remains valuable. Based on these observations, a hybrid strategy appears to be a promising direction. Combining the stability of static allocation with the efficiencyofadaptivemethodsmayofferabalancedsolution.Forexample,anexperimentcouldbeginwithuniformtraffic distributionandtransitiontoadaptiveallocationoncesufficientdatahasbeencollected.Overall,thefindingssuggestthatthe

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

designofA/Btestingsystemsshouldnotrelyonasingleallocationstrategy.Instead,itshouldconsiderthetrade-offsbetween efficiency,reliability,andadaptability,andchooseanapproachthatalignswiththespecificgoalsoftheexperiment.

10. FUTURE WORK

FutureworkcouldexploremoreadvancedadaptivestrategiessuchasThompsonSamplingorUpperConfidenceBound(UCB) methods[2].Theseapproachesexplicitlyaccountforuncertaintyandmayprovideabetterbalancebetweenexplorationand exploitation.Anotherdirectionistovalidatethesefindingsusingreal-worlddata.Whilesimulationsprovidevaluableinsights, practicaldeploymentmayrevealadditionalchallenges.Hybridapproachesalsopresentaninterestingpossibility.Startingwith staticallocationandgraduallytransitioningtoadaptivestrategiescouldhelpmitigateearly-stagebiaswhilestillimproving efficiency.

REFERENCES

[1]Quin,Federico,Weyns,Danny,Galster,Matthias,Silva,CamilaCosta,”A/BTesting:ASystematicLiteratureReview,”2023. https://arxiv.org/pdf/2308.04929v1

[2]Kaufmann,Emilie,Capp´e,Olivier,Garivier,Aur´elien,”OntheComplexityofA/BTesting,”ConferenceonLearningTheory, Jun 2014, Barcelona, Spain. JMLR: Workshop and Conference Proceedings, 35, pp.461-481, 2014. https://arxiv.org/pdf/1405.3224v2

[3]Gong,Junpeng,Wang,Chunkai,Li,Hao,Ma,Jinyong,Li,Haoxuan,He,Xu,”BeyondNormality:ReliableA/BTestingwithNon GaussianData,”2025.https://arxiv.org/pdf/2510.23666v1

[4]Nie,Keyu,Kong,Yinfei,Yuan,TedTao,Burke,PaulineBerry,”DealingWithRatioMetricsinA/BTestingatthePresenceof Intra-UserCorrelationandSegments,”2019.https://arxiv.org/pdf/1911.03553v2

[5]Zhou,Yifan,Liu,Yang,Li,Ping,Hu,Feifang,”Cluster-AdaptiveNetworkA/BTesting:FromRandomizationtoEstimation,” 2020.https://arxiv.org/pdf/2008.08648v1

[6] Zhang, Qiong, ”On the Asymptotics of Graph Cut Objectives for Experimental Designs of Network A/B Testing,” 2023. https://arxiv.org/pdf/2309.08797v1

[7]Chawla,Shuchi,Hartline,JasonD.,Nekipelov,Denis,”A/BTestingofAuctions,”2016.https://arxiv.org/pdf/1606.00908v1

[8]Quin,Federico,Weyns,Danny,”SEAByTE:ASelf-adaptiveMicro-serviceSystemArtifactforAutomatingA/BTesting,”2022. doi:10.1145/3524844.3528081

[9]Zhang,Wenlin,Li,Xiangyang,Ge,Qiyuan,Dong,Kuicai,Jia,Pengyue,Li,Xiaopeng,Zhang,Zijian,Wang,Maolin,Wang,Yichao, Guo, Huifeng, Tang, Ruiming, Zhao, Xiangyu, ”Exploring Recommender System Evaluation: A Multi-Modal User Agent FrameworkforA/BTesting,”2026.https://arxiv.org/pdf/2601.04554v1