Based on your post it sounds like you were not conductiing A/B Split tests, that you were instead running serial tests. A/B Split tests will run the control and treatment over the same time period, randomly splitting the traffic between the control and the treatment. Serial testing does not “split” the traffic, but instead runs a control test for a period of time, then runs a second serial test of the treatment over an approximately equal amount of time.
Serial testing is never going to be as accurate as split tests because the two separate test periods will always have uncontrolled variables. Split tests will typically yield statistically valid test results in about 1/8th the time it takes to get close to the same level of validity with a serial test.
The bottom line is to always use split testing, where possible, for the most valid test results.
