Exclude Existing Users for more confidence

Existing customers can add noise. If they are not potential buyers, you should exclude their visits from your sample size. Implement a cookie that remains even if the customer is logged off. You can then set your test segmentation criteria to target visitors who do not have this cookie. Since this reduces visitor count while sales stay the same, this reduces the margin of error in your results (increases confidence).

Cookies are not fool-proof. If existing customers clear cookies or share their login or use multiple devices, their visits will still count when they come to your page. You can exclude these visitors post-hoc by firing a custom goal when the customer logs in. This way, you can manually subtract the number of logins from total visits. Make sure your primary metric cannot be fired by existing customers visiting the page. For example, existing customers must not be able to visit the goal page URL you're using to track new customers.

Is it worth it? The larger your sample size, the more noise existing customers contribute. However, this will only impact your results if you have a huge sample size (tens of thousands), a high conversion rate (e.g., 10%), or a high return rate (e.g., 30%). There are two reasons you should implement this: (1) A high login rate represents an active user base, so it's a useful metric to keep and (2) It's best to confirm the impact for existing customers instead of guessing. For example, say you get 100 sales and 1000 total visitors on your Control (10% rate) and were after a 20% lift. If you tracked 300 logins, the base purchase rate becomes 100 out of 700 (14% rate). As a result, you would need 1600 fewer visitors for this test.

If you have enough traffic, you may want to exclude visitors who were exposed to previous tests. You can tell VWO to target new visitors only. You can additionally set up a separate, parallel test to target the returning visitor segment to see if they respond differently to your new page.

If you are testing internal pages, you'll want to do the reverse - target existing customers and exclude everyone else.

If you have a cookie to track this already, you may need to modify its values. For example, VWO will not work with cookies that contain multiple values. Keep your cookies focused on one piece of information.