We don’t have many sites on our network that run ads. On the whole, this fact contributes to the success of both our platform and our customers who use it. But in some cases, ads are an inescapable fact of internet life, and we have to deal with them properly.

One problem we have with ads is that they change a lot. Therefore they trigger false positives in our visual regression testing (VRT), and they do so in two ways:

  1. Their content changes very frequently, as different ads come and go within an ad slot, sometimes on every page load.
  2. The ad slots themselves come and go, because ad stakeholders tend to want to manage them closely and frequently.

This kind of false positive looks something like this:

The testing suite says, “Hey! Scott! Look! Something has regressed! But actually it’s just that a site admin has placed a new ad on the page and no one manually excluded it from VRT.

Point #1 is easy to resolve. We have developed a custom UI in WordPress admin for excluding a widget from QA testing:

The admin UI for manually excluding a widget from VRT.

Point #2 is harder because most customers, or even employees, are not in the habit of routinely thinking about excluding widgets from QA testing. So I’ve taken a different tact. I’ve started using some heuristics to determine if a widget seems like it contains an ad. If so, automatically exclude it from QA:

Improving our VRT’s is probably my favorite way of improving platform stability. Ads may have nearly destroyed a democratic society, but they won’t bother our VRT’s!