Platform
Resources
In the world of web development, WYSIWYG visual editors have long been a popular tool for creating and editing web pages.
WYSIWYG editors (an acronym for “what you see is what you get”) promise a user-friendly interface where users can design web pages visually without needing to write or understand code.
These tools are often viewed as an easy button for building experiments on the web, but they come with their own set of challenges and blockers that can bring your experimentation program to a screeching halt. In this blog post, we’ll explore some of the realities of using these types of tools.
Context: Cooper Reid is an engineer who used to work closely with non-technical teams attempting to use visual editors to run experiments without engineers.
“The good:” Advantages of using a WYSIWYG editor
One of the biggest upsides of WYSIWYG visual editors is their user-friendly nature.
Anyone can start building or modifying visual aspects of their web pages using a GUI without any code. This aspect promises that non-technical teams can be entirely self-serve when it comes to designing, building, and running experiments.
Another selling point is the velocity at which you can build and run experiments. Although this may not actually be true in the long run, it’s certainly a reason that users choose WYSIWYG editors.
No-code visual tools reduce the bottleneck of depending on engineering teams to design, code, and deploy tests. Later on, we’ll discuss some of the counterpoints to this theme.
For the sake of this breakdown, we’ll consider marketing use cases as things that land in the realm of top-of-funnel, CRO, and landing page testing.
Visual editors typically cater to accommodating these use cases quite well. Examples of these experiments include button changes, color changes, image changes, CTA changes, and heading changes. This allows marketers to quickly test out new branding and copy with relative ease.
Many of these tools offer the capability to easily preview variations across different devices and screen resolutions, allowing users to simulate the experiment on their live website outside of the WYSIWYG tool itself. This can help give experimenters reassurance that things will work as expected when launched.
The ease of integration and global availability of the tool means that it can be installed site-wide by simply adding a single <script> tag to the <head> of the website. This means the tool can function across all pages regardless of the backend technology powering the experience.
If there are separate backend services (like CMS or IMS tools) responsible for powering the marketing pages, account registration experience, and in-product experiences, having the tool installed globally means that engineers & experimenters won’t have to manage multiple disparate implementations.
All of these benefits amount to a very compelling sales talk track and product demo. At the surface (especially in the confines of a guard-railed demo environment configured by a technical Sales Engineer) everything looks awesome.
Unfortunately, the shortcomings and reality quickly become apparent once it comes time for real-world usage.
“The bad and the ugly”
Limited use-cases
Visual editors are limited to visual experiments. Test plans involving business logic, full-page layouts, pricing, and subscription models all require tools that integrate into the server-side applications responsible for those features.
Furthermore, building tests using a visual editor sans-engineering is often limited to simple static web pages.
More likely a symptom of poor ideation and governance than the tool itself, many of the visual editor use cases are more gratuitous and for the purpose of changing text & images sans engineering, rather than validating a hypothesis.
The reason for buying any SaaS tool is to see ROI—but how will you quantify the ROI of your expensive tool by running surface-level tests that only measure basic clickstream metrics?
There are only so many visual-editor-friendly marketing pages on a site you can experiment within—and you’d better hope your test plans don’t include any complex treatments otherwise you’ll likely be pulling in a dev to do some front-end surgery.
In the next category, we’ll elaborate on some of the technical challenges associated with pushing visual editors past their limits.
Race conditions and incompatibilities abound
WYSIWYG tools operate via a <script> tag that’s installed across your site and is responsible for applying any visual changes. The very nature of running JavaScript to manipulate page elements rendered via your application can unleash tricky-to-reproduce and tricky-to-resolve race conditions that lead to a poor user experience or site bugginess.
In technical terms, the WYSIWYG site script attempts to apply the changes at various stages during the page lifecycle based on the test requirements. This could be during the initial page load, while a user is navigating, upon any URL changes, or while specific elements are appearing in the DOM.
Given that the visual changes and the site code are not reliably coupled together, this can lead to various unwanted outcomes:
“Flicker” (aka ~CLS, Cumulative Layout Shift), where the visual change is applied after the original treatment is shown.
Side note: Search engines literally grade your site’s CLS and will take action against sites with low scores.
The visual change is never applied at all
The visual change is applied to the wrong element
💡 These are typically solvable—but all of these scenarios will require time-consuming developer intervention.
Relying on a visual editor to apply changes to websites built using common SPA frameworks and their accompanying ecosystem of tools can be extremely problematic and even entirely incompatible. This results in engineering whack-a-mole, where devs are pulled in each time incompatibilities are encountered. Here’s a non-exhaustive list of some of the causes:
Unpredictable HTML semantics (randomly generated classes and IDs for elements) will essentially detach your configured changes each time there’s a new site build
SSR with hydration (Vue/React attach components to a page rendered on the server side).
Activations, deactivations, and reactivations of visual tests based on dynamic URL changes and DOM element changes.
Complete subversion of developer best practices and SDLC
A perceived benefit of visual editor-style testing tools is that you can test without developers.
Depending on who you ask, someone else might characterize that as subverting all developer best practices and skirting your entire software development lifecycle, thus resulting in governance blindspots and unforeseen collisions with site code that developers have to identify and fix.
Using visual editor tools with your version control system, QA best practices, unit testing, lower-environment testing, CI/CD pipeline—nope.
Giving marketing & non-technical folks unfettered access to making changes to the website in production—yup.
Performance engineers will hate you
Proper implementation of a WYSIWYG tool includes installing the client script as a blocking tag in the <head> of the HTML page.
In simple terms, this means that the rest of the webpage must await the browser’s downloading, parsing, and execution of the script. This is required in order to mitigate flicker and ensure the changes can be applied prior to elements being painted to the DOM.
It requires a lot of code to transmit all changes to the client side. It may end up being the largest static asset loaded by your webpage, and it’s quite common for the client script to exceed 100KB gzipped over the network!
When your site engineers pinpoint the script as the culprit for degraded site performance, it will be the first thing to get removed. In this case, ease of integration = ease of de-integration. 😢
Create a free account
The visual editors are first-class citizens, other critical capabilities are sacrificed
WYSIWYG tools are extremely sophisticated tools to build and maintain, especially as web technology and security practices evolve.
These tools are the long-standing bread-and-butter products for various providers offering no-code solutions to the market. As a result, many of the modern and critical components required for effective experimentation are missing:
Integrations—for enriching other systems with experiment assignment information and ingesting event data from other platforms (CDPs and analytics tools). Not all integrations are created equally: Be weary of client-side integrations that require custom coding. All Statsig integrations are server-to-server and have product owners.
Support for the data warehouse—to measure experiments using meaningful metrics that are centrally defined, governed, and consumed across the business.
Stats engine and results reliability and data accessibility—You should be able to trust the results of your experiments. There should be tools to detect SRM, deep-dive into your metrics, export your experiment data and the results should be reproducible offline.
Holdouts—allowing you to measure the cumulative impact of experimentation and incorporate long-term metrics like lifetime value and customer retention. Read more about why Holdouts are so important.
Developer experience (diagnostics to monitor experiment health, live stream for debugging assignments, Users tab for auditing all assignments and events for a specific user) and Governance tools (change approvals, access controls).
Join the Slack community
Build fast?
Recent Posts
Building an experimentation platform: Assignment
Take an inside look at how we built Statsig, and why we handle assignment the way we do. Read More ⇾
Decoding metrics and experimentation with Ron Kohavi
Learn the takeaways from Ron Kohavi's presentation at Significance Summit wherein he discussed the challenges of experimentation and how to overcome them. Read More ⇾
It’s normal not to be normal(ly distributed): what to do when data is not normally distributed
Learn how the iconic t-test adapts to real-world A/B testing challenges and discover when alternatives might deliver better results for your experiments. Read More ⇾
How the engineers building Statsig solve hundreds of customer problems a week
See how we’re making support faster, smarter, and more personal for every user by automating what we can, and leveraging real, human help from our engineers. Read More ⇾
Enhanced marketing experiments with Statsig Warehouse Native
Marketing platforms offer basic A/B testing, but their analysis tools fall short. Here's how Statsig helps you bridge the gap and unlock deeper insights. Read More ⇾
Feature rollouts: How Instagram left me behind
When Instagram Stories rolled out, many of us were left behind, giving us a glimpse into the secrets behind Meta’s rollout strategy and tech’s feature experiments. Read More ⇾