Introduction: What is A/B Testing?
A/B testing allows you to compare two variants of the same agent (Variant A and Variant B) against each other. Traffic is split 50/50 between the two variants, and you can measure which one performs better on key metrics like screen loads, impressions, questions asked, time spent, and conversions.
This is perfect for optimizing:
Visual design (colors, fonts, layout)
Placement (paragraph number, popup delay, etc.)
Engagement settings (CTAs, end screens)
Engine selection
Note: A/B tests are a layer on top of your existing agent. Your original agent remains untouched and continues to collect data normally. The test creates a temporary variant (Test B) that runs alongside it for a defined period.
Accessing A/B Tests
From the main navigation sidebar, click on
AB Tests(located underAI Labsor directly in the sidebar).You will see the A/B Test dashboard listing all existing tests with their Status (Active/Inactive), Created Date, and Actions.
Creating an A/B Test
Step 1: Start New Test
Click the + Create New A/B Test button.
Step 2: Configure Basic Settings
Field | Description |
|---|---|
Test Name | Give your test a descriptive name (e.g., "Homepage AB Test") |
Start Date | When should the test begin? |
End Date | When should the test end? (Tests should run for a defined period) |
Select Agent | Choose the base agent (Variant A) you want to test against. |
Important: Traffic will be split 50/50 between Variant A and Variant B. Variant B will not appear as a separate agent in your main Agents list.
Step 3: Create Variant B
After clicking Create, you will be taken to the Variant B editor.
What CANNOT be changed (locked):
Agent name
Knowledge Agent type (channel)
Targeting rules (where it runs)
Collection (knowledge base)
What CAN be changed:
Settings & Configuration (engine, suggest questions, CTAs, end screens)
Placement (insertion method, paragraph number, pattern, popup delay, etc.)
Visualization (theme, colors, fonts, logo, button styles, etc.)
Elements (disclaimer text, ratings, footer, watermark)
Why? The core identity and reach of the agent must stay identical for a fair test. Only the user experience and design variables are tested.
Step 4: Submit
Click Submit to launch the A/B test. From that moment forward, new visitors will be randomly assigned to either Variant A or Variant B, and data collection begins from zero for the test period.
How Traffic Allocation Works
50/50 split – Each visitor has an equal chance of seeing Variant A or Variant B.
Sticky assignment – Once a visitor is assigned to a variant, they will always see that same variant on that device/browser (no confusing switching).
Different devices or incognito windows may see the other variant.
Monitoring A/B Test Performance
A/B Test Overview Page
After creating a test, click on its name in the A/B Test dashboard to see the comparison view.
Key Metrics (Side by Side)
Metric | Variant A | Variant B |
|---|---|---|
Screen Loads | Value | Value |
Screen Impressions | Value | Value |
Questions Asked | Value | Value |
Avg Time Spent | Value | Value |
Conversions | Value | Value |
Timeline Chart
Visual comparison of Total Screen Impressions (or any metric) over the test period. You can switch metrics using the dropdown.
Leads
Leads captured by both variants are visible in the main Leads section (filterable by agent). To see which variant generated which lead, use the A/B Test overview page – it provides variant-specific breakdowns.
Important Notes (from the videos)
Note | Explanation |
|---|---|
Fresh start | When you create an A/B test, all metrics for both variants reset to zero from that exact moment. Historical data is not included in the test. |
Original agent unaffected | Your base agent (Variant A) continues to collect data as usual in the main Agents section. The A/B test is a separate view. |
Variant B not listed as an agent | You will not see "Test B" in your main Agents list. Only the original agent appears, with a label indicating it is part of an active A/B test. |
Leads are aggregated | In the main Leads section, leads from both variants appear together. Use the A/B test view to see which variant performed better on conversions. |
Test duration | Set a clear start and end date. After the test ends, you can analyze the winner and then update your main agent with the winning configuration. |
Best Practices
Test one variable at a time – Change only the color, or only the placement, not both at once. Otherwise you won't know what caused the difference.
Run tests for sufficient time – At least 1–2 weeks or until you reach statistical significance (enough traffic).
Use for high-impact decisions – A/B test major design changes, new CTAs, or different engines before rolling out to all users.
Document winners – Keep a record of which configurations performed best for future reference.
Troubleshooting
Issue | Likely Cause | Action |
|---|---|---|
Variant B not showing | Cache or stale session | Try incognito mode or different device |
Metrics seem low | Test just started | Wait for more traffic |
Can't edit certain fields | Those fields are locked for A/B testing | They are intentionally locked to ensure a fair test |
Leads not appearing | No conversions yet | Check if CTAs are configured correctly on both variants |
Summary
A/B testing empowers you to make data-driven decisions about your agent configurations. By comparing two variants side by side with a 50/50 traffic split, you can confidently choose the design, placement, or settings that drive the most engagement and conversions.