How to Fine-Tune Your Engine
Introduction: What is Fine-Tuning?
Fine-tuning is the process of improving your AI Engine after it has been launched by using real user feedback. It's the "continuous improvement" loop of your AI strategy.
When your agent is live, users can provide feedback on responses—thumbs up/down, star ratings, or text comments. Fine-tuning allows you to collect this feedback, review it, and use it to create improved versions of your Engine.
Think of it as:
Your Engine = The first draft of your AI's brain
Fine-tuning = Learning from mistakes and successes to create a smarter, better version
The Fine-Tuning Workflow
text
Launch Engine → Collect Feedback → Review Feedback → Create Fine-Tuning Job → New Engine Version CreatedPrerequisites
Before you can fine-tune an engine, you need:
A published Engine that has been live with agents collecting feedback (Learn how to create AI engines)
User feedback collected on that engine's responses (star ratings, text comments, etc.)
Access to AI Labs → Fine Tuning section
Step-by-Step: Fine-Tuning Your Engine
Step 1: Access the Fine Tuning Section
From the main navigation sidebar, click
AI Labs.Click the sub-menu item
Fine Tuning.You will see the Fine Tuning dashboard with a list of existing jobs and a
+ Start Tuningbutton at the top left.
Step 2: Start a New Fine-Tuning Job
Click
+ Start Tuning.The Create Fine Tuning Job panel will open.
Step 3: Configure Job Basics
3.1. Title
Enter a descriptive name for this tuning job (e.g., "Q2 Poll Engine Improvements" or "Formal Tone Fixes").
3.2. Select Engine
Click the dropdown and choose the Engine you want to improve. This will be the engine that has been collecting feedback.
3.3. Select Engine Version
Choose which version of the engine you want to base this tuning on.
Important Note on Versions
Feedback is collected at the engine version level, not the engine level. This means:
If you have "Version 1" and "Version 2" of an engine, each has its own feedback history.
When selecting a version here, you are only tuning based on feedback received for that specific version.
This allows you to compare performance across versions and improve incrementally.
3.4. Date Filter
Select the time period for feedback you want to review:
Past Week
Past Month (default)
Past 3 Months
Custom Range
Step 4: Review and Select Feedback
This is the most critical step. You will see a table of Feedback Selection showing all feedback collected during the selected period.
Feedback Table Columns:
Column | Description |
|---|---|
Engine Title | Which engine generated the response |
Engine Version | Which version was used |
Feedback Type | How the user responded (Stars, Text, Manual Edits, Regenerate) |
Feedback Text | Any comments the user left |
Understanding Feedback Types:
Type | Description |
|---|---|
Stars | User rating (1-5 stars). Higher is better. |
Text | User wrote a comment (e.g., "Too informal, should be more professional"). |
Manual Edits | A human editor manually corrected the AI's response. This is high-value feedback. This is normally used in Polls and agents which can have human in the loop |
Regenerate | User clicked "regenerate" to get a different response (implies dissatisfaction). |
Selecting Feedback for Tuning:
Browse through the feedback items.
Check the checkbox next to each piece of feedback you want to include in this tuning job.
Tip: Focus on:
Low-star ratings (1-2 stars)
Text comments with specific guidance
Manual edits showing correct responses
Patterns, multiple users complaining about the same issue
Step 5: Create the Job
Once you have selected all relevant feedback, click
Create Jobat the bottom.You will be returned to the Fine Tuning dashboard, where your new job will appear in the list with a Processing status.
Step 6: Job Completion and New Version Creation
Once a job status shows as Completed or Published, the system has:
Analyzed all the feedback you selected
Updated the engine's guidelines based on that feedback
Automatically created a new version of your engine
What happens next:
The new version appears in the engine selection dropdowns with the next version number (e.g., "Version 2").
The original version remains available and unchanged.
You can now use this new version in any of your agents.
Note: There is no separate dashboard to review the tuned job details. The completion of the job is indicated by the status change, and the result is a new, improved engine version ready for use.
Viewing Feedback Logs
At any time, you can view raw feedback for any engine:
Go to Fine Tuning dashboard.
Click on an engine or job to see its Feedback Logs.
Use Filters to narrow by:
Feedback Type (Stars, Text, Manual Edits, Regenerate)
Date range
Rating value
The logs show:
Time of feedback
Engine + Version used
Feedback Type
Reason / Comment (if any)
Best Practices for Fine-Tuning
1. Focus on Quality Over Quantity
10 detailed text comments are more valuable than 100 star ratings.
Prioritize feedback that includes specific guidance ("too formal," "add bullet points," "wrong tone").
2. Look for Patterns
If multiple users rate responses poorly on formality, focus your tuning on tone guidelines.
If users keep asking for more concise answers, adjust your Engine's structure guidelines.
3. Include Manual Edits
When editors manually correct AI responses, this is gold. It shows exactly what the right answer should be.
4. Tune Regularly
Schedule monthly or quarterly tuning sessions to continuously improve.
Each tuned version should be smarter than the last.
5. Version Control
Each tuning job creates a new version, allowing you to roll back if needed and compare performance across versions.
Troubleshooting
Issue | Likely Cause | Solution |
|---|---|---|
No feedback appears | Engine hasn't collected any feedback yet | Ensure feedback collection is enabled on your agents. Wait for user interactions. |
Only stars, no text | Users are rating but not commenting | Consider prompting for comments after low ratings. |
Job status stuck on "Processing" | Large volume of feedback | Wait a few minutes. If still stuck, contact support. |
Can't select an engine | Engine may not have any feedback yet | Check date filter. Expand to longer time period. |
New version seems worse | Poor feedback selection | Review which feedback you included and try again with more focused selections. |
Fine-tuning transforms your AI from good to great. By systematically learning from real user interactions, you create Engines that truly understand your audience and deliver exceptional experiences.