Listen to this article. Also available on Spotify. Subscribe to PolyAPI Squawk.
Creating effective AI agents requires fine-tuning various parameters, such as match score cutoffs, historical lookbacks, and result sizes. To set these parameters effectively, users need both detailed inspection of individual responses and a broader statistical understanding of AI performance.
For example, when using our PolyAPI Assistant in VS Code or GitHub Copilot to access Poly documentation with the /p command, the system scores how well a documentation page matches a user’s prompt. These scores help us determine the most relevant pages to return.
Refining this process is crucial:
- If the right page isn’t selected due to a low score, we must either improve the content or lower the threshold.
- If the wrong page is selected, we need to either enhance the page or raise the cutoff.
- If too few results are retrieved, increasing the number of pages might help—but fetching too many can slow performance and raise costs.
This tuning is a continuous process to ensure accuracy and efficiency.
What’s Available Today?
Currently, our statistics endpoints are not exposed to customers, as they are highly technical and primarily used for internal diagnostics. However, we continuously analyze this data to refine our AI agent’s behavior.
What’s Coming Next?
We plan to extend our systematic tuning approach by introducing user feedback mechanisms:
- Thumbs Up/Down for AI Responses – Users will be able to rate responses, allowing us to automatically capture statistical insights and make targeted improvements.
- Conversation History for Admins – Admins with Manage Tenant permissions will be able to review team interactions with the AI via the PolyAPI Management UI. This feature is already accessible today through our /chat/conversations/… API.
- Custom AI Agents – Customers using our Assistants API can build and operate their own AI agents with similar analytics capabilities.
Want to try AI Assistants or Build Your Own?
You can sign up for free and create your own server functions. These functions can then be linked to an AI Assistant for seamless interaction.
Right now, this process is API-only, and while it’s powerful, you might need a bit of guidance. Don’t hesitate to reach out.
Have questions about our roadmap or platform? We’d love to hear from you! Reach out to us at hello@polyapi.io,fill out a contact form or book an intro meeting.
Listen to this article. Also available on Spotify. Subscribe to PolyAPI Squawk.