Evaluating the YOU API
We evaluate the effectiveness of the YOU API by adapting several academic question answering datasets to the “open web” setting: passages typically provided to an AI as context must instead be retrieved from the web itself. This evaluation method demands much more of the system as a whole than the academic benchmarks from which they are derived. This provides a more realistic and practical assessment of the performance of web scale RAG systems.
In this first set of results below, we compare using You.com, Google, or Bing search results together with the default LLM deployed in our Web LLM endpoint. The YOU API comes out ahead on every benchmark.
Higher F1 scores across the board show that the LLM is more likely to generate a correct response given search results from the YOU API than even Google search results. The LLM used in these experiments has a context window of 16k tokens and is most similar to OpenAI’s ChatGPT 3.5 Turbo model. Recall numbers have an even wider margin than the F1 scores, which suggests that the LLM cannot fully make use of the extra information that the You.com Web Search API provides to generate better answers.
Recall here means that the correct answer (based on the original ground truth phrase from the datasets) is included in the snippets that are returned by the API:
| FreshQA Recall | SquadV2 Recall | HotpotQA Recall | MS MARCO Recall | |
| You.com | 85.1% | 75.6% | 67.4% | 81.5% |
| 72.2% | 50.4% | 49.0% | 65.3% | |
| Bing | 73.8% | 54.4% | 56.0% | 70.5% |
Using an LLM with a 16k context window filled with snippets to answer questions:
| FreshQA | SquadV2 F1 | HotpotQA F1 | MS MARCO | |
| You.com | 44.7% | 27.5% | 34.7% | 23.8% |
| 41.4% | 27.0% | 33.4% | 20.1% | |
| Bing | 40.7% | 25.1% | 34.5% | 22.6% |
If you would like to evaluate your full scale web-index for LLMs on these datasets, please email us at api@you.com and we can share the evaluation benchmarks.
Future work
We will continue to work on enhancing the API’s functionality and user experience. Efforts are being made to make the APIs more intuitively conversational, allowing users to handle context effortlessly. Moreover, intermediate modules like query rewriting will be exposed as new endpoints, enabling developers to customize their usage of the YOU API according to their specific needs. Additionally, the API team aims to incorporate agent-like capabilities, similar to those of You.com’s Agent and Deep Research modes, making all the functionality of You.com accessible through the API.
Conclusion
The YOU API empowers developers to overcome the limitations of LLMs by providing them with the tools to ground their outputs in the most recent, accurate, and relevant information. With its superior performance compared to existing search providers and its ability to adapt to various LLMs, the API is revolutionizing the way developers can leverage the power of LLMs in their applications. We hope this frees up knowledge on the web from the very limited and restrictive options currently available. We are excited to see the many applications being built and will support our ecosystem with future innovation with up-to-date knowledge from the web.
Learn more about how to use the API at docs.you.com. Visit api.you.com to access the Web Search, News, and Web LLM endpoints. If you have any questions, please email api@you.com.
Yay, you’ve read the whole article. If you enjoyed this story, please recommend 👏 and share to help others find it!
