A great model does not automatically make a great chat platform

Anthropic’s December 2024 architecture guide describes an augmented language model: a model connected to retrieval, tools and memory. It distinguishes fixed workflows from agents that direct their own processes and tool use. The page now warns that its tooling landscape has changed. We use the architectural distinction here, not its old tool list as a current shopping guide. [1]

Spicy's takeFor a reader, the consequence is simple: two services advertising the same underlying model need not deliver the same experience. One might supply useful documents or an action tool; another might provide only a conversation. Those are illustrative possibilities, not services we tested. Before comparing subscriptions, list the jobs you want the whole application to complete. [1][2]

Cloudflare’s AI Search documentation gives a concrete example of the surrounding machinery. It allows system prompts for both query rewriting and answer generation. During generation, the model receives the user’s query, retrieved document chunks and the generation prompt. That prompt can influence format, tone and which information is prioritized. [2]

Spicy's takeThis is why a model’s general knowledge and an application’s access to information are different questions. A service can only pass along the documents its retrieval process supplies. Ask whether sources are available to inspect, whether the answer addresses your question and whether the service acknowledges gaps. A confident paragraph is not a receipt for a successful search. [2][3]

Microsoft’s RAG guidance recommends explicit instructions for missing context, partially answered questions and contradictory sources. It also recommends evaluating the final response after prompt changes. These are design recommendations, not guarantees that every product implements them or that instructions always work. [3]

Spicy's takeTry a small, ordinary task before committing to a service: ask a question whose supporting material you can check, then ask something that material does not answer. Notice whether the service distinguishes the two. This is a suggested reader exercise, not a SpicyRanks test result. The goal is to see whether the application helps you recognize its boundary, rather than merely producing another answer. [3]

Spicy's takeAnthropic’s guide advises starting with simple designs and adding complexity only when it is useful, noting tradeoffs involving latency and cost. More machinery is therefore not automatically a better product. A document assistant and a casual conversation app can sensibly make different design choices because they serve different jobs. [1]

Spicy's takeOur take: write down three things the service must do and one failure you cannot accept. Compare its documented features and your own observations against that list. Keep separate questions for model capability, source access, action tools and how uncertainty is presented. The model deserves attention, but the product around it is what you will live with. [1][2][3]