The model is the product's voice, but it is only one part of the stack. Companion products also need memory, safety layers, routing, evaluation and — increasingly — voice and images. This guide sets out the main decisions and the trade-offs behind each.
Hosted APIs versus open-weight models
Hosted APIs from the large model providers offer strong general capability, minimal infrastructure and fast iteration. You pay per token, and the provider handles scaling.
The trade-offs:
- Content policies. Every major provider publishes usage policies, and many restrict sexual content, certain role-play themes or emotionally manipulative behaviour. A companion product that permits mature content for verified adults may simply be outside a provider's terms.
- Change control. Providers update and retire models. A version update can subtly shift a persona's tone overnight.
- Data handling. Review retention, training-use and regional processing terms carefully; companion conversations are intensely personal.
- Unit economics. Long, chatty sessions consume a lot of tokens. Model costs can dominate gross margin.
Open-weight models, self-hosted or run through a specialist inference provider, give you more control over behaviour, fine-tuning, data residency and versioning. The costs are operational: GPU capacity, inference optimisation, monitoring, and the responsibility for safety layers that a hosted provider might otherwise supply. Note that open-weight licences also carry acceptable-use terms — read them.
Many operators run both: a hosted model for SFW conversation and complex reasoning, and an open-weight model for experiences a hosted provider does not permit.
Read the content policies first
Before writing a line of code, map your intended experience against each candidate provider's policy. Ask:
- Does the policy permit romantic or adult role-play for verified adults, or prohibit it outright?
- Are there requirements around disclosure, age assurance or crisis handling?
- What happens on a violation — warning, throttling, termination?
Building on a provider whose terms you are working around is a business risk, not a technical one.
Latency
Companion conversation feels natural only when responses arrive quickly. Track time-to-first-token as well as total generation time, and stream output to the interface. Smaller models, quantisation, regional deployment and prompt caching all help. For voice, latency budgets are tighter still — any delay is audible.
Memory and retrieval
Users expect companions to remember them. Context windows alone are rarely enough, and stuffing every past message into each prompt is slow and expensive. Common patterns:
- Rolling summaries of recent conversations.
- Structured profile facts — name, preferences, important dates — extracted and stored explicitly.
- Retrieval (RAG) over past conversations using embeddings, pulling relevant moments into context.
- User controls to view, edit and delete what is remembered.
Retrieval quality matters more than volume. Surfacing the wrong memory is worse than surfacing none. See also why memory is the moat.
Routing
Few products need their most capable model for every message. A router can send:
- Casual small talk to a fast, inexpensive model.
- Complex or emotionally sensitive turns to a stronger model.
- Content that matches safety triggers to a dedicated handling path, including crisis responses.
- Adult-mode traffic (for verified users only) to a model whose terms permit it.
Routing adds complexity — inconsistent voice between models is the common failure — so keep persona instructions shared and test transitions explicitly.
Evals
Without evaluations, every model change is a guess. Build a test set that reflects your product:
- Persona consistency — does the character stay in voice across long sessions?
- Memory recall — are facts retrieved correctly and appropriately?
- Safety — crisis detection, refusal of prohibited content, age-related red flags.
- Quality — engagement, coherence, repetition.
Combine automated scoring (including model-graded checks) with regular human review of sampled conversations, handled under strict privacy controls. Re-run the suite before switching models or prompts.
Safety layers
Do not rely on the generation model alone to enforce your policy. Independent input and output classifiers, keyword and pattern rules for hard lines, and logging of interventions give you defence in depth and evidence for processors and regulators. Our trust and safety basics cover the policy side.
Voice and image
Voice typically means speech-to-text, the language model and text-to-speech in a pipeline, or a speech-native model. Check voice-cloning terms: using a real person's voice requires their consent, and many providers restrict it.
Image generation introduces separate risks — particularly realistic depictions of real people and any content that could involve minors. Apply dedicated image classifiers to both prompts and outputs, and keep generation tied to fictional, consistent character designs.
Checklist
- Map your experience against provider content policies before building.
- Decide hosted, open-weight or hybrid based on policy fit, control and cost.
- Set latency budgets and measure time-to-first-token.
- Design memory with retrieval, structured facts and user controls.
- Route by task and sensitivity, with shared persona instructions.
- Build an eval suite and run it on every change.
- Add independent safety classifiers for text, voice and image.
- Pin model versions and plan for deprecations.
Provider terms change frequently; review them directly before relying on them.
