StayPilot blog

What happens when the AI doesn't know the answer?

September 25, 2026 · hosting · trust

It tells the guest it needs to check with the host, and the question comes straight to you with a draft attached. That is the whole design: the one thing an AI answering your guests must never do is guess. Every reply StayPilot drafts carries a score for how well your listing's facts actually answered the question, and a reply that can't be grounded in those facts doesn't go out — it becomes your question. "Trust us, it knows its limits" deserves the same scrutiny as any other claim, so here is the machinery behind that sentence.

It can only answer from what you gave it

The reply engine is grounded: your property's details, house rules, knowledge base, and the guest's reservation are the entire universe it answers from. It is instructed never to invent door codes, prices, policies, or amenities — and when the facts don't answer the question, to say it will check with the host rather than improvise. "What it knows" is a list you can read and edit, not a model's general impression of vacation rentals. Ask about the pool heater when nothing about a pool heater is written anywhere, and there is nothing to answer from — what you put in is all it knows, and that cuts both ways on purpose.

Every reply gets one of three fates

Alongside each draft, the model reports its confidence that the reply is correct and fully grounded. The messaging policy turns that into a decision. A reply is sent automatically only when confidence clears 85 percent, the topic is not sensitive, and you've put that conversation on full autopilot. Below the bar, it is held as a draft for your review — the system would rather ask than guess. And sensitive topics — refunds, safety, injuries, anything legal, cancellations — never auto-send at any confidence; they always route to a human. New accounts start in draft mode anyway, so early on, every reply is one you pressed send on yourself.

The AI doesn't grade its own homework

Self-reported caution has an obvious flaw: the model that wants to be helpful is also the one deciding whether it should have stayed quiet. So a second, independent check runs on the guest's own message — a deterministic keyword screen for refunds, injuries, safety, legal threats, and their relatives. It can only raise the risk level, never lower it. If the model says "this is fine" and the screen says "this is sensitive," the screen wins, the message escalates, and the disagreement is recorded so it can be counted later. An over-helpful model cannot clear its own way to send.

The questions you keep for yourself

Not every escalation is about doubt. You can reserve categories — the personal welcome, the restaurant tips — and those route to you no matter how confident the draft is, with the draft ready if you want it. For plenty of hosts that slice of the messages is the hosting; automating away the part you love is how tools get abandoned. Keeping it is a setting, not a workaround.

What the guest actually experiences

A prompt "let me check with the host on that one — you'll hear back shortly" reads as a careful host. A confident wrong answer about parking reads as a bad stay, and sometimes becomes a bad review. Because the escalated question lands in front of you immediately, with the conversation and a recommended action attached, the guest usually waits minutes — not until you next open the app. Guests don't need the AI to know everything; they need the answer they get to be right.

Each "I don't know" is a to-do, not a shrug

Every held draft comes with one sentence for you: why the confidence was low and what was missing from the knowledge base. Add the missing fact once — the pool heater's season, the gate code hours, the one lamp with the tricky switch — and that question never stumps it again. Over a season, the gaps close in the order guests actually hit them, which is a better curriculum than any onboarding form.

The fastest way to evaluate this is to try to break it: paste your listing into the demo and ask a question your listing doesn't answer. Watching it decline to guess tells you more than any promise on this page.

See what StayPilot would say to your guests.

Paste your listing, get answers from its real details. Free 7-day preview, no card.