How an AI voice agent works, step by step

An AI voice agent answers the phone, understands what the caller says and acts: providing information, booking an appointment, passing on a message. For the caller, the exchange feels like a conversation. For the business, it is a chain of technical building blocks that is worth understanding before getting started.

The three basic building blocks

Each exchange follows a cycle that repeats several times per call, in a fraction of a second:

1. Speech recognition (often called speech-to-text). The caller's voice is turned into text. Quality depends on the language, the accent, background noise and the phone line. In French-speaking Switzerland, place names, proper names and local particularities also need to be handled.
2. The language model. It reads the transcribed text, understands the intent ("I'd like to move my Thursday appointment") and decides on the response or action. It follows precise instructions: tone, permitted information, topics to avoid, situations in which it must hand over.
3. Speech synthesis (text-to-speech). The written response is converted into a natural voice and played to the caller.

Tuning how these steps chain together matters. A good configuration reduces silences, lets the caller interrupt the agent and prevents the agent from talking over them.

The tools: calendar, knowledge base, messages

A voice agent is only truly useful if it can act. It is therefore connected to tools, always with limited rights:

• The calendar: checking free slots, booking, moving or cancelling an appointment according to your rules (duration, type of consultation, minimum notice).
• A knowledge base: opening hours, address, access, services, documents to bring. The agent answers from this validated information and must not make things up.
• Messaging: sending a confirmation by text message or e-mail, creating a message for the team with a summary of the call.
• Your business software, if necessary, through suitable data integration.

Each tool is a door to action. We define precisely what the agent can read and modify, and nothing more.

Transparency, recording and transfer to a person

A voice agent speaks to your clients on your behalf. We consider three rules essential:

• Introduce itself as a virtual assistant. From the start of the call, the agent states that it is a virtual assistant. The caller knows who they are talking to and can ask for a person.
• Announce the recording. If the call is recorded or transcribed, the caller is informed at the start. The data collected is limited to what is necessary, in line with the principles of the nFADP (new Swiss Federal Act on Data Protection), and kept according to defined rules.
• Provide for transfer to a person. For an emergency, a complaint, a complex question or simply at the caller's request, the agent transfers the call or takes a clear message. In a medical practice, for example, the agent never gives medical advice and directs emergencies according to the practice's instructions.

These rules protect callers and your company's reputation. They also make the tool more effective: a well-informed caller cooperates better.

Use cases

• Phone reception outside opening hours: the agent answers during the lunch break, in the evening or at busy times, and sends a summary to the team.
• Booking and changing appointments: see our article on automating appointment booking.
• Qualifying requests: in insurance brokerage or real estate, the agent gathers the basic information before passing the file to the right adviser.
• Frequently asked questions: opening hours, address, documents to prepare.

How we work

We start by listening to how your calls go today: common reasons for calling, recurring questions, sensitive cases. We then write the agent's script with you, along with its limits and transfer rules. An internal testing phase allows us to adjust the voice, vocabulary and answers before any go-live. After launch, we review call summaries with you to correct whatever needs correcting. Data is hosted in Switzerland whenever possible. To find out more, see our AI voice agents page and our method.