How to build an AI voice agent for your phone system, step by step
Building an AI voice agent for your phone system takes no coding, but it does take method. This guide walks you through the practical steps: what to put in the instructions, how to set up messages and FAQs, how to define actions, how to test it and which mistakes to avoid before you put it on a live number.
Before you build an AI voice agent: decide what it should achieve
The first step in building an AI voice agent happens on paper, not in a portal. A voice agent works well when it has a clear, measurable purpose: booking an appointment, logging a support request, answering common questions, routing the call to the right person. Try to make it do everything on day one and you will end up with an agent that does many things half well.
Look at the calls you get today. Which five or six requests come up most often? Which ones end with a piece of information, which need details recorded, which really need a human? That split is the foundation of the whole project: information becomes FAQs, details to record become actions, and complex requests become handoffs to a person.
If you are not yet sure what this kind of assistant is or how it reasons, start with what an AI voice agent is and how it works. If you are still weighing whether you need one at all or whether a keypad menu would do, IVR or AI voice agent will help you decide.
- The main goal: the outcome that makes a call a success.
- Your frequent questions, with the exact answer you would give.
- The details each request needs, such as name, phone number, address or type of job.
- The situations where the call must go straight to a person.
- Hours: will the agent answer all the time, only out of hours, or only when your team is busy?
Step 1: create the agent and choose its voice and language
First you create the agent, give it a recognisable name and choose how it speaks. The name is for you and your colleagues, since it appears when you pick the agent as the destination of a number or menu and in reports. A name that says what it does, such as "Clinic bookings" or "After-hours support", beats a generic one.
The voice matters more than you might think. Too fast and callers feel rushed; too slow and the call seems to drag on. Always listen to a sample before you decide, ideally a real sentence from your line of business with names and numbers in it, because that is where voices differ.
In the Robottino AI voice agent you choose the voice, the reading speed (from 1.0x to 1.5x) and the language: Italian, English, Spanish, French, German or Portuguese. If a caller speaks another language, the agent follows them. You can hear the voice read a sample sentence before saving.
Step 2: write the instructions (the operating prompt)
The instructions are the heart of the agent. You write them in plain language, as if you were explaining the job to a new colleague on their first day. There are no magic formulas: what you need is information that is clear, well organised and free of contradictions.
A good operating prompt answers four questions: who you are, what you know, what you need to achieve and which special rules you must follow. A few sentences per question is usually enough. The longer and messier the prompt, the more likely the agent is to give weight to the wrong part.
You do not need to repeat the basics of any good phone conversation. Well designed systems already include them: in Robottino, for example, the system adds rules such as not making things up, asking for details one at a time and confirming them, asking whether there is anything else, saying goodbye and ending the call. Spend your words on what is specific to your business.
| Part of the prompt | What to write | Example |
|---|---|---|
| Who you are | Name, role, tone | You are Sara, the assistant at Bianchi Dental. You are warm and polite. |
| What you know | Services, hours, address, terms | The practice is open Monday to Friday and offers hygiene, orthodontics and implants. |
| What to achieve | The purpose of the call | Your job is to book appointments and answer practical questions. |
| Special rules | Limits and edge cases | Never give medical advice. If someone is in severe pain, pass the call to reception at once. |
Step 3: standard messages and frequent questions
Besides the prompt, a voice agent needs a few fixed lines. The opening message is the first thing callers hear: it should say who is answering and invite them to speak. The error message covers the moments when something fails or the caller asks for something the agent cannot do. The closing message says goodbye before the call ends.
Frequent questions are the most underrated part. Each question with its exact answer lowers the risk of the agent improvising on sensitive topics such as prices, timescales, required documents or payment methods. The agent uses these answers as they are, so write them the way you would say them on the phone: short sentences, no long lists and no abbreviations that text-to-speech would read awkwardly.
- A short opening: who you are and how you can help, in one or two sentences. Skip long introductions and legal wording up front.
- A helpful error message: offer an alternative, such as a call back or leaving a message.
- A clear close: confirm what happens next, then say goodbye.
- FAQs written for the ear: numbers and times in plain words, one piece of information per sentence.
Step 4: add the agent’s actions (skills)
An agent that can only talk is useful; one that can act is far more so. Actions, often called skills, are the concrete tasks the agent can carry out during the conversation: checking availability and booking an appointment, sending the collected details to your business software, passing the call to a colleague.
Each skill has a name, a description of when to use it and the list of details to ask the caller for. The agent picks the right skill on its own by reading the name and description, so you do not need to explain it in the prompt. That is why the "when to use it" description deserves care: it has to set each skill clearly apart from the others.
For each detail you set a label, a type (text, number, whole number, yes/no, list), optional guidance, an example, allowed values, a required format and whether it is mandatory. The agent asks for the details one at a time and confirms them before running the action.
| Action type | What it does | When to use it |
|---|---|---|
| None | Collects the details and only sends email notifications | Requests your office will handle by hand |
| Call a webhook | Sends the details to your URL with the method, headers and JSON body you choose | Opening a ticket or lead in your CRM or business system |
| Use a connector | Google Calendar, Calendly or a Zapier Zap | Booking appointments or linking tools without code |
| Pass the call to the phone system | Transfers to a queue, operator, IVR menu, callback and other destinations | Emergencies, complex requests, callers asking for a person |
How to design a skill that works
Skill quality decides much of the outcome. A good skill has a single job, asks only for the details it really needs and has a description that does not overlap with any other. If two skills could both seem to fit the same request, the agent may pick the wrong one.
For actions that send data to your own system, test the webhook from the portal before going live and write what the agent should say if your system does not respond: "I have logged your request and we will call you back" is far better than an awkward silence. To learn more about receiving and verifying that data, see how to integrate your phone system with your CRM, which covers events, formats and security.
You can also mark a skill as a conversion, so every time it runs the conversation counts as a result in your reports, and switch on an email notification that sends the collected details to your team.
Pick the job
One action per skill, for example "book a visit" rather than "book, move or cancel".
Write when to use it
Describe the caller’s situation, not the mechanics: "when the caller wants to book a new visit".
List the details
Only the essential ones, with type, example and format. Every extra detail is one more question on the phone.
Choose the action
Webhook, connector, transfer to the phone system, or no action with a notification only.
Plan the fallback
Write what to say if the action fails and where the call should go.
Step 5: connect the AI voice agent to your phone system
A voice agent becomes useful once it receives the right calls. In a cloud phone system you connect it like any other destination: straight to a number, to a key in a phone menu, to the exit of a queue when the wait gets too long, or behind a time condition so it only answers out of hours.
There is no single right setup. Many businesses start small and widen the scope later. Four common patterns:
- Out of hours: the agent answers in the evening and at weekends, your team during office hours.
- Queue overflow: if nobody answers within the maximum wait of a call queue, the call goes to the agent instead of being lost.
- Menu option: in an IVR menu, one key leads to the AI agent for requests that fit none of the other choices.
- First filter: the agent takes the caller’s name and reason for calling, then passes the call to the right queue.
Step 6: test the agent and read the conversations
Never put an AI voice agent on a public number without testing it. Test the way a real caller would behave: half sentences, changes of mind, background noise, off-topic questions. Reuse the sample conversations you wrote at the start and add a few awkward cases.
After each test, read the full transcript. Look for where the agent hesitated, asked for the same detail twice, picked the wrong skill or gave an overlong answer. Nearly always the cause is in the instructions: an ambiguous rule, a missing FAQ, or a skill description too close to another one.
With the Robottino agent you can make a test call from your browser, which is not billed and stays out of the reports, then read the conversation in the log with its transcript, the skills that ran and the details collected.
- The caller changes their mind halfway: does the agent adapt without starting over?
- The caller asks for a person: does the call actually reach the right destination?
- The caller asks something you have not covered: does the agent admit it rather than make something up?
- Your system does not respond: does the caller hear the fallback message?
Mistakes to avoid when building an AI voice agent
Most disappointing voice agents do not have a technical problem but a design one. These are the most common mistakes and how to avoid them.
| Mistake | What happens | How to avoid it |
|---|---|---|
| Overlong prompt | Important rules get lost among details | Keep identity, goal and rules in the prompt; put facts in the FAQs |
| No way out | The caller gets stuck and hangs up | Always include a skill that passes the call to a person or a callback |
| Overlapping skills | The agent picks the wrong action | One skill per job, with clearly different descriptions |
| Asking for too much | The call drags on and the caller gives up | Ask only for essentials and collect the rest later |
| Answers written for reading | The voice reads lists and acronyms unnaturally | Write short sentences meant to be heard |
| No review | The same errors repeat for weeks | Read conversations every week and fix what you find |
After launch: measure and improve
You do not set up a voice agent once and forget it. What callers ask changes with the seasons, your offers and new services, and the agent has to keep up. The conversation log is your main tool: it shows how many conversations took place, how many led to a result and how long they last on average. Look closely at conversations with no conversion and at handoffs to a person: each one may point to a missing FAQ or skill.
Frequently asked questions
Do I need to code to build an AI voice agent?
No. Instructions, messages and FAQs are written in plain language, and common actions such as calendar booking or call transfer are set up in the portal. You only need technical skills if you want to send data to your own system through a webhook.
How long should an AI voice agent prompt be?
Long enough to say who the agent is, what it knows, what it should achieve and which special rules apply, usually a few sentences for each. Detailed facts such as prices or required documents work better as separate FAQs.
How do I get an AI voice agent to book appointments?
Connect a calendar such as Google Calendar or Calendly and create a skill that checks availability and books the appointment with the details asked of the caller. In the skill’s description, explain when it should be used.
Can an AI voice agent transfer a call to a human?
Yes, if you create a skill that passes the call to the phone system. You can send it to a queue, an operator, an IVR menu or a callback, and say in the instructions when it should happen, for example in emergencies.
How do I test an AI voice agent before going live?
Make test calls that mimic real callers, including difficult cases, then read the transcript of each conversation. If your platform offers it, use a test call that is not billed.
What should I do if the AI voice agent gives a wrong answer?
Read the transcript to find the cause: often an FAQ is missing, a rule is ambiguous or a skill description overlaps with another. Fix one point at a time and test again with the same question.