"AI engineer" shows up in more and more job listings, but the title can mean very different things depending on who's hiring. To get a clear picture, I'm working through the AI Engineer roadmap on roadmap.sh one section at a time and writing down what I learn.
This first part covers the basics:
- what an AI engineer is
- what the work actually looks like
- how the role changes the way products are built
- how it differs from an ML (machine learning) engineer
What Is an AI Engineer?
An AI engineer is a software engineer who builds products on top of foundation models (large AI models that someone else has already trained on huge amounts of data, such as GPT, Claude, or Gemini).
The key words are "on top of." An AI engineer usually doesn't train a model from scratch, because that takes massive datasets, specialised hardware, and a research team. Instead, they take a model that already exists, usually through an API (a way for your code to send a request to the model and get a response back), and build a useful product around it.
This was the idea that made the role click for me:
The model is not the product. The system around the model is.
Anyone can call the same model API. What makes one AI product better than another is everything around that call: what information the model receives, what it's allowed to do, how its answers are checked, and what happens when something goes wrong. Building that system is the AI engineer's job.
Roles and Responsibilities
A list of responsibilities sounds abstract until you attach it to something concrete. So this section uses one example throughout: a support chatbot for an online shop called ShopX.
Choosing and Evaluating Models
(deciding which AI model to use)
Models differ in answer quality, speed, and price. A bigger model isn't always the better choice.
Example: Ask three different models the same 20 real customer questions. Then compare which one gives the best answers at a speed and price that make sense for ShopX.
Prompt and Context Design
(writing the instructions and information the model sees)
A model only knows what you give it in the request. Clear instructions and the right background information make a big difference to the answers.
Example:
You are a support agent for ShopX.
Answer only using the policy below.
Keep answers under 3 sentences.
If you don't know, say so and offer to connect the customer to a human.Grounding with RAG and Tools
(connecting the model's answers to real, up-to-date data)
Without grounding, a model may hallucinate (confidently give a wrong or made-up answer). There are two common ways to ground it:
- RAG (Retrieval-Augmented Generation: finding relevant documents and giving them to the model before it answers)
- Tools (functions the model is allowed to call, such as an API in your own system)
Example: A customer asks, "Can I return shoes after 30 days?" The system first pulls ShopX's refund policy (RAG), so the model answers from the real policy instead of guessing. For "Where's my order?", the model calls the order-status API (a tool) to get the actual tracking info.
Agents
(an AI that plans and takes several steps by itself)
A plain chatbot answers one question at a time. An agent can decide what to do next, step by step, until a task is done.
Example: A customer complains their order is late. The agent:
- checks the order status
- sees it's delayed by a week
- issues a discount voucher
- emails the customer an apology with the voucher
Evals
(automated tests that check the quality of AI answers)
Small changes to a prompt or model can quietly make answers worse. Evals catch that before your users do.
Example: Keep a set of 50 sample customer questions with expected answers. Every time you change the prompt or switch models, rerun them and check whether the scores went up or down.
Managing Cost and Latency
(cost: how much each request costs; latency: how long the user waits for an answer)
Model APIs charge per request, based on how much text goes in and comes out. Larger models are usually slower and more expensive.
Example: Send simple questions like "What are your opening hours?" to a small, cheap model. Use the bigger model only for complex cases. Common answers can also be cached (saved and reused) instead of generated again every time.
Guardrails
(rules that block unsafe, wrong, or off-topic answers)
A model will try to answer almost anything. Guardrails keep it within safe limits.
Example: The chatbot must never show one customer's order details to another customer. It should also politely refuse questions that have nothing to do with ShopX, like legal or medical advice.
Deploying and Monitoring
(shipping it to real users, then watching how it behaves)
An AI feature is deployed like any other service, but it needs extra attention in production because its behaviour isn't fully predictable.
Example: Log every question and answer, and track how many users click "This didn't help." If that number suddenly spikes after a change, something is wrong and needs investigating.
The Whole Picture
Put together, a single customer question flows through the system like this:
flowchart LR
U[Customer question] --> A[ShopX app]
A --> R["Retrieval & tools<br/>(refund policy, order API)"]
R --> M[Foundation model]
M --> G[Guardrails]
G --> Res[Response to customer]The model is just one box. Everything else is engineering.
Impact on Product Development
Before foundation models, adding an AI feature usually meant collecting labelled data (examples tagged with the correct answer), training a model, and tuning it, often over months. Today you can call a model API and have a working prototype in days.
That speed is the good news. It also brings problems that traditional software doesn't have:
- Non-deterministic outputs (the same question can get a slightly different answer each time). A normal test that expects one exact output doesn't work here.
- Evals alongside unit tests. Unit tests still check that your code works. Evals check that the AI's answers are good enough. You need both.
- Model drift and deprecation. Vendors update or retire models, so the same prompt can start behaving differently. Pinning a specific model version and rerunning evals before upgrading helps.
- Cost per request. Every call costs money. A bug that sends requests in a loop can quickly become a billing problem.
This all comes down to one lesson: a demo is not a product. Getting a chatbot to answer five questions well in a demo takes an afternoon. Getting it to handle thousands of real customer questions reliably, safely, and affordably is the actual engineering work.
AI Engineer vs ML Engineer
The two titles are often mixed up, so here's a side-by-side comparison:
| AI Engineer | ML Engineer | |
|---|---|---|
| Starting point | An existing foundation model | Raw data |
| Main focus | Building a product around the model | Building and training the model itself |
| Core skills | Software engineering, APIs, prompt and context design, RAG, evals | Data processing, statistics, training and tuning models |
| Typical work | Chatbots, AI assistants, agents, AI features in apps | Recommendation systems, fraud detection, custom classifiers (models that sort inputs into categories) |
| Math needed | A basic understanding is enough to start | Strong (linear algebra, statistics, calculus) |
| Key question | "How do I make this useful and reliable for users?" | "How do I make this model more accurate?" |
The simplest way to put it: ML engineers build models. AI engineers build products with models.
In practice, the line isn't always that clean. AI engineers sometimes fine-tune models (train an existing model a bit more on their own data), and ML engineers sometimes build products on top of model APIs. Job titles also vary between companies, so it's worth reading the actual job description rather than relying on the title.
Key Takeaways
- An AI engineer builds products on top of existing foundation models instead of training models from scratch.
- The model is only one part. The real work is the system around it: context, tools, evals, guardrails, and monitoring.
- Foundation models make prototyping fast, but they bring new challenges: unpredictable outputs, model changes, and cost per request.
- AI engineers and ML engineers overlap. ML engineers focus on building models; AI engineers focus on building products with them.
What's Next
In the next part, I'll move on to the next section of the roadmap and keep building from these basics.