Chat messages, uploaded files, search results and web pages all reach the model as text. We test the routes in and out of your AI application, then help your team close the gaps.
- ChatHostile prompts
- Uploaded fileHidden text
- Search indexWho is asking
- ModelCan be fooled
- AnswerLinks and markup
- ActionsPermission checks
Example
The same hidden instruction, before and after a fix.
A customer uploads a contract to your support assistant. Inside it is a line written for the model, not for people. Switch between the two versions to see what changes.
Can you summarise this contract for me?
contract-renewal.pdfHere's a summary: 12 months, 30 days' notice, prices fixed for year one. The latest tickets are below.
- Customer 1027d.m•••@•••Refund for order 5531
- Customer 2211k.o•••@•••Change of address
- Customer 0934s.a•••@•••Card declined twice
What the search service did
search_tickets(query: "latest")5 tickets from 4 customers
The model followed the hidden text, and search let it see everyone's tickets.
Where untrusted text gets in.
Chat.
Anyone can type anything, including instructions meant for the model.
Uploaded files.
PDFs, spreadsheets and images can carry text people never see.
Your own search index.
Search can surface pages written by someone else, or meant for someone else.
Web pages and tools.
Browsing and tool results bring outside text into the conversation.
Fixes that hold when the model is fooled.
No model can be made impossible to fool. So we put the limits around it, where they keep working.
Answers
Treat every answer as untrusted.
An answer can carry links, code or markup that your app then runs.
We test what your front end and downstream systems do with each one.
Your order 5531 ships on Friday. Tracking is on its way.
img src = outside-host/pixel.png?d=order-5531The link would have sent order details to an outside site.
Actions
Let the user approve what leaves.
If the model suggests sending, sharing or deleting, the user confirms first.
We test that no answer can trigger an outbound action on its own.
The assistant wants to
Email a copy of this chat to an address outside your company.
- To
- someone@outside-domain
- Includes
- Full chat, 1 attachment
Outbound actions need your yes
Questions we hear.
Do you test the model itself?
We test the application around it: inputs, search, answers and actions. That's where most real weaknesses sit. If you also need the model's behaviour evaluated, we'll say so.
Will testing touch real customer data?
Only if you approve it in writing. We usually test in a staging copy with realistic test accounts.
Does it matter which AI provider we use?
Not much. We test applications built on hosted AI services and on models you run yourself. The risks sit in how your application uses the model.
What do we get at the end?
A short list of confirmed weaknesses. Each comes with a test you can rerun, the fix we recommend and a retest once it's fixed.
How do we start?
Start with a free Technical Audit of one AI application. If it needs deeper testing, we'll scope that with you.
Test one AI feature, start to finish.
Bring its prompts, data sources and the systems that use its answers. We'll test the whole path.