Shadow AI: Why Uploading Submission Documents into ChatGPT Is a Bad Idea

• Shadow AI (staff using unapproved tools on public data) is already happening in councils and municipalities, and the Medicare/OpenAI breach illustrates the consequences.
• Uploading applications into personal AI accounts sends data offshore, risks it training the model, breaks privacy law, and leaves no audit trail.
• Archistar keeps data sovereign and rules-based, with human sign-off.
• Discover five questions councils should ask any AI vendor.

Shadow AI refers to staff using AI tools their organization hasn’t approved, through accounts it doesn’t control. In planning and building, it usually looks like this: Someone drags a development application or a set of plans into ChatGPT and asks it to “check this against the code.”

I get why people do this. The queue is long, the tools are right there, and they are impressive. But it’s a really bad idea, and we have already seen this happening inside councils and state government departments. If you work in a council or a city and you are doing this, you should know better, I will explain why.

This week the Australian Prime Minister revealed that an OpenAI agent got into the Services Australia Medicare statistics portal back in June. It was given a harmless research task, couldn’t find what it wanted on the public pages, and in the Acting PM’s words, “effectively climbed over” the fence. OpenAI took until September 10, 2026, to tell the government. There’s now a forensic investigation, a taskforce, and a PM who has rung Sam Altman to express Australia’s “extreme concern.”

To be fair, the government and OpenAI both say no personal Medicare details were accessed. But the country still lost its mind because the idea of an AI company getting hold of our information freaks people out.

Now think about what happens every day inside some organizations. Nobody has to climb a fence. A curious staff member just opens the front door and hands the public’s personal information over. They include names, addresses, floor plans of people’s homes, and neighbors’ complaints, and they upload it all into a personal AI account like ChatGPT, Claude, or DeepSeek. These are people who are supposed to be protecting personal information. If the Medicare story worried you, this should worry you more.

A development application has the owner’s name, address and contact details, neighbors’ objections, floor plans of someone’s home, and the architect’s copyrighted work. Council holds it for one purpose: assessing the application. Paste it into a personal AI account and:

1. It leaves the country of origin. ChatGPT and Claude usually process it in the US. A Chinese tool like DeepSeek or Kimi K2.0 stores it under Chinese law. And there is no Australian or other country option to fall back on. None of the major foundation models people use are built or run by an Australian company.
2. It may be used for training. On personal plans, conversations can be used to improve the model unless someone has switched that off. Once someone’s details are part of what a model has learned, you can’t pull them back out. The conversation also sits on the provider’s servers, where it can be read by reviewers or exposed in a breach.
3. It breaks the rules you work under. In Australia, that’s state privacy law, the PPIP Act in NSW, the Information Privacy Act in Queensland, the PDP Act in Victoria, plus records law and your own council’s ICT and AI policies. Canadian municipalities have MFIPPA and FIPPA, and US cities have their own state privacy and public records obligations. Different acts, same principle.
4. There is no record. Council can’t audit what was uploaded, what came back, or whether the answer ended up in an assessment report.

This isn’t about OpenAI, Anthropic, or any one model being bad. Plenty of good products, ours included, run on foundation models. The problem is the path the data takes. A personal chat account has no contract with council, no data residency, no audit trail, and no guardrails.

I want people in cities, counties, and councils playing with AI. We saw plenty of it this month at AWS LG Innovate in Melbourne, where more than 20 councils from around Australia shared the AI work they’re doing in customer service and communications, and how they’re making permit and development application information easier for residents to find. Really good work, done through proper channels.

The line in the sand is using the public’s data. The moment a real applicant’s documents go into an unapproved tool, it stops being an experiment and becomes a privacy breach.

There’s also a second risk, and it hits planners directly. A general chatbot reading a planning scheme from scratch will give a confident answer that is sometimes plain wrong, like a setback that doesn’t exist or a missed overlay. Now the planner has to check everything, which is more work than before, and every bad answer is another reason not to trust AI at all.

We’ve spent years working out how to use AI on planning and building applications without these problems. I can’t give away the recipe, but I can explain the principles.

Certified, not self-declared. Archistar holds and maintains international standards certifications: ISO/IEC 27001 for information security and ISO/IEC 42001 for AI management systems. Our controls have passed the ICT procurement reviews of Austin, New York, Los Angeles, Surrey, and Markham. And here in Australia, our controls have passed the NSW and South Australian state governments; NSW councils like Cessnock, Burwood, Canterbury-Bankstown and Bayside; and a handful of councils in Western Australia.

Data sovereignty. For Australian councils, every document, piece of personal information, and backup is stored and processed in Australia, and our North American cities get the same in their own jurisdictions. No offshore staff can access it.

Inference stays in the same place. Inference is the moment the AI reads something and produces an answer. Ours runs in our own secured cloud environment in the same region as the data, through AWS Bedrock. No public chatbot, no offshore processing.

Nothing is used for training. Council data and applicant documents are never used to train or improve any foundation model, and under the AWS terms we operate on, Bedrock content isn’t shared with the model providers.

Accuracy comes from the rules, not the model. We digitize each council’s planning controls into structured rules, pinned to the version in force on the day. Measurable controls like height and setbacks are calculated, not guessed, and the AI explains the result in plain language. It can describe a control; it can’t invent one.

Tested before it ships. Our AI reads plans like a trained eye, and every result links back to the exact spot on the drawing and the rule it came from. Every change is tested before it reaches a customer, including running the same application 20 times to catch any result that flips. Think of it as a spell-checker for building plans.

A human makes the call. We don’t issue approvals or refusals. Every finding goes to a planner to accept, amend, or override, and every prompt and response is recorded, so the municipality or council can audit it or defend it on appeal.

In the last week alone, my LinkedIn feed has shown me somewhere between 10 and 20 new AI tools for planning and building assessment. A lot of them were built over a weekend. Good on them for experimenting. It keeps us on our toes. But getting one demo to work on one application isn’t cracking it.

At Archistar, we’ve put over 200,000 real submissions through our platform, digitized 14,000 to 15,000 planning-rule types, and run models for around 85 cities globally. Every one of those is a progressive council or city that was willing to have a go, and what we’ve learned from them goes straight back into the platform. Any Australian council that wants to have a go now gets the benefit of all of it from day one.

If I was a council or municipality, I’d want a partner who has been doing this for years, not someone who turned up last week.

If a vendor can’t give you a straight answer on any of these, keep walking.

1. Can we see your certificates? Not “aligned with” or “working towards.” Ask for the actual ISO/IEC 27001 and 42001 certificates, with the certifying body and expiry date. Plenty of vendors say they have them and can’t produce them when asked.
2. Can you show us a live product? Not a video or a slide deck. A working product, running on a real application, where every finding points back to the rule and the spot on the drawing.
3. Where does our data go, and is it used for training? Find out where it’s stored, where the AI processing happens, and who can access it. Get it in writing. Better still, ask for their solution architecture diagram.
4. Where is your AI governance documented? A mature vendor has a published trust center covering its AI governance policy and the quality gates every change passes before release.
5. Who is using it in already, and can we call them? Real councils, cities, and counties; real submissions; and a reference who will pick up the phone.

These tools should exist to make life easier for planners and put a real dent in approval times. A tool that leaks the public’s data or hands planners wrong answers to double-check does the opposite.

So, keep experimenting—just not with your applicants’ documents. And if you’re looking at an AI vendor, take these five questions into the room with you.

If you want to see how we answer them, send me a message, give me a call, or book a meeting with the Archistar team, and we’ll walk you through it.