When the Chatbot Went Wrong: A Governance Recovery Story

The situation

A medium-sized community housing not-for-profit deployed a tenant-facing chatbot to handle common enquiries: maintenance requests, rent payment queries, and after-hours contact details. The chatbot was sourced from a SaaS vendor and went live after a management decision, without a formal board paper or prior approval.

Six months later, a board member asked how the chatbot was performing. During that conversation, management discovered that nobody had checked the vendor's default data retention settings.

What they did

Management commissioned a brief internal audit of the vendor's data handling arrangements. The audit found that chat transcripts — including exchanges where tenants had shared personal details such as their name, unit number, and the nature of their maintenance issue — were being retained on the vendor's servers by default, under the vendor's standard terms. There was no tenant-facing privacy notice on the chatbot interface.

The board was briefed at the next meeting.

What it cost

The chatbot subscription was approximately $4,200 per year. The internal audit cost two days of management time. The vendor review, and the subsequent negotiation to have retention settings changed, took a further week of management time and a legal review of the vendor's updated terms, which cost around $800.

Total one-off governance cost: approximately $1,600 in staff and legal time, on top of the ongoing subscription.

Where it went wrong

Three things went wrong, and they are worth naming plainly.

First, the chatbot was deployed without board approval. This is not unusual — management deploys software regularly — but client-facing AI tools that handle personal information sit in a different risk category from internal productivity tools. The board should have been asked.

Second, no one checked the vendor's default data settings before go-live. Vendor defaults are set for the vendor's convenience, not for the client's compliance. Assuming that defaults are safe is a common and avoidable mistake.

Third, there was no privacy notice on the interface. Tenants were not told they were talking to an AI, or that their conversation might be retained.

What the board did

At the meeting where the audit findings were presented, the board passed a motion to pause the chatbot service immediately, pending a vendor review and a board decision on whether to relaunch.

Over the following six weeks, management:

• Negotiated with the vendor to confirm that retention could be disabled and that existing transcripts would be deleted;
• Obtained legal confirmation that the revised terms met the organisation's obligations under the Privacy Act;
• Drafted a privacy notice for the interface and a clear "You are talking to an AI" disclosure;
• Prepared a board paper recommending relaunch with those safeguards in place.

The board approved the relaunch at the following meeting and adopted a new policy: any client-facing AI deployment requires prior board approval, with a management paper covering data handling, privacy disclosures, and human escalation paths.

The outcome

The chatbot relaunched two months after the pause, with vendor data retention disabled, a privacy notice added to the interface, and a human escalation path made available at every step of the conversation. No tenants filed complaints during or after the pause. The board now receives a six-monthly report on chatbot performance and any privacy-related incidents.

Lessons for directors

• Prior approval for client-facing AI is not bureaucracy — it is risk management. Internal tools can be managed at management level; tools that interact with clients, tenants, students, or beneficiaries deserve a board paper before go-live.
• Vendor data defaults are the risk. SaaS vendors retain data because it is in their interest to do so. Turning off retention, or negotiating retention limits, requires someone to ask the question — and that question needs to be on a checklist, not left to chance.
• A pause is a governance success, not a failure. The board in this story identified a problem, stopped the harm, fixed the root cause, and relaunched with better controls. That is what good governance looks like. The mistake was not deploying a chatbot; the mistake was deploying one without asking the right questions first.