Australian Businesses Need to Treat AI as a New Attack Surface
Estimated Reading Time: 8 minutes
- AI should be treated as a new attack surface involving various cybersecurity risks.
- Common threats include data leakage, prompt injection, and AI chatbots acting as backdoors.
- Organisations need to integrate AI into their cybersecurity strategies effectively.
- Practical takeaways and guidance from ACSC and ASD can help mitigate risks.
- Regular assessment and employee training are vital to counter AI-related threats.
Table of Contents
Key AI Threats for Australian Businesses
AI introduces familiar cyber risks in new contexts, such as web chatbots, internal copilots, and decision engines. Given the breadth of AI applications, organisations must be cognizant of the specific risks they face, including:
- Data Leakage via AI Tools: AI systems handling business or customer data must adhere to strict data security controls such as access control and encryption.
- Prompt Injection and “AI Jailbreaks”: These attacks can exploit AI systems to override guardrails or improve an attacker’s chances of success.
- AI Chatbots as Backdoors: Integrating AI chatbots with internal systems without proper security measures can create vulnerabilities that attackers can exploit.
- Supply Chain Compromise: Leveraging third-party AI models may expose businesses to risks associated with malicious code or compromised vendor environments.
- Social Engineering and AI-powered Scams: Attackers can create highly convincing scams using generative AI, complicating traditional cybersecurity measures.
Website Chatbots Leaking Data
AI chatbots are becoming increasingly popular, allowing businesses to enhance customer engagement and support. However, when integrating chatbots with knowledge bases like SharePoint or CRMs, unintended data disclosures can occur. Recent incidents have highlighted cases where attackers exploited poorly configured AI chatbots to exfiltrate sensitive data.
Common Failure Modes Include:
- Indexing Incorrect Content: Staging sites or internal wikis accidentally exposed to chatbot interactions.
- Over-Permissive Retrieval: Allowing chatbots to access any document without applying proper role-based controls.
- Logging Personal Information: Retaining chat transcripts that contain sensitive data can lead to breaches when logs are improperly handled.
Practical Takeaways:
- Secure Your Knowledge Base: Treat chatbot knowledge bases like data stores. Implement classification, access controls, and data minimisation practices.
- Implement Retrieval-Time Access Control: Ensure that chatbots can only access content appropriate for the current user.
- Disable Conversation Training: Where possible, turn off training on conversations for SaaS AI solutions and implement policies to prevent sensitive data sharing.
Prompt Injection and Jailbreak Attacks
Prompt injection poses a significant risk as attackers can input crafted instructions that manipulate AI behaviour, leading to data leakage or harmful outputs. Real-world examples of prompt injection attacks reveal vulnerabilities present in common Australian businesses.
Common Scenarios:
- A customer uploads a document containing hidden prompt instructions, leading the AI support assistant to exfiltrate sensitive data.
- An attacker embeds instructions in a public comment on a business’s blog, manipulating the chatbot into revealing customer payment details.
Practical Takeaways:
- Adopt OWASP Guidelines: Apply the OWASP Top 10 for Large Language Model (LLM) applications, focusing on input validation and adversarial suffix detection.
- Segment High-Risk Tools: Keep sensitive systems and actions under human oversight to prevent escalated permissions through misconfigured AI agents.
- Limit AI Insight into Secrets: Avoid exposing API keys, credentials, or sensitive data within the AI’s accessible context. Use secure middleware for sensitive assets.
AI as a Backdoor into Your Environment
Prominent security researchers warn that deeply integrated AI systems can inadvertently create backdoors for attackers. Failure to treat AI applications as privileged applications can lead to severe breaches where attackers can exfiltrate valuable data assets.
Concrete Attack Paths:
- Vulnerability Exploitation: Compromise of chatbot applications can provide attackers with direct access to internal data stores.
- Stolen API Keys: Access to sensitive systems can be gained through compromised API keys used for AI integrations.
- Misconfigured Single Sign-On: Poor OAuth configurations can allow lateral movement within an organisation’s network.
Practical Takeaways:
- Implement Application Security Controls: Apply secure coding practices, regular testing, and a comprehensive patching strategy.
- Secure AI Connectors: Lock down API scopes to the principle of least privilege and ensure centralised logging for monitoring.
- Include AI Systems in Incident Response Plans: Incorporate these systems into your organization’s incident response strategy as critical components.
Model Poisoning and Supply-Chain Risks
The ASD’s Information Security Manual (ISM) explicitly highlights the necessity of curbing “unattended or harmful behaviours” in AI models. Businesses must be proactive in mitigating risks associated with compromised model weights, tampering with open-source models, and data poisoning.
Key Risks Include:
- Downloading Compromised Models: Acquiring pre-trained models from unverified sources can embed backdoors.
- Data Poisoning: Attackers can corrupt training datasets leading to undesirable AI behaviours.
Practical Takeaways:
- Regularly Assess Supply Chain Security: Source models and AI components solely from trusted vendors and maintain a solid inventory system.
- Isolate Training and Production: Maintain separation between training and production environments with strict access controls.
- Implement Anomaly Detection: Deploy monitoring tools to identify deviations in AI model behaviour or output quality.
Deepfakes, Scams, and Social Engineering
Deepfake technology is increasingly being leveraged for malicious activities such as phishing and business email compromise. Australian organisations must prepare for the reality that convincingly simulated audio or video can be produced with AI, enabling fraudster success in acquiring sensitive information.
Potential Scenarios:
- Deepfake audio of an executive instructing staff to process urgent payments.
- AI-generated websites impersonating a legitimate business to solicit customer credentials.
Practical Takeaways:
- Strengthen Verification Processes: Implement secondary verification for substantial financial changes.
- Employee Training on AI Scams: Regularly update training content, integrating the latest examples of AI-generated threats.
- Encourage Verified Communication: Guide customers on how to securely interact with your business.
ACSC and ASD/ISM Guidance on AI
The ACSC’s guidance for AI ethics and security significantly encourages organisations to formalise their approach to integrating AI while adhering to established governance practices. Core principles from these agencies emphasise the importance of robust security infrastructure in utilising AI systems.
- Implement Essential Eight Controls: Ensure your organisation adopts basic hygiene practices, like regular patching and multi-factor authentication.
- Secure-by-Design Approach: Apply least-privilege access principles to AI applications and data handling.
- Align with Privacy and Governance Frameworks: Fulfil compliance with Australian privacy laws and ethical AI guidelines.
Practical Checklist for Businesses
To help organisations navigate the complexities of AI-related cybersecurity threats, use the following checklist:
- Inventory AI Systems: Identify all AI applications and classify the data they access or store.
- Update Data Policies: Adapt data classification and access controls to encompass AI prompts and logs explicitly.
- Implement OWASP Controls: Follow secure development guidelines for any AI features and integrations.
- Vendor Due Diligence: Ascertain that AI product vendors undergo stringent security assessments.
- Embed AI Risks in Staff Training and Incident Response: This allows for a holistic approach to cybersecurity that takes into account the emergent risks associated with AI.
Conclusion
As AI becomes a staple in Australian business operations, it’s critical to view it through the lens of cybersecurity. From data leakage to deepfakes, the risks associated with AI are real and present. By understanding these threats and implementing robust security measures, Australian organisations can better protect themselves against potential breaches.
At Summit Cyber Group, we are committed to enhancing your organisation’s cybersecurity maturity. Let us help you navigate the complexities of AI security and ensure you have the appropriate controls in place. Contact us to discuss how we can assist you in developing a trust-based cybersecurity strategy tailored to meet your unique needs.
For more information, visit our website: Summit Cyber Group.
FAQ
What are the main cybersecurity risks associated with AI?
The main risks include data leakage, prompt injection, supply chain compromises, and deepfake technology used for scams.
How can businesses mitigate AI-related security threats?
Businesses can implement essential security controls, secure AI applications, and ensure regular assessments of their cybersecurity posture.
What guidelines should businesses follow for AI security?
Businesses should adhere to ACSC and ASD guidance, follow the OWASP guidelines, and implement a secure-by-design approach for AI systems.
How can employee training help in combating AI threats?
Regular employee training can help staff recognize AI-related scams and keep them informed about the latest cybersecurity practices.
