Editor’s Note: This blog was originally published as part of our ongoing research into AI security in telecom and written by Khalifa University’s Dr. Lina Bariah. We’re republishing and updating to coincide with our upcoming webinar, “Social Engineering the Algorithm: A Red Teamer’s Guide to AI”, taking place on October 29. The insights from our TelecomGPT red-team exercise directly inform the session’s focus on AI-driven social engineering threats and defences.
In telecom, large language models are being embedded into mission-critical systems for both network operations and customer interactions. However, as we tested TelecomGPT in a red-team environment, we found that adversarial prompts, particularly those delivered through phishing, role-playing, or ‘developer’ spoofing, offer genuine opportunities for exploitation. These findings are especially relevant as we enter our live session on AI and social engineering: when AI systems can be manipulated themselves, the human and network vectors of fraud expand dramatically.
Key Red Team Findings & Their Relevance to Social Engineering
- The red-team exercise used 25 adversarial prompts specifically crafted for telecom domain vulnerabilities — covering historical milestones, spectrum misinformation, performance claims, and roleplay override attempts.
- An authority-override tactic, such as claiming to be a developer or instructing the model to ignore its rules, successfully bypassed safeguards. This highlights the risk of models implicitly trusting user roles without verification.
- Prompts that were technically plausible (e.g., “Wi-Fi 6 in 3.5 GHz”, “5G core not cloud-native”) exploited the model’s tendency to rely on pattern-matching rather than true domain understanding, leading to confident but incorrect responses (hallucinations).
- Attack categorisation:
Other (generic misinformation) accounted for 43.5% of successful prompts, indicating that plausible but loosely defined claims often bypass defences.
For details, please refer to this article.
Why the Telecom AI Case Matters to Social Engineering
The telecom-AI findings underscore that the attack surface is shifting: it’s not just devices or networks anymore, but models, APIs, and trust frameworks.
Our upcoming webinar will bring this into focus: how AI is used by attackers (deep-fake voice, automated phishing) and for defence (verification, anomaly detection). The telecom findings provide a clear blueprint for what to assess, protect, and train.
Implications for AI-Powered Social Engineering
The red-team results show that combining domain-specific knowledge with roleplay framing is highly effective for attackers, similar to tactics used in deep-fake voice, AI-generated phishing, or SMS attacks. For telecom, fintech, and identity sectors, large language models (LLM) remain vulnerable. When deployed in mission-critical environments, their weaknesses can contribute to social engineering-driven fraud.
Key defensive priorities:
- Retrieval-based verification: Models should cross-check inputs against verified databases, such as spectrum assignments or technical standards, rather than relying solely on pattern matching.
- Multi-layer guardrails: Implement dynamic filters or human review triggers when prompts display known jailbreak patterns, such as authority override tactics, as identified by the red team.
- Structured, regular red-teaming of deployed models – simulating how attackers adapt. The GSMA study emphasises this as a best practice for trustworthy telecom AI.
Join the Upcoming Webinar to Learn More
In the upcoming session on October 29, “Social Engineering the Algorithm: A Red Teamer’s Guide to AI“, our speakers will share insights into how AI is being used by both fraudsters and defenders. The TelecomGPT red-team findings map directly into that theme:
- Fraud via AI isn’t just phishing — it’s model manipulation, role-play attacks, and domain-knowledge abuse.
- Defending against it means applying telecom-grade AI rigor (verification, standards, role checks) to fintech, identity, and telecom sectors.
- Attendees will gain actionable insights grounded in empirical red-team data (such as those from TelecomGPT) on how to build resilient, human-augmented defenses.
Register today to secure your place at this important session: