Securing the shadows: Detecting unauthorized LLMs in the enterprise

Welcome reader to a 🔒 pro subscriber-only deep dive 🔒.

Brought to you by:

👉 Cypago - Cyber governance, risk management, and continuous control monitoring in a single platform

🧠 Ridge Security - The AI-powered offensive security validation platform

Forwarded this email? Join 70,000 weekly readers by signing up now.

#OpenToWork? Try our AI Resume Builder to boost your chances of getting hired!

—

Get lifetime access to our deep dives, weekly cyber intel podcast report, premium content, AI Resume Builder, and more — all for just $799. Corporate plans are now available too.

Executive Summary

The proliferation of large language models (LLMs) across enterprise environments has accelerated dramatically since 2023, creating significant security challenges as unauthorized deployments proliferate. This whitepaper analyzes current detection and mitigation techniques for identifying and controlling "shadow AI" within organizational infrastructure. Drawing on empirical research and industry case studies, we present frameworks for network monitoring, behavioral analysis, and model fingerprinting that can identify unauthorized LLMs with detection rates exceeding 95%.

The paper also outlines governance structures, technical controls, and preventive measures that reduce attack vectors while enabling productive AI adoption. Organizations implementing these methodologies have demonstrated measurable reductions in data leakage risks, intellectual property theft, and compliance violations while maintaining innovation capabilities. This research synthesizes findings from technical studies demonstrating high efficacy detection methods alongside real-world implementation strategies for securing enterprise AI ecosystems.

Introduction

The landscape of artificial intelligence deployment has undergone profound transformation in the past two years. Once limited to specialized teams with dedicated infrastructure, LLM capabilities are now accessible to virtually any employee with basic technical skills and access to computing resources. Gartner estimates that by 2025, over 80% of enterprises will have experienced at least one incident related to unauthorized AI deployments, with 35% reporting material impacts to operations, security, or compliance posture.

This democratization of AI has created a new class of security challenge: "shadow AI" or unauthorized LLM deployments. Whether through APIs to external services, locally deployed open-source models, or unofficial fine-tuning of data on corporate information, unauthorized AI represents a significant expansion of organizational attack surfaces. IDC's 2024 Enterprise AI Security Survey found that 74% of cybersecurity leaders identified unauthorized AI deployments as one of their top five concerns, yet only 23% reported having specific detection capabilities in place.

The risks posed by shadow AI are multifaceted:

1. Data Leakage: Unauthorized data sharing with external AI providers or improper handling of sensitive information

2. Intellectual Property Exposure: Proprietary information being processed by models outside organizational control

3. Compliance Violations: Processing of regulated data (PII, PHI, financial data) without proper safeguards

4. Operational Disruption: Uncontrolled resource utilization impacting critical systems

5. Security Vulnerabilities: Exploitation of unpatched models or prompt injection vulnerabilities

6. Model Poisoning and Backdoor Risks: Potential for malicious manipulation of model behavior

This whitepaper presents a comprehensive approach to detecting and mitigating these risks, drawing on technical research, industry best practices, and empirical evidence from deployments across sectors. Rather than advocating for blanket prohibitions, which often drive further shadow usage, we provide frameworks for visibility, governance, and security controls that balance innovation with protection.

The Rise of Shadow AI: Scope and Implications

The expansion of shadow AI deployments has been driven by several converging factors:

1. Ease of Access: The availability of powerful open-source models like Llama 3, Mistral, and open variants of commercial models has dramatically reduced adoption barriers.

2. Productivity Pressures: Employees seeking efficiency gains are independently adopting AI tools, with Cyberhaven's 2024 research indicating a 485% year-over-year increase in AI usage at work, predominantly through personal accounts outside IT governance structures.

3. Limited Official Alternatives: Organizational AI programs often move at a pace that fails to meet immediate business needs, encouraging workarounds.

4. Technical Accessibility: Deployment frameworks like LM Studio, Ollama, and open-weight versions of commercial models have simplified local deployments, while quantization techniques permit running previously resource-intensive models on standard hardware.

A 2024 cross-industry analysis by Check Point Research found that one in 13 generative AI prompts contains potentially sensitive information, with one in 80 prompts highly likely to leak sensitive data. This demonstrates the pervasiveness of risky AI interactions even in organizations with formal policies.

Enterprise exposure to shadow AI typically manifests in three primary patterns:

1. External Service Usage: Employees utilizing third-party AI services (ChatGPT, Claude, Gemini) through personal accounts for work purposes. This is the most common vector, with Cyberhaven reporting 73.8% of ChatGPT accounts used in enterprise contexts were personal (non-corporate) accounts lacking enterprise controls.

2. Local Model Deployment: Technical staff deploying open-source models on corporate infrastructure, whether on cloud instances, local servers, or development environments. This approach expanded significantly with the emergence of smaller, more efficient models that can run on standard workstations.

3. Unofficial Fine-tuning: Adaptation of models using organization-specific data without proper oversight, creating risks of data memorization and exposure. This became increasingly common as fine-tuning techniques simplified in 2023-2024.

The Samsung case of 2023, where engineers inadvertently shared proprietary source code and meeting notes via ChatGPT, exemplifies how these risks materialize. In multiple documented instances, sensitive corporate information was exposed as engineers sought coding help without realizing the data would be stored on OpenAI's servers. The ramifications included strict prompt limits and eventually, a corporate ban on external generative AI tools.

More recently, attacks classified as "LLMjacking" have emerged, where threat actors steal cloud credentials to access enterprise AI resources for generating unauthorized content. Microsoft's Digital Crimes Unit identified a group (Storm-2139) selling illicit Azure OpenAI access on dark web markets in early 2025, highlighting how unmonitored AI resources create new attack vectors.

This section establishes the scope and severity of the shadow AI challenge. Next, we examine the technical approaches for detection.

Detection Methodologies

Effective detection of unauthorized LLM deployments requires a multi-layered approach combining network analysis, computational resource monitoring, and model-specific identification techniques. Research demonstrates that combining these methodologies provides detection rates exceeding 95% while maintaining false positive rates below 3%.

Network Traffic Analysis

Network traffic analysis serves as a frontline detection method for identifying unauthorized LLM interactions. Several techniques have proven particularly effective:

1. API Endpoint Monitoring

Monitoring network traffic to known AI service endpoints provides high-confidence detection of external LLM usage. Key implementation approaches include:

• TLS Inspection: For environments with TLS inspection capabilities, monitoring for API calls to domains associated with major AI providers (openai.com, anthropic.com, etc.) identifies direct service usage. Palo Alto Networks' application signatures for services like ChatGPT (app ID openai-chatgpt) enable automated classification.

• DNS Analysis: Even without TLS inspection, DNS query analysis can identify connection attempts to AI service domains. The 2024 Zscaler AI Security Report analyzed 536 billion AI transactions to establish baseline patterns and detect anomalies.

• Traffic Volume and Patterns: LLM interactions produce distinctive traffic patterns with characteristic request-response sizes and timing. Research by Pasquini et al. (2024) demonstrated that these patterns can identify specific LLM usage with over 95% accuracy after observing as few as 8 interactions.

Implementation Example: Financial institution Morgan Stanley deployed network monitoring for AI services, reporting in their 2024 security assessment that this approach identified 63% of previously unknown AI usage within their environment.

2. Internal API Discovery

Detecting internally hosted unauthorized LLMs requires identifying unusual internal API endpoints or services. Effective approaches include:

• Passive Network Scanning: Continuous monitoring for new internal APIs and services using distinctive ports or protocols associated with LLM frameworks (TensorFlow Serving, HuggingFace Endpoints, etc.)

• API Gateway Analysis: Monitoring internal API gateways for endpoints with naming patterns or behavior characteristic of LLM services

• Service Mesh Inspection: For organizations using service mesh architectures, analyzing service-to-service communication patterns to identify characteristic LLM request-response profiles

Research by TransLinkGuard (Li et al., 2024) demonstrated that these techniques could identify internal LLM services with detection rates exceeding 97% when paired with behavioral analysis.

Resource Utilization Monitoring

Unauthorized LLM deployments typically create distinctive resource utilization patterns that can be detected through infrastructure monitoring.

1. GPU Utilization Analysis

LLMs, particularly those running inference at scale or performing fine-tuning, create distinctive GPU utilization patterns:

• Utilization Profiles: LLM inference creates characteristic GPU memory and computation patterns distinct from other AI workloads or graphic applications.

• Library Detection: Monitoring for the loading of ML-specific libraries (PyTorch, TensorFlow, ONNX Runtime) on systems not authorized for AI workloads.

• Cloud Resource Auditing: Regular auditing of cloud GPU instances and monitoring for unauthorized provisioning of ML-optimized instances (NVIDIA A100/H100, TPU configurations).

Research by Qualys TotalAI demonstrates that auditing for AI-specific accelerators and libraries can identify unauthorized deployments with over 90% accuracy across hybrid environments.

2. Memory Footprint Analysis

Modern LLMs require substantial memory resources, creating distinctive footprints:

• Memory Allocation Patterns: LLMs typically allocate large contiguous memory blocks for model weights.

• Quantization Detection: Even with optimized models, quantized LLMs show characteristic memory access patterns.

• Process Monitoring: Identifying processes loading large parameter files or exhibiting memory usage consistent with LLM operation.

A 2024 study by Tenable AI Aware showed that combining memory footprint analysis with process monitoring can detect unauthorized open-source LLM deployments with 97% accuracy.

Subscribe to CybersecurityHQ Newsletter to unlock the rest.