FORGE-02
Network & Interconnect Vulnerabilities
Weak fabric isolation, management protection, and monitoring can expose InfiniBand, RoCE, and RDMA paths to discovery, abuse, or lateral movement.
FORGE-02 Network & Interconnect Vulnerabilities
Definition
Network & Interconnect Vulnerabilities occur when the high-performance fabrics used by AI clusters, such as InfiniBand, RoCE, and RDMA-based communication paths, are not properly isolated, monitored, or protected. Because these fabrics are optimized for throughput and low latency, weak controls can expose backend AI environments to discovery, lateral movement, data interception, denial of service, or workload manipulation.
Description
Modern AI infrastructure relies on high-performance interconnects to move model weights, gradients, checkpoints, and other data across GPU clusters. These environments commonly use InfiniBand and RoCE, often through RDMA, to provide low-latency communication with less dependence on the traditional kernel networking path. This improves performance, but can reduce the visibility and control available to conventional network and host security tools.
A recurring weakness is the gap between the frontend network and the backend training fabric. Frontend networks used for APIs, orchestration, monitoring, and administrative access often receive stronger access control and monitoring. Backend fabrics that carry distributed training and storage traffic may be flatter, less inspected, and weakly segmented. When fabric isolation, management controls, or workload boundaries are weak, even routine diagnostic queries from a tenant node can reveal topology, endpoints, and management details that should not be visible.
The security model depends on the transport. InfiniBand uses fabric-specific controls such as partitions, management keys, and subnet-management policies. RoCE runs over Ethernet and does not use an InfiniBand Subnet Manager, so it depends more heavily on Ethernet/IP segmentation, access controls, and device isolation. In both cases, weak defaults or poorly protected management interfaces can turn a performance fabric into a path for reconnaissance, lateral movement, or disruption.
Impact and Failure Modes
Backend traffic may lack cryptographic protection
Fabric traffic is often not encrypted by default. In native InfiniBand, keys such as P_Key and M_Key are access-control mechanisms, not cryptographic protections. In RoCE and other Ethernet-based deployments, model weights, gradients, and training data may traverse the backend network without wire-level encryption unless operators explicitly deploy protections such as MACsec or IPsec where supported.
InfiniBand partitioning may be weak or left at defaults
InfiniBand relies on P_Key for partition-level isolation. If no partition configuration is applied, OpenSM can place end ports in the broad default partition, commonly 0x7fff, increasing lateral reachability and the blast radius of a compromised node. Hardened or vendor-managed deployments may override these defaults, so the exposure depends on the actual subnet-manager configuration.
InfiniBand management-plane protection may be weak
InfiniBand uses M_Key to protect management operations on switches and adapters. In OpenSM, documented defaults such as m_key=0 and m_key_protection_level=0 leave management-key protection disabled or at the weakest level unless operators harden them explicitly. When combined with broad fabric reachability, this can expose topology, port state, and management functions that should be restricted.
RDMA memory-access credentials may be abused
RDMA uses keys such as rkeys to authorize access to registered memory regions. Research has shown that weaknesses in RDMA security mechanisms can enable unauthorized reads or writes in some environments, with limited visibility to normal host-based security tools.
Fabric management software can become a control point
Vulnerabilities in fabric management software can expose the management plane to privilege escalation, tampering, denial of service, and broader fabric compromise. For example, CVE-2024-0130 affected NVIDIA UFM and could allow improper authentication via the Ethernet management interface
RDMA isolation can be weak in containerized environments
In Kubernetes and similar platforms, shared HCA access or shared RDMA-capable devices can weaken workload isolation if operators rely only on ordinary pod-network controls and do not separately isolate RDMA devices, namespaces, or virtual functions.
Prevention and Mitigation Strategies
For providers and datacenter operators
- Enforce protocol-specific fabric controls, do not rely on defaults: For InfiniBand, explicitly configure and review controls such as P_Key, M_Key, SM_Key, SA_Key, and allowed_sm_guids. For RoCE, which runs over Ethernet and has no subnet manager, apply explicit Ethernet/IP segmentation such as VLANs and ACLs. Many of these protections are weak or off in upstream defaults, so confirm they are actually set rather than assuming the fabric is isolated.
- Harden and isolate the fabric management plane: Restrict access to subnet and fabric management interfaces, keep management traffic off untrusted networks, limit topology discovery and sensitive management operations from untrusted nodes, and monitor counters and management events for signs of spoofing or misconfiguration.
- Provide backend isolation comparable to frontend isolation: Give the RDMA training fabric documented segmentation, workload boundaries, and tenant isolation rather than operating it as a flat internal network.
- Use wire-level encryption where supported and required: For high-sensitivity workloads, deploy MACsec or IPsec on RoCE and other Ethernet-based fabrics where supported. Note that native InfiniBand keys such as P_Key and M_Key are access controls, not encryption, and in-fabric encryption on native InfiniBand is hardware-dependent and not always available, so treat backend traffic as unencrypted unless a specific protection is in place.
- Harden orchestration and RDMA device isolation: Use Kubernetes NetworkPolicies for ordinary pod traffic, assign unique least-privilege service accounts per job, and regularly audit and rotate credentials to limit the blast radius of a compromise. Do not assume Kubernetes-layer controls alone secure RDMA-capable traffic, because RDMA may bypass, or be enforced outside, the ordinary pod-networking path where NetworkPolicies apply. For RDMA-capable workloads, isolate the underlying devices and network namespaces separately, for example by assigning dedicated SR-IOV virtual functions to each tenant or job rather than having multiple tenants share the same HCA interface.
For customers evaluating a provider
- See how much of the fabric you can enumerate: From your node, run standard diagnostic queries such as ibstat, ibv_devinfo, saquery, sminfo, and, where permitted, ibnetdiscover. You should expect to see enough information to operate your assigned ports, but not enough to map unrelated tenants, switch topology, or management-plane details. If you can enumerate the full fabric, discover other tenants’ nodes, or access management functions outside your allocation, that suggests weak fabric partitioning or management-plane protection.
- Check your partition membership: Confirm which P_Key partitions your ports belong to, for example via the P_Key table exposed under /sys/class/infiniband/ <device>/ports/<port>/pkeys/ or with vendor tools. If your port appears to be in the broad default partition, commonly shown as 0x7fff or full-membership 0xffff, with wide reachability rather than a dedicated tenant/job partition, isolation may have been left at defaults.
- For RoCE / Ethernet fabrics, scan your backend subnet: Since RoCE rides on Ethernet, ordinary IP tools apply: look at what's in your ARP table, check your subnet mask (is the backend a flat /16 with everyone on it?), and see whether you can reach other hosts' IPs and ports on the storage or training network. Reaching other tenants' addresses indicates missing VLAN/segmentation.
- Look at port counters for noisy-neighbor signs: Your own port error and congestion counters, for example via perfquery or /sys/class/infiniband/<device>/ports/ <port>/counters/, can show link errors, congestion, retransmits, drops, or other symptoms that may point to shared-fabric contention. These counters are useful evidence, but they should be interpreted against a baseline and your workload profile rather than treated as proof of co-tenant interference.
Attack Scenarios
An attacker gains a foothold on a tenant node through a compromised container, workload, or account. From that node, they use fabric diagnostic and management queries to map the backend interconnect, discover topology, identify node addresses, and inspect port or switch state. If the InfiniBand fabric is weakly partitioned, such as broad use of the default P_Key 0x7fff, and management protections such as M_Key are unset or weakly configured, the attacker may gain visibility or influence over fabric behavior outside their allocation. In a hardened environment, tenant nodes should not be able to enumerate unrelated topology or perform management actions beyond their assigned fabric scope.
An attacker abuses weaknesses in RDMA memory-access protections to obtain or misuse credentials such as rkeys for registered memory regions. They then issue one-sided RDMA reads or writes against memory that should not be accessible to them. Because one-sided RDMA operations can complete without active involvement from the remote CPU, the attacker may extract training data, checkpoints, gradients, or model-related buffers with limited visibility to normal host-based security tools.
In a shared AI environment, a malicious tenant abuses RDMA-capable devices, shared HCAs, or interconnect resources to consume bandwidth, trigger congestion, or exhaust RNIC resources. Neighboring distributed training jobs may experience degraded throughput, higher latency, failed synchronization, or stalled execution. The attacker does not need to read or modify another tenant’s data. The failure is that one tenant can materially degrade another tenant’s workload through shared low-level fabric resources.
References
NVIDIA InfiniBand Security Overview and Guidelines. NVIDIA. Vendor guidance on IB fabric security controls including P_Key, M_Key, and subnet management.
OpenSM Partition Configuration. linux-rdma. P_Key partition configuration reference for InfiniBand subnet managers.
https://github.com/linux-rdma/opensm/blob/master/doc/partition-config.txt
NVIDIA MLNX_OFED, Kubernetes with Shared HCA. NVIDIA. RDMA device sharing and isolation in Kubernetes environments.
https://docs.nvidia.com/networking/display/mlnxofedv23106161lts/Kubernetes-with-Shared-HCA
Distributed Training Attack Surface. Red Teams AI. Security analysis of distributed training communication and orchestration.
https://redteams.ai/topics/training-pipeline/advanced/distributed-training
opensm(8) man page, M_Key Configuration. Debian. OpenSM management key defaults and hardening options.
https://manpages.debian.org/testing/opensm/opensm.8.en.html#MKEY_CONFIGURATION
OpenSM Source, osm_base.h. linux-rdma. Default M_Key and protection level constants in source code.
https://github.com/linux-rdma/opensm/blob/master/include/opensm/osm_base.h
Set up InfiniBand on HPC VMs. Microsoft Azure. InfiniBand configuration for cloud HPC and AI workloads.
https://learn.microsoft.com/en-us/azure/virtual-machines/setup-infiniband
CVE-2024-0130. NVIDIA UFM improper authentication via Ethernet management interface.
CVE-2024-0101. Mellanox OS ipfilter bypass.
ReDMArk: Bypassing RDMA Security Mechanisms. Rothenberger et al., USENIX Security 2021. Demonstrates weaknesses in RDMA memory-access credentials that enable unauthorized remote reads and writes.
NeVerMore: Exploiting RDMA Mistakes in NVMe-oF Storage Applications. Taranov et al., arXiv 2022. RDMA vulnerabilities in storage applications.
Noisy Neighbor: Exploiting RDMA for Resource Exhaustion in Containerized Clouds. SecAssure 2025. RDMA-based denial-of-service in shared container environments.
RFC 4391, Transmission of IP over InfiniBand. IETF. Section 13 covers security considerations for IPoIB deployments.
NIST SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations.