What Is a VPN? IPsec, WireGuard & Architecture

Underlay, overlay and where tunnels terminate

The underlay is the physical forwarding network: ISP routers, transit providers and internet exchanges that move packets between public addresses. The overlay is the logical point-to-point link the VPN creates between two tunnel endpoints. Underlay routers only see the outer packet; they never need routes to the private networks inside.

Endpoint placement defines the design. In a Site-to-Site VPN, tunnels terminate on firewalls or routers at each location, and ordinary LAN hosts send unencrypted traffic to their default gateway without any VPN software. In a Remote Access VPN, the tunnel terminates inside the user device on a virtual network adapter, so the gateway must decide explicitly what that user is allowed to reach once connected.

Four functions of an enterprise VPN

A production VPN combines four separate functions. Missing any one of them produces either a broken tunnel or an insecure one.

• Encapsulation: the original packet, including its IP header, is carried inside a new packet with a routable outer header, which lets private RFC 1918 addresses cross the internet.
• Encryption: symmetric ciphers such as AES-GCM or ChaCha20-Poly1305 turn the inner packet into ciphertext that intermediate devices cannot read.
• Integrity and anti-replay: an authentication tag or HMAC detects modification, and a sequence-number window rejects replayed packets (RFC 4303 for ESP).
• Access control: authenticating the peer does not authorise everything. The terminating gateway must still apply firewall rules and least-privilege policy to the decrypted traffic.

Security boundary: gateway-to-gateway is not end-to-end

In a Site-to-Site design, encryption exists only between Gateway A and Gateway B. The LAN segments from the client to Gateway A and from Gateway B to the server are carried in clear text on the local network.

If the threat model includes insiders, lateral movement or shared local switching, protect the application itself as well: HTTPS/TLS for web and API traffic, SIP over TLS and SRTP for voice. A VPN adds a protected transport between sites; it does not replace application-layer encryption on the LAN.

Outbound: from Client A to Server B

Client A builds a packet from 10.10.10.11 to 10.20.20.20, for example a TCP SYN to port 443. The destination is off-link, so the packet goes to Gateway A. Gateway A then processes it according to the IPsec architecture in RFC 4301, and Gateway B reverses the process.

• Route lookup: Gateway A resolves 10.20.20.0/24 to the virtual tunnel interface.
• Policy lookup: the Security Policy Database (SPD) marks this traffic as PROTECT.
• SA lookup: the Security Association Database (SAD) provides the outbound Child SA keys, algorithms and Security Parameter Index (SPI).
• Encapsulation: the whole inner packet is encrypted, an ESP header and trailer plus an integrity check value (ICV) are added, and a new outer header from 203.0.113.10 to 198.51.100.20 is prepended.
• Transit: internet routers see only the outer header, carried as native ESP (IP protocol 50) or as ESP-in-UDP on port 4500 when NAT Traversal is active.
• Decapsulation: Gateway B finds the inbound SA by SPI, verifies the ICV and the anti-replay window, decrypts, checks its firewall policy, and forwards the original packet to 10.20.20.20.

Return path and routing symmetry

Server B replies from 10.20.20.20 to 10.10.10.11. That reply must reach Gateway B, either because Gateway B is the default gateway of Server B or because Server B has a specific route for 10.10.10.0/24 pointing to it.

If Server B uses a different internet router without that route, the reply leaves unencrypted toward a network where the private destination is not routable, and it never returns to Client A. The tunnel status still shows up while the application times out. A working VPN requires routing that is correct in both directions at both sites.

Control plane (IKE) and data plane (ESP)

The control plane is IKE, today IKEv2. It authenticates the peers, runs a Diffie-Hellman exchange, negotiates algorithms and traffic selectors, and creates, rekeys and deletes security associations. IKE uses UDP port 500, and UDP port 4500 once NAT is detected.

The data plane is ESP (Encapsulating Security Payload, IP protocol 50, RFC 4303), which provides confidentiality, data origin authentication, integrity and anti-replay for each packet. AH (Authentication Header, IP protocol 51) provides integrity without encryption, but it also protects fields of the outer IP header, so it breaks when NAT rewrites addresses. AH is rarely used in modern designs.

IKE SA, Child SA, SPD and SAD

Four data structures drive IPsec processing. Knowing which one is involved narrows troubleshooting quickly.

• IKE SA: the authenticated, encrypted control channel between the peers. It carries rekeying, liveness checks and Child SA negotiation.
• Child SA (IPsec SA): the keys and algorithms used for data. Child SAs are unidirectional and created in pairs, one inbound and one outbound, each with its own SPI.
• Security Policy Database (SPD): rules keyed on selectors such as addresses, protocol and ports, with the actions PROTECT, BYPASS or DISCARD.
• Security Association Database (SAD): the active SAs, indexed for inbound packets by SPI, holding keys, algorithms and replay counters.

Tunnel mode versus transport mode

Tunnel mode encrypts the entire original IP packet and adds a new outer IP header. It is the normal choice for Site-to-Site VPNs because internal addresses are hidden inside the encrypted payload and private address space can cross the internet.

Transport mode inserts the ESP header after the original IP header and protects only the payload, leaving the original addresses visible. It suits host-to-host protection and is commonly used underneath GRE over IPsec or L2TP/IPsec, where another protocol already supplies the tunnel header.

IKE_SA_INIT, IKE_AUTH and CREATE_CHILD_SA

In the normal case a tunnel is ready after two request and response exchanges, four messages in total. Extra round trips occur with cookie challenges, a rejected Diffie-Hellman group, EAP authentication or additional intermediate exchanges, so four messages is a minimum rather than a fixed rule.

• IKE_SA_INIT: the peers agree on algorithms (encryption, integrity, PRF, Diffie-Hellman group), exchange nonces and key-exchange values, detect NAT, and derive the keys that protect the IKE SA.
• IKE_AUTH: inside that protected channel, each side sends its identity (IDi and IDr), proves it with a certificate signature or pre-shared key, and negotiates traffic selectors for the first Child SA pair.
• CREATE_CHILD_SA: used later to add Child SAs or to rekey existing SAs without repeating authentication.

Pre-shared keys versus X.509 certificates

A pre-shared key (PSK) is the same secret configured on both peers. It is simple for a handful of tunnels, but reusing one key across branches makes individual revocation impossible. Weak keys are easier to guess or expose, so use a sufficiently long random secret and a distinct key for each peer pair.

Certificates give each gateway its own private key and a certificate issued by a trusted certificate authority. This scales better, avoids distributing shared secrets and allows one site to be revoked through a CRL or OCSP without touching the others. The cost is operating a PKI: issuance, renewal and revocation checking.

Algorithms: encryption, integrity, PRF and Diffie-Hellman

RFC 8247 (IKEv2) and RFC 8221 (ESP) list which algorithms implementations must, should and should not support. They are updated over time, so check the current versions and your compliance requirements rather than copying an old proposal list.

• Encryption: AEAD ciphers such as AES-GCM combine encryption and integrity in one transform and are the usual modern choice.
• Integrity: only needed as a separate transform with non-AEAD ciphers such as AES-CBC, for example HMAC-SHA2-256-128.
• PRF: derives key material from the Diffie-Hellman result and nonces, for example PRF_HMAC_SHA2_256.
• Diffie-Hellman groups: 2048-bit MODP (group 14) and elliptic-curve groups such as 19, 20 or Curve25519 (group 31) are common choices; small legacy groups such as 1, 2 and 5 should be disabled.

Rekeying, reauthentication and Perfect Forward Secrecy

Rekeying creates fresh keys and a new SA before the old one expires, so traffic continues without interruption. Reauthentication goes further and repeats peer authentication by building a new IKE SA, which re-validates credentials such as certificates.

Perfect Forward Secrecy (PFS) performs a new Diffie-Hellman exchange when Child SAs are created or rekeyed. This limits the damage of a later key compromise: traffic protected with keys derived from an earlier, independent exchange is not exposed simply because a later key or long-term credential leaks. It is a strong mitigation, not a guarantee against every attack.

What must match between peers and what is local policy

Some parameters must be compatible on both sides or negotiation fails: at least one common proposal, acceptable identities, valid credentials and trust chains, and traffic selectors that the responder accepts. Other settings are local policy: SA lifetimes can differ because the side with the shorter lifetime simply rekeys first, and liveness-check intervals and queuing are chosen per gateway.

OpenVPN: TLS control channel and data channel

OpenVPN uses a TLS session as its control channel for authentication and key exchange, and a separate data channel that encrypts tunnelled packets with keys derived from that session.

UDP transport is preferred. When OpenVPN runs over TCP and the underlay loses packets, both the inner TCP connections and the outer TCP connection retransmit and back off at the same time, a problem often called TCP meltdown, and throughput can collapse. Reserve TCP mode for networks that block UDP.

WireGuard: cryptokey routing

WireGuard, described in its whitepaper by Jason Donenfeld, uses a fixed set of modern primitives (Noise protocol framework, Curve25519, ChaCha20-Poly1305, BLAKE2s) with no algorithm negotiation. Each peer is identified by a public key and a list of AllowedIPs. When sending, the destination address selects the peer; when receiving, the decrypted source address must fall inside that peer’s AllowedIPs or the packet is dropped.

Because WireGuard has no built-in user directory, certificate authority or configuration push, enterprise deployments usually add a control plane for key distribution, single sign-on and device posture. A fixed algorithm set also means organisations with mandated algorithm lists should confirm that WireGuard satisfies them.

TLS VPNs and legacy protocols

"SSL VPN" is a product label, not a protocol. What matters is the TLS configuration underneath: allow only TLS 1.2 and TLS 1.3, following RFC 8996, which deprecates TLS 1.0 and 1.1, and the recommendations in RFC 9325.

Retire legacy protocols. PPTP relies on MS-CHAPv2 authentication, which is considered broken, and should not be deployed. L2TP on its own provides no encryption and should only be used as L2TP/IPsec, if at all.

Full Tunnel versus Split Tunnel

Full Tunnel replaces the client default route (0.0.0.0/0, and ::/0 for IPv6) so that all traffic, including internet browsing, goes through the corporate gateway. The benefit is central inspection, filtering, data loss prevention and logging. The cost is corporate bandwidth and extra latency to cloud services, because traffic detours through the gateway before reaching the internet.

Split Tunnel installs routes only for corporate prefixes, for example 10.0.0.0/8 or 172.16.0.0/12, and sends everything else directly through the local ISP. It saves bandwidth and gives users direct access to cloud services, but the device is connected to the corporate network and the internet at the same time with less central visibility. The NSA and CISA remote access VPN guidance treats split tunnelling as a risk to be justified and compensated with endpoint controls.

Policy-based VPN versus Route-based VPN

A Policy-based VPN (the classic crypto map model) matches traffic against selectors such as source subnet and destination subnet and encrypts whatever matches as it leaves the physical interface. It is simple for one or two fixed subnet pairs, but adding subnets means changing selectors on both peers, per-tunnel counters are limited, and dynamic routing protocols generally cannot run over it.

A Route-based VPN binds IPsec to a virtual tunnel interface (VTI) or GRE interface. Whatever the routing table sends to that interface is encrypted. This supports BGP or OSPF over the tunnel, redundant tunnels with automatic failover, multicast where the platform allows it, and zone-based firewall policy on the tunnel interface. Most multi-site and cloud designs use the route-based model.

Internal DNS, DNS leaks and dual-stack clients

Name resolution needs the same attention as routing, because a correct tunnel is useless if clients cannot find internal services or silently send queries elsewhere.

• Internal names: remote clients must receive corporate DNS servers or conditional forwarding so names such as app.corp.example resolve to internal addresses.
• DNS leaks: in a Full Tunnel design, clients that keep querying the home router or ISP resolver reveal which sites they visit. Operating system policy, for example the Windows Name Resolution Policy Table (NRPT), can force the right queries through the VPN.
• IPv6: if the tunnel carries only IPv4 while the local network offers IPv6, IPv6 traffic can bypass the tunnel entirely. Either carry IPv6 through the VPN or block it on the client while connected.

NAT Traversal (NAT-T) for ESP

Port address translation (PAT, also called NAPT) lets many private hosts share one public address by tracking TCP and UDP port numbers. Native ESP is its own IP protocol and has no ports, so a PAT device cannot build a reliable mapping for it and return traffic is lost.

IKEv2 detects NAT during IKE_SA_INIT using NAT detection payloads defined in RFC 7296. If NAT is present, IKE moves from UDP 500 to UDP port 4500 and ESP packets are wrapped in UDP port 4500 as defined in RFC 3948 (ESP-in-UDP). To the NAT device this looks like an ordinary UDP flow. Periodic NAT keepalives stop the mapping from expiring when the tunnel is idle.

NAT exemption for tunnel traffic

Between sites, hosts should normally keep their real private addresses, for example 10.10.10.0/24 at headquarters and 10.20.20.0/24 at the branch, so routing, logging and access policy stay consistent.

Perimeter firewalls usually apply source NAT to everything heading to the internet. Unless a NAT exemption (no-NAT) rule for site-to-site traffic is evaluated before that general rule, packets are translated to the public address, no longer match the tunnel selectors or policy, and never reach the tunnel correctly.

Public addresses, double NAT and CGNAT

At least one peer must be reachable: a static public address, or a dynamic DNS name that tracks it, acting as responder. The other peer can sit behind NAT and act as initiator, keeping the mapping open with NAT keepalives.

If both sites sit behind carrier-grade NAT (CGNAT, typically 100.64.0.0/10 from RFC 6598), neither can accept inbound connections. The options are a public address from the ISP, a relay or hub in the cloud that both sites connect out to, or an overlay product that provides relaying.

Two layers of firewall policy

Protect the outside interface by allowing only what the VPN needs: UDP 500, UDP 4500 and, if NAT-T is not always used, IP protocol 50, restricted to known peer addresses where possible. Then treat traffic that comes out of the tunnel as untrusted too: inspect it with an internal policy on the tunnel interface or zone and allow only the destinations and ports each site or user group needs.

Point-to-point, Hub-and-Spoke and Full Mesh

Each topology trades path efficiency against the number of tunnels to build and maintain.

• Point-to-point: one tunnel between two sites, for example a data center replication link.
• Hub-and-Spoke: every branch connects only to a central hub. Policy is managed in one place, but branch-to-branch traffic must pass through the hub (hairpinning), which adds an extra leg of latency and makes the hub a critical dependency that needs redundancy.
• Full Mesh: every site has a direct tunnel to every other site, which gives the shortest paths. The number of tunnels grows as N(N-1)/2, quadratically with the number of sites, so large networks use dynamic approaches such as DMVPN or SD-WAN that build direct tunnels on demand.

Scenario 1: Site-to-Site branch interconnect

Requirement: connect headquarters (10.10.0.0/16) with two branches (10.20.0.0/16 and 10.30.0.0/16) for ERP and database traffic. Design: route-based IPsec tunnels with IKEv2, AEAD encryption such as AES-256-GCM and an elliptic-curve Diffie-Hellman group, certificate authentication, and BGP over the tunnel interfaces so prefixes are advertised automatically and traffic reroutes when a link fails.

Scenario 2: Remote Access for a mobile workforce

Requirement: staff on hotel or public Wi-Fi need the ERP system and internal file servers. Design: IKEv2 with EAP, or a managed WireGuard or OpenVPN service, integrated with the corporate identity provider; mandatory multi-factor authentication; a dedicated client address pool such as 172.16.100.0/24; device posture checks; and firewall rules that grant each group only the applications it needs.

Scenario 3: Hybrid cloud interconnect

Requirement: resilient connectivity between an on-premises data center and cloud VPCs or virtual networks. Design: two IPsec tunnels to the cloud provider’s VPN gateway endpoints, ideally from two on-premises devices or ISPs, with BGP on both tunnels. Use provider-supported BGP attributes such as AS-path prepending or MED to prefer one path, and follow the provider documentation for supported algorithms and limits.

Scenario 4: Overlapping subnets with twice NAT

Requirement: headquarters uses 10.10.10.0/24 and a newly connected partner uses the same range, so neither side can route to the other. The permanent fix is to renumber one side. When that is not possible yet, each site is presented across the tunnel as a unique virtual subnet: headquarters as 10.110.10.0/24 and the partner as 10.210.10.0/24, with the gateways translating source and destination (twice NAT).

• Client A (10.10.10.11) sends to the partner’s virtual address 10.210.10.20.
• Gateway A translates the source to 10.110.10.11 and sends the packet into the tunnel.
• Gateway C translates the destination to the real server 10.10.10.20 and delivers it.
• Server C replies to 10.110.10.11; Gateway C translates the source to 10.210.10.20 and sends the reply into the tunnel.
• Gateway A translates the destination back to 10.10.10.11 and delivers the reply to Client A.

MTU, Path MTU Discovery and TCP MSS clamping

Ethernet paths usually carry a 1,500-byte MTU. Encapsulation makes a full-size inner packet larger than that. If the packet has the Don’t Fragment (DF) bit set, a router on the path drops it and should send back an ICMP Fragmentation Needed message (IPv4 type 3, code 4). When a firewall blocks that ICMP message, Path MTU Discovery fails and a black hole appears: small packets, pings and the TCP handshake work, but large transfers and TLS handshakes with big certificates hang.

Adding the example overhead below to a full inner packet shows why. Overhead depends on the cipher, NAT-T, IP version and any additional encapsulation such as PPPoE or GRE, so treat these figures as one worked example rather than a universal constant.

• Outer IPv4 header: 20 bytes; UDP header for NAT-T: 8 bytes.
• ESP header (SPI and sequence number): 8 bytes; AES-GCM initialization vector: 8 bytes.
• ESP trailer (pad length and next header): 2 bytes plus 0 to 3 bytes of padding; AES-GCM ICV: 16 bytes.
• Total in this example: 62 to 65 bytes, so the largest inner packet on a 1,500-byte underlay is about 1,435 bytes and the arithmetic TCP MSS is about 1,395 bytes (inner packet minus 40 bytes of IPv4 and TCP headers).
• In practice many teams clamp the MSS lower, commonly around 1,350 to 1,380 bytes, to leave headroom for PPPoE (8 bytes), GRE, an IPv6 outer header, CBC ciphers or TCP options. Measure the real path MTU and confirm your vendor guidance.

Throughput and cryptographic offload

Firewall datasheet throughput is usually measured without encryption. IPsec throughput is lower and depends on packet size, cipher and whether the platform uses hardware acceleration such as AES-NI instructions or dedicated crypto engines. Size gateways on the vendor’s IPsec figures at realistic packet sizes, including small voice packets, not on plain firewall throughput.

Latency, loss and tunnel failover

A tunnel cannot be better than its underlay. Latency lowers TCP throughput for a given window size, and packet loss causes TCP retransmissions and audible gaps in real-time media.

Liveness detection lets a gateway notice a dead peer. IKEv2 includes liveness checks natively (RFC 7296); Dead Peer Detection in RFC 3706 is the equivalent extension for IKEv1. When a peer stops responding, the gateway removes the stale SAs, and with route-based tunnels a dynamic routing protocol or tracked static routes can move traffic to a backup tunnel or second ISP.

What a VPN does not do

A VPN protects traffic between its endpoints. It does not protect a device that is already compromised: malware on a laptop sees data before it is encrypted and can use the tunnel like the legitimate user. It does not provide anonymity: the VPN operator, and the services you reach, can still observe and log activity. It does not replace application security, patching or backup.

Least privilege, Zero Trust and ZTNA

NIST SP 800-207 describes Zero Trust architecture: no implicit trust based on network location, and access decisions made per request using identity, device state and context. A VPN can be one component of such an architecture, but a VPN by itself is not Zero Trust.

In practice this means microsegmentation, per-group firewall rules that allow only specific applications and ports, and for many remote users a ZTNA (Zero Trust Network Access) service that brokers access to individual applications instead of placing the device on the internal network.

Credential lifecycle and revocation

Keys and certificates need governance just like passwords.

• Store gateway private keys in non-exportable storage or a hardware security module (HSM) where the platform supports it.
• Track certificate expiry and automate renewal so tunnels do not fail unexpectedly.
• Check revocation through CRL or OCSP so that departing staff and lost or compromised devices lose access promptly.

Gateway hardening, endpoint posture and logging

Internet-facing VPN gateways are frequent targets, and published vulnerabilities in them are exploited quickly. The NSA and CISA guidance on selecting and hardening remote access VPN solutions (September 2021) and NIST SP 800-77 Rev. 1 recommend the following controls.

• Choose supported products, apply security updates promptly and reduce the exposed management surface.
• Require multi-factor authentication for remote users and check device posture where possible: endpoint protection, disk encryption and operating system patch level.
• Send connection, authentication and traffic-volume events to a central SIEM and alert on anomalies such as impossible travel or repeated failures.

Multi-site PBX interconnect

Linking PBXs at headquarters and branches over VPN allows extension dialling between sites and central trunking. For example, a headquarters PBX at 10.10.10.50 can form a SIP trunk with a branch PBX at 10.20.20.50 across the IPsec tunnel between the site gateways, so inter-site calls stay on the private network.

SIP signalling and RTP media are separate flows

SIP (RFC 3261) sets up and tears down calls, usually on UDP or TCP port 5060. RTP (RFC 3550) then carries the audio on dynamic UDP ports negotiated in SDP. The port range is set by each PBX or phone; many systems default to a range such as 10000 to 20000.

The classic fault is one-way audio or no audio: the call rings and connects but nobody hears anything. Common causes are firewall or tunnel policy that allows SIP but not the RTP port range, SDP bodies that advertise addresses the other side cannot reach, or a SIP ALG on the gateway rewriting packets incorrectly.

Voice quality: QoS, DSCP and the path

Voice sends small packets at fixed intervals, for example every 20 ms for G.711. ITU-T G.114 recommends keeping one-way delay under about 150 ms for good interactivity; jitter below roughly 30 ms and packet loss below about 1 percent are common engineering targets rather than formal limits.

Mark RTP with DSCP EF (Expedited Forwarding, value 46) and make sure the gateway copies the inner DSCP value to the outer tunnel header, then prioritise it in the egress queue. QoS only works on links you control or where a provider honours markings; the public internet generally ignores them. Priority queuing after encryption can also reorder packets, so confirm the anti-replay window is large enough on busy links.