Network Protocols for Troubleshooting, Tunneling, and Security

Computer Networking for Beginners series — Part 8. This post continues Part 1: A Complete Guide to How Networks Work, Part 2: What Is the OSI Model?, Part 3: Networking Devices Explained, Part 4: CDNs, VPNs, QoS, and TTL, Part 5: Cloud Networking Explained, Part 6: Internet Protocol Explained, and Part 7: What Are the Well-Known Port Numbers?. Each post in the series takes one topic and covers it in depth. This one covers three protocols that keep networks diagnosable, connected, and secure: ICMP, GRE, and IPsec.

In Part 6, you learned how IP carries data in packets, and how TCP and UDP deliver it to the right application. But IP carries more than TCP and UDP. Riding alongside them are protocols that do very different jobs:

• ICMP reports problems and tests whether devices are reachable — it’s what makes ping and traceroute work.
• GRE builds tunnels, wrapping one packet inside another so it can travel across a network it otherwise couldn’t.
• IPsec secures traffic with encryption and authentication, and it’s one of the most common ways VPNs protect data crossing the internet.

Together, they cover three needs every network has: finding out what’s wrong, connecting networks together, and keeping the traffic between them private.

Not Every Protocol Uses a Port

Before diving in, there’s one important detail that sets these protocols apart. In Part 7, every service had a port number — HTTPS on TCP port 443, DNS on UDP port 53, and so on. Ports belong to TCP and UDP.

ICMP, GRE, and IPsec’s ESP and AH protocols don’t use TCP or UDP at all, so they don’t have port numbers. Instead, they’re identified by an IP protocol number in the IP header, which tells the receiving device what kind of data the packet carries:

This matters when you configure a firewall: to allow a GRE tunnel or an IPsec VPN, you have to allow those protocols, not just open a port. (IPsec’s key exchange is the exception — it runs over UDP, as you’ll see below.)

ICMP: The Network’s Messenger

The Internet Control Message Protocol (ICMP) is how devices report problems and check whether other devices are alive. ICMP doesn’t carry your data; it carries messages about your data — “this host is unreachable,” “this packet’s time ran out,” “yes, I’m here.”

Every IP device uses ICMP, and you’ve already seen it in action in this series:

• In Part 1, ping tested whether example.com was reachable.
• In Part 4, a packet with a TTL of 3 came back with “Time to live exceeded,” and traceroute used the same message to map the path to a website.

Common ICMP Messages

Each ICMP message has a type, and many have a code that adds detail:

Using ICMP to Troubleshoot

When ping gets replies, you know the device is alive and reachable, and the times tell you how fast the connection is:

64 bytes from 104.20.23.154: icmp_seq=0 ttl=54 time=23.882 ms

When nothing comes back, ping reports a timeout for each request. Here’s what pinging an address where no device answers looks like:

ping -c 3 192.0.2.1

PING 192.0.2.1 (192.0.2.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

--- 192.0.2.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

A timeout doesn’t always mean a device is down, though. Many devices and firewalls block ICMP on purpose, so they simply don’t answer pings. If a router along the way knows a destination can’t be reached, it may instead send back a Destination Unreachable message, which ping shows as something like “Destination Host Unreachable” — a more specific clue about where the problem is.

Should You Block ICMP?

Because ICMP can be used to discover which devices exist on a network, some administrators block it entirely. That’s usually a mistake. Blocking all ICMP can break important network functions — for example, the messages that help devices find the right packet size for a connection — and it makes troubleshooting much harder. A better approach is to allow the essential ICMP messages and limit things like replying to pings from the internet. In IPv6 networks, ICMP (called ICMPv6) is even more essential, because IPv6 uses it to find neighboring devices and routers.

GRE: Building Tunnels

Generic Routing Encapsulation (GRE) creates a tunnel between two endpoints, usually two routers. A tunnel lets two networks talk as if they were directly connected, even when there are many other networks in between.

How GRE Works

GRE uses encapsulation, the same idea you’ve seen throughout this series: it takes a complete packet and wraps it inside a new one.

Original packet:          [ IP header (private addresses) | Data ]

GRE-encapsulated packet:  [ New IP header (public addresses) | GRE header | IP header (private) | Data ]

1. A router at one end takes a packet from its local network — complete with its private IP addresses.
2. It adds a small GRE header, then a new IP header addressed to the router at the other end of the tunnel.
3. The packet crosses the internet like any other packet. The routers in between only see the outer header.
4. The router at the far end removes the outer header and the GRE header, and forwards the original packet into its own network as if it had arrived directly.

The “generic” in the name is important: GRE can carry almost any kind of traffic, including multicast traffic and the routing protocols routers use to share routes with each other. That’s why it’s often used to connect sites so they can exchange routing information.

GRE’s Big Limitation: No Encryption

GRE doesn’t encrypt anything. It wraps packets in a tunnel, but anyone who captures the traffic on the internet can unwrap the GRE header and read the original packet inside. GRE provides the tunnel, but not the privacy.

That’s why GRE is rarely used alone across the internet. Instead, it’s commonly combined with a security protocol — most often GRE over IPsec: GRE builds the tunnel and carries any kind of traffic, and IPsec encrypts the whole thing.

IPsec: Securing Traffic

IPsec (Internet Protocol Security) is a suite of protocols that secures IP traffic. It’s one of the most widely used technologies for VPNs, especially site-to-site VPNs between offices and the cloud (see Part 4 and Part 5).

IPsec protects data in four ways:

• Confidentiality — data is encrypted, so anyone who intercepts it sees only scrambled data.
• Integrity — each packet includes a cryptographic check (a hash), so if anyone changes even one bit along the way, the receiver knows and discards it.
• Authentication — both ends prove who they are (with a pre-shared key or certificates), so you know you’re connected to the real VPN gateway and not an impostor.
• Anti-replay — every packet carries a sequence number, so an attacker can’t capture valid packets and send them again later to repeat an action.

AH and ESP: IPsec’s Two Main Protocols

IPsec uses two protocols to protect the data itself:

• AH (Authentication Header) provides integrity and authentication: it proves the packet is genuine and hasn’t been changed — including much of the IP header. But it does not encrypt anything, so the data is still readable. Because AH protects the IP addresses in the header, it also doesn’t work through NAT, which changes those addresses.
• ESP (Encapsulating Security Payload) provides encryption for confidentiality and integrity and authentication. Because it does everything most VPNs need — and works with NAT — ESP is what almost every IPsec VPN uses today.

Setting Up the Tunnel: IKE Phase 1 and Phase 2

Before two devices can encrypt traffic, they have to agree on how: which encryption and integrity algorithms to use, and which secret keys. Sending keys across the internet in the open would defeat the purpose, so IPsec uses the Internet Key Exchange (IKE) protocol to negotiate everything securely. IKE runs over UDP port 500, and when there’s NAT between the two endpoints, IPsec switches to UDP port 4500 (called NAT traversal).

IKE works in two phases:

Phase 1: Build a secure channel for negotiating.

1. The two VPN gateways agree on the security settings for the negotiation itself, using a framework called ISAKMP (Internet Security Association and Key Management Protocol).
2. They use a Diffie-Hellman key exchange — a clever bit of math that lets two devices create the same shared secret key without ever sending it across the network.
3. They authenticate each other with a pre-shared key or certificates.

The result is a secure, encrypted management channel between the two gateways — the IKE security association (SA). No user data travels through it yet.

Phase 2: Negotiate the tunnel for the data.

Using the secure channel from Phase 1, the gateways negotiate the IPsec security associations that will actually protect the user traffic: which protocol (ESP or AH), which encryption and integrity algorithms, and fresh keys. An SA only works in one direction, so they create a pair — one for each direction. These SAs are renewed regularly with new keys, so even if one key were ever compromised, it would only expose a small amount of traffic.

Once Phase 2 is complete, data starts flowing through the encrypted tunnel. (The newer IKEv2 simplifies the exchange into fewer messages, but the same two-step idea remains: first secure the negotiation, then set up the data tunnel.)

Transport Mode vs. Tunnel Mode

IPsec can protect packets in two ways, depending on how much of each packet it wraps.

Transport mode protects only the data (the payload). The original IP header stays in place, so the real source and destination addresses are still visible:

Original packet:   [ IP header | Data ]
Transport mode:    [ IP header | ESP header | Data (encrypted) | ESP trailer ]

Transport mode is used when the two endpoints themselves are the ones communicating — for example, securing traffic directly between two servers. It’s also commonly used to encrypt a GRE tunnel, because GRE already provides the outer header.

Tunnel mode encrypts the entire original packet — header and all — and puts it inside a new packet with a new IP header:

Original packet:   [ IP header | Data ]
Tunnel mode:       [ New IP header | ESP header | IP header + Data (all encrypted) | ESP trailer ]

The new header only shows the addresses of the two VPN gateways. Anyone watching the internet can’t see the original private addresses, which devices are talking, or what they’re saying. That makes tunnel mode the more secure option, and it’s the standard for site-to-site VPNs, where gateways protect traffic on behalf of whole networks.

Putting It All Together: A Branch Office VPN

Here’s how all three protocols work together when a branch office connects securely to headquarters:

1. IKE Phase 1: the two offices’ firewalls contact each other on UDP port 500, use Diffie-Hellman to agree on keys, and authenticate each other, creating a secure management channel.
2. IKE Phase 2: through that channel, they negotiate a pair of ESP security associations in tunnel mode.
3. Traffic flows: when an employee at the branch opens an internal web application at headquarters, the branch firewall encrypts the entire packet with ESP, wraps it in a new IP header addressed to the headquarters firewall, and sends it across the internet. Headquarters decrypts it, checks its integrity and sequence number, and forwards the original packet to the web server.
4. GRE, if needed: if the offices also need to share routing information or multicast traffic, they can run a GRE tunnel between the firewalls and protect it with IPsec — GRE over IPsec.
5. ICMP for troubleshooting: when something stops working, the network team uses ping and traceroute — both built on ICMP — to check whether each firewall can reach the other, and where along the path packets are being lost.

Quick Summary

What’s Next in This Series

You now know the protocols that make networks diagnosable, connected, and secure. Upcoming posts in the Computer Networking for Beginners series will keep building on these foundations. In the meantime, try running ping and traceroute against a few websites and notice which ones answer ICMP and which ones don’t. And if you’d like a modern, easy-to-set-up alternative to IPsec for your own devices, look at WireGuard, the protocol behind tools like Tailscale.

Conclusion

ICMP is the network’s messenger: it carries the echo requests and replies behind ping, and error messages like “destination unreachable” and “time exceeded” that make troubleshooting possible. GRE builds tunnels that can carry almost any traffic between two endpoints — but it doesn’t encrypt anything. IPsec adds the security: confidentiality, integrity, authentication, and anti-replay protection. It uses ESP for encryption (or the rarely used AH for integrity alone), sets up its tunnels with IKE in two phases — first a secure channel, then the security associations for the data — and protects packets in either transport mode, encrypting just the payload, or the more secure tunnel mode, encrypting the entire original packet.

Unlike TCP and UDP services, these protocols are identified by IP protocol numbers rather than ports, so remember to allow them by protocol when you configure a firewall for a tunnel or VPN.

Computer Networking for Beginners series: Part 1: A Complete Guide to How Networks Work · Part 2: What Is the OSI Model? · Part 3: Networking Devices Explained · Part 4: CDNs, VPNs, QoS, and TTL · Part 5: Cloud Networking Explained · Part 6: Internet Protocol Explained · Part 7: Well-Known Port Numbers · Part 8: Protocols for Troubleshooting, Tunneling, and Security