IPsec is designed to provide security functions for authentication and encryption for TCP/IP at the Internet level. To better understand how IPsec works, you first have to understand the IPsec header and where it is located. When looking at IPv4, the header contains addressing and control fields, while the payload carries the actual data to be sent over the internet. A standard IP header is 20 bytes long. The complete IP header can be seen in the illustration below.
To break down the IP header, we start at the top with the version (ver). This identifies the version of the IP used, which is IPv4 for this example. It is used to ensure compatibility between devices running on the version of IP. It is 4 bits long and is followed by the Internet Header Length (hlen). The hlen specifies the length of the IP header, in 32-bit words. This includes the length of any options fields and padding.
Next, there is the type of service (TOS) block. This field is designed to carry information to provide quality of service features, such as prioritized delivery, for IP datagrams. The packet length (pkt len) specifies the total length of IP datagrams in bytes. After the pkt len, there is the ID field. This field contains a 16-bit value that is common to each of the fragments belonging to a particular message.
Continuing on to the next block, you have the flags (flgs). The flags are used to manage fragmentation. It is followed by the fragment offset. This field specifies the offset, or position, in the overall message where the data in this fragment goes. This is followed by the time to live (TTL) block. This specifies how long the datagram is allowed to “live” on the network, in terms of router hops.
After the TTL, there is the protocol (proto) field. This identifies the protocol used for transport and encapsulation. The next field is the header checksum which detects errors in the transport to ensure the message is not corrupted. This is followed by the source IP, destination IP address, and IP options to complete the full IP header.
When using IPsec, it is easier to understand how the IP header is used for authentication when in transport mode. It can be used for either authentication or encryption but the IP heard being transported is not an encryption protocol but is a secure IP connection. As illustrated in the diagram below, when used for authentication and authentication header (AH) is added to the IPv4 IP header for transport.
When using ESP in transport mode you are adding an encapsulating Security Payload (ESP) to support encryption and optional authentication. This encrypted payload is constructed by encapsulating the datagrams payload by adding security parameters Index and Sequence number fields on one side of the payload and ESP trailer consisting of padding, the next header, and optional authentication data at the end. This can be seen in the illustration below.
Friedl, S. (2005). AN Illustrated guide to IPsec. Retrieved from http://www.unixwiz.net/techtips/iguide-ipsec.html#ip