IPsec uses the 32 bit sequence number in ESP/ AH protocol packets.
This counter should be incremented for each packet while transmitting out.
When anti-replay check is enabled, this reuse of sequence number is not allowed, new SA should be established before it reaches the max value.
Anti-replay check is performed at inbound processing that is when an IPsec applied packet is received.
The receiver must check if the received packet has a duplicate sequence number. This duplicate check is performed using the sliding window mechanism. This sliding window will have the window size and the max received sequence number. This max received sequence number gets updated whenever a valid packet is received with a higher value sequence number.
Packets that contain a sequence number that is lesser than the max valid sequence number — window size, are discarded.
Packets that contain a sequence number that is in between the max valid sequence number (MSN) and MSN - window size are checked for duplicates.
If the packet is received with a sequence number that is higher than max valid sequence number, based on integrity and encryption verification, the sliding window is updated with MSN as the received sequence number, window is also updated.
For implementation purposes, this sliding window can be assumed as a bit map where each bit indicates one sequence number. This can be updated using bit operations and bit shift operations while updating the max sequence number.
Multi senders:
As per IPSec RFCs same SA can be used by multiple senders to encrypt IP packets. In this case, the receiver has to maintain multiple sliding windows for each sender.
Our patent uses this multi-sender approach. This patent explains how to handle anti-replay checks for multiple senders. This involves enhancing IKEv2 protocol and ESP trailer. I and Srini Addepalli together worked on this patent which is useful for virtualized environments using IP security.
