Host Networking: Protocol Implementation

• The scores for milestone 2 were not fully ready yet.
• CI scores were expected to appear on CMS soon.
• Design document scores were expected to follow within one or two days.

• Milestone 4 was explicitly made extra credit.
• The lecturer recommended doing it only if one enjoys the milestones.
• If the goal is to recover lost points, preparing for the final exam is probably a better use of time.

• This lecture ended half an hour early.
• After the lecture, students had the opportunity to review their exams.
• The next lecture would continue with virtualization.

This was covered in the previous host-networking lectures.

Main question:

• How does the application communicate with the network?
• What abstractions does the OS expose to applications?

Typical abstraction:

• sockets;
• read / write or send / recv;
• byte streams for TCP;
• datagrams for UDP.

This lecture briefly reviews the protocol side.

Main question:

• How do we translate the application-level abstraction into network packets?
• How do we communicate reliably and efficiently over an unreliable network?

The lecturer emphasized that this is not a full networking lecture. The goal is only to understand what the OS network stack must implement.

This is the main OS-relevant part.

Main question:

• How does the OS implement the plumbing between applications and the NIC?
• What is done in applications, libraries, the OS, drivers, and hardware?

At a high level:

The OS network stack translates between application-facing socket abstractions and packet-facing network hardware.

Real OS networking can be more complicated because of:

• container networking;
• virtualized networking;
• tunneling;
• encapsulation.

For example, a packet may contain:

• an outer Ethernet/IP layer;
• an inner Ethernet/IP layer;
• some tunneled payload.

This does not fundamentally change the basic OS problem: the OS still has to parse, construct, queue, and deliver packets.

Ethernet is mainly for communication within one local network.

Examples:

• machines connected to one switch;
• machines connected through a small number of switches;
• a local LAN.

Ethernet provides local link-layer packet delivery.

Ethernet uses MAC addresses.

Properties:

• typically 48 bits;
• usually written as six hexadecimal byte pairs;
• often called “physical addresses”;
• usually assigned by hardware.

For virtual machines, MAC addresses may be generated or configured in software. But from the OS network stack’s point of view, they are usually treated as externally given.

A simplified Ethernet frame contains:

• destination MAC address;
• source MAC address;
• EtherType;
• payload;
• CRC checksum.

The EtherType tells the receiver what the payload contains, for example:

• IPv4;
• IPv6;
• ARP.

Ethernet works with bounded-size frames.

Typical sizes:

• minimum Ethernet frame size: about 64 bytes;
• common maximum payload / MTU: about 1500 bytes;
• jumbo frames in specialized networks may allow larger packets, for example around 9 KB.

The important mental model:

Ethernet lets the OS send and receive individual bounded-size packets.

Ethernet provides very weak guarantees.

It does not guarantee:

• delivery;
• ordering;
• reliability;
• retransmission.

A packet may arrive, or it may be lost.

Ethernet frames include a CRC checksum, but this only checks whether an individual frame was corrupted. Higher-level protocols usually do their own checks as well.

IP adds communication across networks.

Ethernet is about one local network. IP is about sending packets across potentially many networks.

The main addition is routing.

IP introduces IP addresses.

Examples:

• IPv4 addresses;
• IPv6 addresses.

Unlike MAC addresses, IP addresses are configured.

They may be configured:

• statically;
• dynamically, for example through DHCP.

IP addresses may be:

• public and globally routable;
• private/internal and only routable within a local network.

Conceptually, IP itself does not care whether an address is public or private. That depends on routing configuration.

At the IP layer, the host needs routing information.

A routing table maps:

destination IP address or address range
    -> next hop

The next hop may be:

• the destination host itself, if it is on the same local network;
• a router, if the destination is outside the local network.

A common simple case is a default gateway.

The default gateway is used when no more specific route exists.

Example:

• in a home network, the default gateway is usually the home router;
• packets to unknown external destinations are sent to that router;
• the router forwards them toward the ISP and beyond.

IP still does not provide reliable delivery.

It does not guarantee:

• delivery;
• ordering;
• retransmission;
• duplicate suppression.

It mainly provides:

• addressing across networks;
• routing between networks.

IP can technically fragment packets.

This means:

• a large IP packet may be split into smaller fragments;
• fragments may be sent over a network with a smaller MTU;
• the receiver must reassemble them.

However, the lecturer emphasized that modern networks try to avoid IP fragmentation when possible.

Reasons:

• it complicates reassembly;
• it is expensive to implement;
• it is messy because fragmentation happens at the IP-packet level rather than at the connection level;
• it can lead to inefficient behavior.

Practical mental model:

Applications and transport protocols should usually try to send packets that fit within the network MTU.

UDP is a thin transport-layer protocol on top of IP.

Its main job is multiplexing.

It lets multiple applications on the same host use the network at the same time.

UDP introduces:

• source port;
• destination port.

Ports identify application endpoints on a host.

Thus, IP identifies the machine, while UDP ports help identify the application or socket on that machine.

A UDP header contains:

• source port;
• destination port;
• UDP length;
• UDP checksum.

Then comes the data payload.

UDP provides almost no reliability guarantees.

It does not guarantee:

• delivery;
• ordering;
• retransmission;
• duplicate suppression.

It may include a checksum, depending on the version and implementation details, but the checksum only checks packet integrity. It does not ensure that the packet arrives.

UDP does not turn the network into a byte stream.

It preserves a datagram/message abstraction.

In practice, a UDP message is constrained by packet size.

If the UDP payload is too large, IP fragmentation may happen, but this is usually undesirable.

Practical mental model:

With UDP, you typically send one message that fits into one packet.

TCP exposes a reliable byte-stream abstraction.

Instead of sending independent bounded-size messages, an application writes bytes into a stream.

The other side reads bytes out of the stream.

Important properties:

• the abstraction is a byte stream;
• there are no preserved message boundaries;
• data is delivered in order;
• data is retransmitted if necessary;
• corrupted data should not be delivered as valid data.

A TCP connection is bidirectional.

After connection setup, there is no permanent “sender” and “receiver” role.

Instead:

• both peers can send data;
• both peers can receive data;
• conceptually, there are two independent byte streams, one in each direction.

Important TCP header fields include:

• source port;
• destination port;
• sequence number;
• acknowledgement number;
• flags such as SYN and ACK;
• window size;
• checksum;
• options;
• optional data payload.

The source and destination ports are used for multiplexing, similar to UDP.

The more TCP-specific mechanisms rely on:

• sequence numbers;
• acknowledgement numbers;
• retransmissions;
• flow control;
• congestion control.

TCP connections are established using a three-way handshake.

The basic pattern is:

Client -> Server: SYN, seq = x
Server -> Client: SYN-ACK, ack = x + 1, seq = y
Client -> Server: ACK, ack = y + 1, seq = x + 1

After this, data transmission can begin.

The handshake establishes shared connection state.

Most importantly, it agrees on the initial sequence numbers for both directions.

Modern TCP implementations do not simply start at sequence number 0. They choose relatively random initial sequence numbers.

Reason:

• predictable sequence numbers are easier to manipulate or spoof;
• randomization makes some attacks harder.

The handshake may also negotiate optional TCP features.

Examples of optional features include various TCP options, such as selective acknowledgements.

A key point:

TCP sequence numbers are not packet numbers. They are byte-stream offsets.

If a TCP segment carries data, its sequence number identifies the offset of the first byte in that segment.

The length of the segment determines the byte interval carried by that packet.

For example, if a segment has:

seq = 1000
length = 500

then it carries the byte interval:

[1000, 1500)

TCP sends slices of the byte stream as packets.

The sender must track:

• which bytes have been written by the application;
• which bytes have been transmitted;
• which bytes have been acknowledged;
• which bytes may need retransmission.

The receiver must track:

• which byte intervals have arrived;
• which bytes can be delivered in order to the application;
• which bytes are duplicates;
• which bytes are missing.

The underlying network gives almost no guarantees.

Therefore, TCP achieves reliability by:

• sending data with sequence numbers;
• receiving acknowledgements;
• retransmitting data that appears to be lost;
• ignoring duplicate data at the receiver.

The sender can only know that data arrived if the receiver explicitly acknowledges it.

After TCP sends data, the sender cannot immediately discard it.

Reason:

• the packet may be lost;
• the acknowledgement may be lost;
• the sender may need to retransmit the data.

Therefore, the OS must keep unacknowledged data in a send buffer.

TCP transmissions must be idempotent.

This means:

• the receiver must handle duplicate transmissions correctly;
• receiving the same byte interval twice must not duplicate data in the application-visible byte stream.

For example:

• the sender sends bytes [0, 2);
• the receiver receives them;
• the ACK is lost;
• the sender retransmits bytes [0, 18) later.

The receiver must notice that bytes [0, 2) were already received and must not deliver them twice.

When retransmitting, TCP does not have to resend exactly the same packet boundaries.

Because TCP is a byte-stream protocol, not a message protocol, retransmission can use different segment boundaries.

Example:

• the sender first sends 2 bytes;
• before retransmission, the application writes 16 more bytes;
• the sender may retransmit all 18 bytes in one packet.

This is valid because each TCP segment is annotated with sequence numbers.

TCP acknowledgements are cumulative.

An acknowledgement number means:

I have received everything in order up to this byte offset.

It does not merely mean:

I received this one particular packet.

Example:

ACK = 5000

means:

All bytes before offset 5000 have been received in order.

Suppose the sender sends two packets.

If the receiver gets both packets and sends two acknowledgements, but the first ACK is lost, the second ACK can still acknowledge all earlier bytes.

Thus, losing an ACK does not necessarily force retransmission.

If packet 2 arrives but packet 1 is missing, the receiver cannot advance the cumulative ACK beyond packet 1.

The receiver may either:

• buffer packet 2 temporarily;
• drop packet 2 if it lacks memory or chooses not to buffer it.

The protocol still works because the sender will eventually retransmit missing data.

TCP can optionally use selective acknowledgements, often called SACK.

SACK lets the receiver say, roughly:

I am still missing earlier data, but I have already received this later interval.

This is an optimization.

It can reduce unnecessary retransmission, especially on lossy networks.

But it is not required for correctness. Basic cumulative ACKs plus retransmission are enough for correctness.

The most fundamental loss-detection mechanism is a timeout.

The sender records when it sent data.

If no suitable acknowledgement arrives after some time, the sender assumes loss and retransmits.

This is necessary because the network usually does not explicitly report packet loss.

Choosing the timeout is a performance issue.

If the timeout is too short:

• the sender retransmits unnecessarily;
• CPU cycles are wasted;
• network bandwidth is wasted.

If the timeout is too long:

• recovery from real loss is slow;
• latency increases.

TCP implementations dynamically estimate timing, often using observed round-trip time.

For example:

• if a SYN takes several seconds to receive a SYN-ACK, a one-millisecond timeout would be unreasonable.

TCP also uses duplicate acknowledgements as a faster loss signal.

Suppose the sender transmits many packets and the first one is lost.

The receiver gets later packets, but cannot advance the cumulative ACK. So it keeps sending acknowledgements with the same ACK number.

The sender observes repeated ACKs with the same acknowledgement number.

This suggests:

• later packets arrived;
• an earlier byte interval is missing;
• retransmission should happen before waiting for a timeout.

This is called fast retransmit.

Fast retransmit is an optimization. Timeouts are still the fundamental correctness mechanism.

Fast retransmit only works if later packets arrive and produce duplicate ACKs.

If the last packet in a burst is lost, there may be no later packets to trigger duplicate ACKs.

Then the sender must rely on a timeout.

Flow control protects the receiver.

It prevents the sender from sending more unacknowledged data than the receiver is willing to buffer.

The issue is not only CPU speed. It is especially about memory/buffer capacity.

TCP uses the window size field for flow control.

The receiver advertises how much more data it is currently willing to receive.

For example:

window = 16 KB

means:

Do not have more than 16 KB of unacknowledged data outstanding for this connection.

As acknowledgements advance, the window slides forward.

The advertised receive window is not an absolute guarantee that all future packets will be accepted.

Packets may still be lost for other reasons.

Also, a robust receiver must handle misbehaving senders.

If the sender sends more than the receiver can handle, the receiver can drop packets. TCP correctness is preserved because lost packets can be retransmitted.

Congestion control protects the network.

The problem is not only the receiver. Routers, switches, and links inside the network can become overloaded.

If senders transmit too fast:

• buffers in the network fill up;
• queues grow;
• latency increases;
• packets may be dropped.

The network often gives very little explicit feedback.

The sender may only observe symptoms such as:

• packet loss;
• increasing delay;
• duplicate ACKs;
• timeouts.

Based on these signals, TCP estimates how fast it should send.

Congestion control is typically managed per TCP connection.

Reason:

• different connections may take different paths through the network;
• bottlenecks may be different for different destinations;
• the sender usually does not know the full network topology.

A TCP sender generally tries to:

• send faster when the network appears healthy;
• slow down when loss or congestion is detected;
• probe occasionally to see whether more bandwidth is available.

The lecture did not go into the detailed congestion-control algorithms.

The important OS-level point is:

The TCP implementation must sometimes delay sending even if the application has already written data.

Reasons include:

• congestion control;
• flow control;
• retransmission scheduling;
• packetization decisions.

The application calls something like:

write(fd, buffer, len);

or an equivalent socket send operation.

The bytes do not immediately disappear onto the network.

Instead, they are inserted into the socket’s send buffer.

For TCP, the socket has a write queue / send buffer.

This buffer stores data that:

• has been written by the application;
• may not yet have been sent;
• may have been sent but not yet acknowledged.

The last point is crucial.

TCP must keep data until it is acknowledged, because it may need retransmission.

The TCP implementation decides when to create packets.

It considers:

• current sequence numbers;
• acknowledgement state;
• flow-control window;
• congestion-control state;
• packet size;
• retransmission needs.

In the common case, TCP may immediately create a packet after write.

But this is not guaranteed.

It may have to wait because:

• the receiver’s flow-control window is full;
• congestion control says to slow down;
• the NIC or queues are busy.

TCP adds the TCP header and passes the packet down to the IP layer.

The relevant Linux function name on the slide was ip_queue_xmit().

The IP layer determines where the packet should go next.

This includes routing-table lookup.

The next hop may be:

• the destination host itself, if it is on the same network;
• a router, if the destination is on another network.

Eventually, an Ethernet frame needs a destination MAC address.

For off-network destinations, this is usually the MAC address of the next-hop router.

After packet creation, Linux may pass the packet through the traffic-control subsystem.

This includes queueing disciplines, often called qdisc.

This layer can implement:

• traffic shaping;
• traffic policing;
• scheduling between packets;
• rate limits;
• prioritization.

Conceptually, it decides when a packet is allowed to be sent.

The lecturer described this as similar to network scheduling.

The packet is passed to the NIC driver.

The driver prepares the packet for the hardware.

Modern NICs typically use DMA and ring buffers.

For sending, the driver places descriptors into a transmit ring buffer.

A descriptor tells the NIC things such as:

• physical address of the packet buffer;
• length of the packet;
• metadata or flags.

The NIC asynchronously reads the packet from memory and transmits it.

After the NIC has sent the packet, it marks the descriptor as complete or otherwise notifies the driver.

Then the driver/OS can reuse the ring slot and free or recycle buffers when safe.

A packet arrives from the network at the network interface controller.

The NIC writes packet data directly into main memory using DMA.

The driver has already prepared receive buffers and made them available to the NIC.

For receiving, the driver maintains a receive ring buffer.

The ring contains descriptors pointing to empty packet buffers.

Before packets arrive, the driver populates descriptors with physical addresses of buffers.

When the NIC receives a packet:

• it takes the next available receive descriptor;
• writes the packet into the associated memory buffer;
• marks the descriptor as complete;
• may raise an interrupt or otherwise notify the driver.

This is separate from the transmit ring buffer.

Modern NICs usually have both:

• TX ring buffers for sending;
• RX ring buffers for receiving.

The driver picks up completed receive descriptors.

It obtains the packet buffer from memory.

It passes the packet upward into the OS network stack.

On Linux, the slide showed the path toward netif_rcv_skb().

The IP layer examines the packet.

It may:

• check header fields;
• check checksums;
• verify whether the destination IP address belongs to this host;
• drop the packet if it is not relevant;
• forward it if the machine is configured as a router;
• pass it upward to TCP or UDP based on the protocol field.

If the packet contains TCP, the TCP layer processes it.

It uses information such as:

• local IP;
• remote IP;
• local port;
• remote port.

This identifies the corresponding TCP connection and socket.

Then TCP examines:

• sequence number;
• acknowledgement number;
• flags;
• payload length;
• checksum;
• current connection state.

Depending on the packet, TCP may:

• process an acknowledgement;
• update retransmission state;
• update congestion-control state;
• insert received data into the receive queue;
• send an acknowledgement back;
• discard duplicate or invalid data.

If valid in-order data is available, TCP places it into the socket receive queue.

The queue must support two timing cases:

• data arrives before the application calls read;
• the application is already blocked in read before data arrives.

Thus, the OS needs buffering and synchronization.

The application calls something like:

read(fd, buffer, len);

The OS copies or maps available bytes from the socket receive queue to the application.

Once bytes have been delivered to the application, the OS can remove them from the receive queue.

This frees receive-buffer space.

A socket stores common state such as:

• socket type;
• protocol;
• local IP address;
• local port;
• remote IP address;
• remote port.

Sockets track memory usage and buffer limits.

Examples from Linux:

• sk_rcvbuf;
• sk_rmem_alloc;
• sk_sndbuf;
• sk_wmem_alloc.

These track receive/send buffer sizes and allocated memory.

A socket contains queues such as:

• receive queue;
• transmit queue.

For UDP, these queues are closer to packet queues.

For TCP, they represent byte-stream data and packet buffers associated with byte ranges.

Sockets also store protocol-specific state.

For UDP, this state is minimal.

For TCP, it is much larger and includes things such as:

• congestion window, often cwnd;
• round-trip-time estimates, such as rtt;
• sequence numbers;
• acknowledgement state;
• retransmission state;
• flow-control state;
• timers.

Socket buffer management is non-trivial.

If a system has many connections, send and receive buffers can consume a large amount of memory.

Therefore, the OS may dynamically tune buffer sizes.

It may:

• grow buffers when more space is useful;
• shrink buffers under memory pressure;
• enforce system-wide limits;
• allow applications to provide hints.

Applications and system administrators may configure socket buffer sizes for performance.

For example, high-performance databases or networking applications may require tuning maximum buffer sizes.

A packet moves through multiple protocol layers.

On the send path:

• TCP adds a TCP header;
• IP adds an IP header;
• Ethernet adds an Ethernet header.

So the packet grows as it moves down the stack.

On the receive path:

• Ethernet header is parsed/removed;
• IP header is parsed/removed;
• TCP header is parsed/removed;
• payload is delivered upward.

So the packet logically shrinks as it moves up the stack.

The OS wants to avoid copying packet data at every layer.

Therefore, packet buffers contain metadata and pointers, not just a raw byte array.

When creating a packet at the TCP layer, the OS does not necessarily start writing at the beginning of the buffer.

It may leave empty space at the front.

This is called headroom.

Reason:

• lower layers will need to prepend headers;
• leaving headroom avoids moving the whole payload later.

A packet buffer tracks positions such as:

• where the allocated buffer begins;
• where the current packet data begins;
• where the packet data ends;
• where protocol headers begin.

The lecture mentioned pointers conceptually like:

• head;
• data;
• tail.

Real implementations also track more protocol-specific pointers, for example:

• where the TCP header starts;
• where the IP header starts;
• where the link-layer header starts.

A packet may not always live in one contiguous memory region.

It may span multiple buffers.

Therefore, packet buffers can be chained.

For example:

skb -> skb -> skb

This is useful when:

• packet data is scattered across memory;
• the NIC uses scatter/gather DMA;
• the OS wants to avoid copying;
• large packets or segmented data are represented by multiple fragments.

Packet buffers also store metadata needed by the network stack.

Examples:

• which interface the packet came from;
• which protocol it belongs to;
• pointers to headers;
• packet length;
• checksum status;
• sequence-related information;
• queue links.

This metadata lets the OS process packets efficiently without repeatedly reparsing or copying data.

Ethernet:

• local-network packets;
• MAC addresses;
• weak guarantees.

IP:

• communication across networks;
• IP addresses;
• routing;
• still weak guarantees.

UDP:

• ports and multiplexing;
• datagram abstraction;
• no real reliability.

TCP:

• ports and multiplexing;
• reliable ordered byte stream;
• connection state;
• sequence numbers;
• acknowledgements;
• retransmission;
• flow control;
• congestion control.

TCP works because it carefully manages:

• byte-stream offsets;
• sequence numbers;
• cumulative acknowledgements;
• retransmission;
• duplicate suppression;
• timers;
• optional optimizations such as SACK and fast retransmit.

The OS translates between:

• application calls such as read and write;
• protocol state such as TCP sequence numbers;
• packet buffers;
• routing decisions;
• driver queues;
• NIC DMA.

Important queues include:

• socket send queue;
• socket receive queue;
• qdisc / traffic-control queue;
• NIC TX ring;
• NIC RX ring;
• driver receive lists.

Buffers are necessary because:

• applications and hardware run asynchronously;
• TCP must keep unacknowledged data;
• received data may arrive before the application reads;
• out-of-order data may need temporary storage;
• the NIC performs DMA asynchronously.

Conceptually, the OS network stack does this:

application byte stream / datagrams
        <-> socket buffers
        <-> protocol implementation
        <-> packet buffers
        <-> driver ring buffers
        <-> NIC
        <-> network

The complexity comes from making this efficient, reliable, and memory-safe under real-world conditions.