Transport Layer 2

TCP provides a point-to-point, reliable, in-order byte stream.

Important properties:

• point-to-point: one sender and one receiver;
• full-duplex: both directions can carry data in the same connection;
• reliable: lost data is retransmitted;
• in-order: the receiver application sees bytes in order;
• byte stream: TCP does not preserve application message boundaries.

The byte-stream abstraction is important. TCP does not know whether the application is sending HTTP messages, file chunks, or any other higher-level object. It only sees bytes.

TCP therefore numbers bytes in the sequence number space.

sequence number = number of the first byte in a segment
ACK number      = next byte expected by the receiver

ACKs are cumulative:

ACK = x
means:
  "I have received all bytes before x,
   and I am now waiting for byte x."

This has two advantages:

1. Lost ACKs are often harmless. If a later cumulative ACK arrives, it also acknowledges the earlier data.
2. Duplicate ACKs can indicate packet loss. If the receiver repeatedly ACKs the same sequence number, it means it is still waiting for the missing byte, while later segments may already have arrived out of order.

TCP can also use delayed ACKs to reduce overhead, and ACKs can be piggybacked on data sent in the opposite direction.

TCP uses pipelining: multiple segments may be in flight before their ACKs return.

Without pipelining, throughput would be very poor, especially when RTT is large.

Approximate throughput without pipelining:

\[ \text{throughput} \approx \frac{\text{segment size}}{\text{RTT}} \]

So if only one packet can be sent per RTT, a long RTT severely limits the throughput.

However, sending too many very small packets is also bad, because headers become a large fraction of the transmitted data. This is why mechanisms such as Nagle’s algorithm try to avoid many tiny packets in flight.

Flow control prevents the sender from overwhelming the receiver.

The receiver has a receive buffer. Data may arrive from the network before the application reads it. If the receive buffer becomes full, the sender must stop sending more data.

The receiver advertises its available buffer space as the receiver window:

\[ rwnd = \text{free space in receiver buffer} \]

The sender may not have more unacknowledged data in flight than allowed by the receiver window.

Important distinction:

flow control:
  do not overwhelm the receiver

congestion control:
  do not overwhelm the network

If the advertised receiver window becomes zero, the sender cannot send normal data. TCP may send small probe segments to discover when the receiver window opens again.

TCP needs connection state because it maintains many variables:

• sequence numbers;
• ACK numbers;
• send and receive windows;
• retransmission timers;
• congestion-control state.

Connection setup uses the three-way handshake.

Conceptually:

Client -> Server: SYN
Server -> Client: SYN + ACK
Client -> Server: ACK

The purpose is to synchronize both directions of the byte stream. Each side must learn the other side’s initial sequence number.

Connection closing uses FIN segments.

A FIN consumes one sequence number. Therefore, it must be acknowledged. Since packets can be lost, the host must keep state for some time in order to retransmit or re-acknowledge connection-closing packets if needed.

The closing exchange can look like a three-way exchange if one side’s FIN is combined with the ACK for the other side’s FIN, but the two directions are logically independent. The FINs do not have to occur at exactly the same time.

Congestion means that too many sources are sending too much data too fast for the network to handle.

It appears as:

• long delays: packets wait in router queues;
• packet losses: router buffers overflow and packets are dropped.

Congestion is different from flow control.

congestion control:
  too many senders are too fast for the network

flow control:
  one sender is too fast for one receiver

End hosts usually do not know the internal state of the network.

For example, a sender may have a 100 Gbit/s access link, but somewhere on the path there may be an 8 Mbit/s bottleneck. The sender does not know:

• the capacity of every link on the path;
• which link is the bottleneck;
• how many other senders are sharing the same bottleneck;
• how much traffic the other senders are currently sending.

Therefore, congestion control is a distributed systems problem.

Each sender must infer the network’s available capacity indirectly, usually from signals such as:

• packet loss;
• duplicate ACKs;
• timeouts;
• increasing RTT;
• explicit congestion marks from routers, if available.

Congestion collapse happens when increasing the offered load decreases the useful work done by the network.

In other words:

\[ \text{more input traffic} \quad \Rightarrow \quad \text{less useful output} \]

This is similar to a traffic jam: adding a little more traffic can make the entire system much slower.

Congestion collapse is not just theoretical. It historically occurred in the early Internet/NSFnet, and Van Jacobson’s TCP congestion control mechanisms were introduced to address this kind of problem.

Assume:

• two senders;
• two receivers;
• one bottleneck router;
• infinite buffers;
• no retransmissions.

In the ideal case, as the input rate increases, the output rate increases up to the bottleneck capacity.

However, as the offered load approaches capacity, queues grow. With infinite buffers, packets are not dropped, but delay grows dramatically.

Main lesson:

\[ \text{Even without packet loss, congestion causes large delay.} \]

Real routers have finite buffers.

When the buffer is full, packets are dropped. TCP then retransmits lost data.

Let:

\[ \lambda_{in} \]

be original data sent into the network, and

\[ \lambda'_{in} \]

be original data plus retransmitted data.

If packets are lost and retransmitted, the network must carry more traffic for the same useful output.

This reduces goodput.

goodput = useful application data delivered per second
throughput = total data transmitted per second, including retransmissions

Costs of congestion:

• retransmissions consume bandwidth;
• a link may carry multiple copies of the same packet;
• if a packet was not really lost but only delayed, retransmitting it is pure waste;
• unnecessary retransmissions can make congestion worse.

This is why TCP should not retransmit too aggressively. Bad timers can create spurious retransmissions, which waste capacity and may contribute to congestion collapse.

In a multi-hop network, a packet may travel over several links before being dropped downstream.

If a packet is dropped after already crossing several upstream links, all upstream transmission capacity used for that packet was wasted.

This is worse than dropping the packet immediately at the source-side edge.

Main lesson:

\[ \text{Dropping packets late wastes more network resources.} \]

Congestion in one part of the network can also starve other flows. A router may spend its capacity forwarding packets that will later be dropped, while other packets cannot get through.

Congestion causes:

• delay due to queueing;
• packet loss due to buffer overflow;
• retransmission overhead;
• duplicate packets;
• wasted upstream capacity;
• wasted buffering;
• lower effective goodput;
• possible congestion collapse.

Throughput can never exceed bottleneck capacity, but delay can grow very large as the network approaches capacity.

In end-to-end congestion control, routers do not explicitly tell senders what to do.

The sender infers congestion from end-system observations, such as:

• timeout;
• duplicate ACKs;
• packet loss;
• RTT increase.

Traditional TCP mostly follows this approach.

Advantages:

• routers stay simple;
• deployment is easier;
• the mechanism works without requiring every router to participate.

Disadvantages:

• congestion is detected indirectly;
• packet loss may already have happened by the time the sender reacts;
• the sender only has partial information.

In network-assisted congestion control, routers provide feedback to end hosts.

Possible forms:

• choke packets: router sends a special packet to tell sender to slow down;
• single congestion bit: router marks packets to indicate congestion;
• explicit rate: router tells the sender what rate to use.

Choke packets are problematic because they add traffic and can be abused.

Modern TCP/IP can use ECN, Explicit Congestion Notification, where routers mark packets instead of dropping them.

End hosts can prevent congestion collapse if they behave correctly, but the network must trust them.

Routers cannot prevent all forms of congestion collapse alone, but they can help by:

• sending accurate congestion signals;
• isolating well-behaved from ill-behaved sources;
• using buffer-management policies;
• marking packets early.

The congestion window limits how much data may be in flight.

A rough sending rate is:

\[ \text{rate} \approx \frac{cwnd}{RTT} \]

So increasing \(cwnd\) increases the sending rate, while decreasing \(cwnd\) reduces the sending rate.

A key design principle is packet conservation:

At equilibrium, inject a new packet into the network only when one packet has left the network.

ACKs are evidence that data has left the network.

Therefore, TCP can use returning ACKs to clock the sending of new data.

This avoids sending one large burst all at once. Instead, ACKs naturally pace the sender according to the bottleneck link.

TCP wants two properties:

1. Efficiency: use the available capacity.
2. Fairness: if multiple TCP flows share a bottleneck, they should converge toward a fair share.

For two flows, a phase plot can show their allocations:

• efficiency line: total allocation equals bottleneck capacity;
• fairness line: both flows get equal share.

The ideal point is the intersection of the efficiency and fairness lines.

Different control choices are possible:

• additive increase;
• additive decrease;
• multiplicative increase;
• multiplicative decrease.

The desirable combination is AIMD:

Additive Increase:
  slowly probe for more bandwidth

Multiplicative Decrease:
  quickly reduce load after congestion

In congestion avoidance:

\[ cwnd \leftarrow cwnd + 1 MSS \quad \text{per RTT} \]

On loss:

\[ cwnd \leftarrow \frac{cwnd}{2} \]

This creates the classic TCP saw-tooth behavior:

increase, increase, increase, loss,
cut in half,
increase, increase, increase, loss,
cut in half,
...

AIMD is simple but powerful. It probes for unused capacity and backs off when loss indicates congestion.

At the beginning of a connection, TCP does not know the available network capacity.

Starting too aggressively could immediately overflow queues.

Starting too slowly would waste capacity, especially on high-bandwidth paths.

TCP uses slow start to quickly discover a reasonable sending rate.

Despite the name, slow start is not actually slow after the first RTT. It grows exponentially.

Initially:

\[ cwnd = 1 MSS \]

Modern TCP implementations often use a larger initial window, but conceptually the algorithm starts small.

For every ACK received:

\[ cwnd \leftarrow cwnd + 1 MSS \]

This means that \(cwnd\) approximately doubles every RTT:

1 MSS
2 MSS
4 MSS
8 MSS
16 MSS
...

Slow start continues until:

• a loss occurs;
• or \(cwnd\) reaches the slow-start threshold \(ssthresh\).

When a loss occurs, TCP sets:

\[ ssthresh = \frac{cwnd}{2} \]

Then future growth changes:

• below \(ssthresh\): slow start, exponential growth;
• above \(ssthresh\): congestion avoidance, linear growth.

The intuition is:

• before the loss, the network could almost support that window;
• half of that value is a safer point to resume from.

A timeout is a severe signal.

It means the sender did not receive enough ACKs in time. This may indicate that the ACK clock is broken and the network is heavily congested.

Reaction:

\[ ssthresh = \frac{cwnd}{2} \]

\[ cwnd = 1 MSS \]

Then TCP restarts with slow start.

Timeouts are expensive because:

• the sender waits for the retransmission timeout;
• the RTO must be conservative to avoid spurious retransmissions;
• the congestion window collapses to a small value;
• throughput recovery takes time.

Three duplicate ACKs are less severe than a timeout.

They indicate that:

• one segment is probably missing;
• but later segments are still arriving;
• therefore the network is still delivering some data.

This allows fast retransmit:

on 3 duplicate ACKs:
  retransmit the missing segment before waiting for timeout

TCP Reno also performs fast recovery:

\[ cwnd \leftarrow \frac{cwnd}{2} \]

Then it continues with congestion avoidance rather than restarting from one MSS.

TCP Tahoe:

• uses slow start;
• uses congestion avoidance;
• uses fast retransmit after three duplicate ACKs;
• but after timeout or three duplicate ACKs, it resets \(cwnd\) to 1 MSS.

TCP Reno:

• behaves like Tahoe for timeout;
• but after three duplicate ACKs, it cuts \(cwnd\) roughly in half instead of resetting to 1 MSS.

Main difference:

Tahoe:
  duplicate ACK loss -> cwnd = 1 MSS

Reno:
  duplicate ACK loss -> cwnd roughly halved

Reno performs better because duplicate ACKs show that the ACK clock is still alive.

In Reno, after three duplicate ACKs:

\[ ssthresh = \frac{cwnd}{2} \]

\[ cwnd = ssthresh + 3 MSS \]

The extra \(3 MSS\) accounts for the three duplicate ACKs, which indicate that three later segments have left the network and reached the receiver.

For each additional duplicate ACK, Reno may increase \(cwnd\) by one MSS and send another segment if allowed. This keeps the ACK clock running.

When a new ACK arrives that acknowledges new data, Reno exits fast recovery and sets:

\[ cwnd = ssthresh \]

If \(N\) TCP sessions share the same bottleneck link, ideally each should get:

\[ \frac{1}{N} \]

of the link capacity.

For two flows sharing a bottleneck of capacity \(R\), each should ideally get:

\[ \frac{R}{2} \]

AIMD helps TCP flows converge toward fairness:

• additive increase: both flows increase by roughly the same amount;
• multiplicative decrease: both flows reduce proportionally.

This tends to move flows toward equal sharing over time.

TCP fairness is not automatic in all cases.

It depends on assumptions such as:

• same RTT;
• large enough congestion windows;
• similar TCP parameters;
• both flows have enough data to send;
• flows are long-lived enough to converge.

If one flow has a much shorter RTT, it receives ACKs more frequently and can increase its window faster. Therefore, it may get more bandwidth than a flow with a larger RTT.

This is why TCP fairness is mainly a long-term and idealized property.

There are many TCP congestion control variants.

They usually do not change the basic TCP abstraction:

• byte stream;
• sequence numbers;
• ACKs;
• connection setup;
• flow control.

They mainly differ in congestion control:

• what signal indicates congestion;
• how aggressively to increase \(cwnd\);
• how to react to loss;
• how to handle different RTTs;
• how to behave in high-speed, wireless, or data-center environments.

A TCP sequence number plot visualizes TCP behavior in two dimensions:

x-axis: time
y-axis: sequence number

Events are plotted as points:

X = data packet
+ = ACK packet

This plot can reveal much of what TCP is doing.

A sequence number plot can show:

• RTT: time between a data packet and the corresponding ACK;
• TCP segment size: vertical distance between consecutive data packets;
• connection duration: time between first packet and final ACK;
• number of bytes sent: sequence number range covered;
• average throughput: total bytes divided by connection duration;
• access link bandwidth: slope of closely spaced outgoing packets;
• sender window size: amount of data sent before waiting;
• slow start: 1, 2, 4, 8 packet bursts;
• delayed ACKs: ACKs for every second segment rather than every segment;
• packet loss: missing data point followed by duplicate ACKs or timeout;
• retransmission: same sequence number sent again;
• cumulative ACK: one ACK advances over multiple segments;
• RTO: long waiting time before retransmission.

The lecture emphasized:

Not all packet losses are created equal.

The impact of a loss depends on where it occurs in the transfer.

A TCP flavor is a variant of TCP, usually differing in its congestion control algorithm.

The basic TCP service remains the same, but the congestion-control behavior changes.

Examples:

• Tahoe;
• Reno;
• New Reno;
• Vegas;
• BIC;
• CUBIC;
• Compound TCP;
• BBR.

TCP Tahoe includes:

• slow start;
• congestion avoidance using AIMD;
• RTO estimation from RTT;
• fast retransmit after three duplicate ACKs.

However, Tahoe is conservative.

On either timeout or three duplicate ACKs:

\[ cwnd = 1 MSS \]

This makes Tahoe robust but inefficient, because it often restarts from a very small window.

TCP Reno is like Tahoe, but adds fast recovery.

On timeout:

\[ cwnd = 1 MSS \]

On three duplicate ACKs:

\[ cwnd \approx \frac{cwnd}{2} \]

Reno’s insight:

Duplicate ACKs likely imply a loss, but the returning ACKs also indicate that some data is still being delivered.

Therefore, it is unnecessary to go all the way back to one MSS after duplicate ACKs.

Reno performs poorly when multiple packets are lost in one window.

Problem:

• Reno may retransmit one missing segment;
• a partial ACK may acknowledge some data but not all outstanding data;
• Reno may exit fast recovery too early;
• another loss then causes a timeout.

New Reno modifies fast recovery.

Key idea:

• partial ACKs do not immediately take the sender out of recovery;
• a partial ACK triggers retransmission of the segment following the ACKed segment;
• the sender stays in recovery until all data outstanding at the start of fast recovery has been acknowledged.

New Reno is therefore better when multiple packets are lost in the same window.

Tahoe, Reno, New Reno, and CUBIC are mainly loss-based.

They infer congestion by observing packet loss.

This is effective but conceptually late:

The network has already become congested enough to drop packets.

Loss-based algorithms often operate near or beyond the congestion knee.

Instead of waiting for packet loss, TCP Vegas tries to detect congestion earlier.

Observation:

If queues start to fill, RTT increases.

RTT consists of:

• propagation delay;
• transmission delay;
• processing delay;
• queueing delay.

Under no load, queueing delay is near zero. The minimum observed RTT is an estimate of the base RTT:

\[ RTT_{base} = \min RTT \]

If current RTT becomes larger than the base RTT, that suggests queueing delay, therefore congestion is beginning.

Vegas estimates expected throughput using the base RTT:

\[ B' \approx \frac{cwnd}{RTT_{base}} \]

It compares this with the actual throughput based on current RTT.

If RTT increases, queues are filling, so Vegas reduces the sending rate.

If RTT stays low, the network can likely handle more data, so Vegas increases the sending rate.

The slide describes decision boundaries using parameters \(\alpha\) and \(\beta\):

\[ B \cdot RTT_{base} + \alpha \leq cwnd \leq B \cdot RTT_{base} + \beta \]

Conceptually:

• below lower bound: increase \(cwnd\);
• above upper bound: decrease \(cwnd\);
• inside the target range: keep roughly stable.

Vegas uses additive increase and additive decrease.

Benefit:

• detects congestion before packet loss;
• can reduce queueing delay;
• can avoid filling buffers.

Limitations:

• accurate RTT measurement was historically difficult;
• path changes or variable wireless delays can look like congestion;
• delay-based flows may be disadvantaged when competing with aggressive loss-based flows.

Bufferbloat occurs when routers or access devices have very large buffers.

Large buffers reduce packet loss, but they can create huge queueing delays.

Example intuition:

• link rate is small;
• buffer is very large;
• once the buffer fills, packets may wait seconds before transmission.

This is bad for interactive applications:

• video calls;
• online games;
• remote shells;
• real-time audio.

Sometimes packet loss is easier to handle than extremely long queueing delay.

On high bandwidth-delay product paths, linear increase can take too long to reach available capacity.

Bandwidth-delay product:

\[ BDP = \text{bandwidth} \times RTT \]

A large BDP means a large amount of data must be in flight to fully use the link.

With AIMD increasing by only one MSS per RTT, reaching the correct window size can be too slow.

BIC stands for Binary Increase Congestion control.

Core idea:

Treat congestion avoidance as a search problem.

Instead of linearly probing upward, use a binary-search-like strategy between:

• \(W_{min}\): known safe lower window;
• \(W_{max}\): window near which loss occurred.

BIC jumps faster toward the estimated available bandwidth, while bounding the step size with \(S_{max}\) and \(S_{min}\).

Main lesson:

\[ \text{Binary search reaches high bandwidth faster than linear search.} \]

BIC is designed for high-speed, high-BDP networks.

CUBIC refines BIC and is suitable for high-speed sessions.

It replaces simple AIMD growth with a cubic function.

The exact formula is not important for the exam according to the lecture. The shape is important.

CUBIC behavior:

• after a loss, reduce the window;
• then ramp up quickly toward the previous maximum window;
• enter a stable region near the previous maximum;
• then slowly accelerate beyond it to probe for more bandwidth.

Conceptually:

fast ramp up -> stable region -> slow probing beyond previous maximum

CUBIC is the default TCP congestion control implementation on Linux.

BBR stands for Bottleneck Bandwidth and Round-trip propagation time.

It was proposed by Google.

Unlike Reno or CUBIC, BBR tries not to use packet loss as the main congestion signal.

BBR tries to estimate:

• bottleneck bandwidth;
• round-trip propagation time.

It aims to operate near the point where throughput is high but queues have not grown much.

This is sometimes called Kleinrock’s point:

maximum bandwidth with minimum RTT

BBR is similar to Vegas in that it wants to avoid standing queues.

But BBR actively probes for available bandwidth.

Periodically, BBR sends above its estimated rate for about one RTT.

If available bandwidth has increased, BBR can use it.

If available bandwidth has not increased, the probe creates a queue and RTT increases, telling BBR that the extra rate is not sustainable.

BBR uses pacing.

Instead of sending a whole window as a burst, it spreads packets over time at a controlled rate.

This is better for the network:

bad:
  send a big burst, then wait

better:
  send smoothly at approximately the target rate

Pacing reduces queue spikes and burst-induced losses.

BBR has phases such as:

• startup: rapidly estimate available bandwidth;
• probing bandwidth: periodically test if more bandwidth is available;
• estimating minimum RTT: occasionally reduce inflight data to measure base RTT.

Compared with Reno or CUBIC, BBR tends to keep queues smaller, but it still has periodic probing spikes.

BBR is not perfect.

Problems mentioned in the lecture:

• it may not perform well in dynamic environments such as mobile networks;
• RTT variation may be caused by wireless access delays, not congestion;
• it may be unfair to CUBIC or other loss-based flows.

BBRv2 aimed to improve fairness to loss-based congestion-control algorithms and added support for ECN.

BBRv3 further refined BBRv2, especially convergence behavior and performance tuning.

Reno increases its sending rate until packet loss occurs.

This tends to fill the bottleneck queue.

After loss, it reduces \(cwnd\), then begins increasing again.

Queue behavior:

queue builds -> packet loss -> cwnd reduced -> queue drains
-> cwnd grows -> queue builds again

This creates a saw-tooth pattern in both cwnd and queue occupancy.

CUBIC ramps up quickly toward the previous maximum and then probes more gently.

Its queue behavior is different from Reno because the window-growth function is not simple linear AIMD.

CUBIC is designed to be more efficient on high-speed links.

BBR tries to avoid maintaining a large queue.

It estimates the delivery rate and paces traffic.

However, during probing phases, BBR may intentionally send more than the estimated bandwidth-delay product, creating temporary queue spikes.

Overall, BBR often keeps network queues smaller than Reno/CUBIC, but its probing can still cause periodic queue variation.

ECN is network-assisted congestion control.

Instead of dropping packets when congestion begins, a router can mark packets.

The receiver then informs the sender using TCP header bits.

This gives the sender an early congestion signal.

ECN uses both IP and TCP headers.

At a high level:

1. Sender indicates that it supports ECN.
2. Router sees growing congestion and marks the IP packet.
3. Destination receives the marked packet.
4. Destination sets the ECE bit in an ACK.
5. Sender receives the ACK and reduces its sending rate.

This avoids using packet loss as the only congestion signal.

Congestion is directional.

The path from A to B and the path from B to A may be different and may have different congestion.

Therefore, ECN information has to be handled per direction.

ECN is especially useful in data centers, where operators control the network and can configure routers and hosts consistently.

Not all Internet paths support ECN.

Data centers have special properties:

• high throughput;
• very low latency;
• bursty applications;
• incast traffic: many senders sending to one receiver;
• controlled network environment.

The goal is to get close to raw network latency and bandwidth even under heavy load.

NDP is not merely a TCP flavor. It changes more of the stack, including switch behavior, routing, and the transport protocol.

NDP aims for:

• low latency between hosts;
• receiver prioritization;
• predictable high throughput;
• avoiding congestion collapse.

Key insight:

Packet headers are small compared to full packets.

If a switch cannot forward a full data packet because of congestion, it can trim the packet: drop the payload but keep and forward the header.

This header is enough to inform the receiver that the packet existed but the payload did not arrive.

Then the receiver can send a NACK.

normal TCP:
  missing data inferred indirectly by ACK behavior

NDP:
  trimmed header reaches receiver;
  receiver explicitly knows which packet payload was missing;
  receiver sends NACK

NDP switches use:

• separate queues for data packets and trimmed packets;
• higher priority for trimmed header packets;
• weighted round-robin scheduling;
• decisions about whether to trim new packets or packets at the end of a low-priority queue.

This sends congestion feedback early and helps avoid congestion collapse.

NDP’s transport can send a full window in the first RTT.

If some packets are trimmed, the receiver sends NACKs, and the sender retransmits the missing payloads.

This works well in data centers because the environment is controlled and predictable.

It is not directly deployed as general Internet TCP because the Internet has more diverse paths, policies, and failure modes.