Router Architecture

The Internet cannot be operated as one single global ISP.

Instead, it consists of many interconnected networks:

• access and regional ISPs;
• large transit and tier-1 providers;
• enterprise and university networks;
• Internet exchange points;
• content-provider networks and CDNs.

Content-provider networks operate on top of the underlying Internet infrastructure. They use the Internet for transport while implementing replication, caching, and server selection at the application layer.

This approach is often preferable to implementing these functions inside the network layer because maintaining large amounts of application-specific state in routers would not scale.

Internet infrastructure is not distributed uniformly.

Important network facilities tend to be concentrated around:

• large population centers;
• economically important regions;
• Internet exchange points;
• data centers;
• locations with inexpensive electricity and available space;
• geographically suitable cable routes and landing points.

The same geographical constraints apply to submarine cables. A large fraction of intercontinental traffic passes through a limited number of cable corridors and landing regions.

Multiple cables and landing points provide redundancy, but submarine cables remain strategically important infrastructure.

Cutting a cable can disrupt connectivity, although networks may reroute traffic through alternative cables. Monitoring traffic directly from an optical cable is also technically difficult because the cable carries enormous amounts of multiplexed optical traffic.

An AS knows:

• the routes it receives from its own neighbors;
• the routes it selects;
• the routes it exports.

However, it normally cannot directly observe:

• the policies used by every other AS;
• how another AS selected a route;
• how its own prefix is seen from a remote part of the Internet;
• the return path from a remote network.

Routing policies are often private and are strongly influenced by business relationships.

Therefore, the path observed locally does not reveal the complete global routing situation.

A looking glass is a service that allows users to execute routing or diagnostic queries from a router in another network.

Depending on the service, it may support commands such as:

• displaying a BGP route;
• displaying a BGP summary;
• showing routes received from a particular neighbor;
• running traceroute;
• running ping.

Looking glasses provide remote vantage points.

For example, a local traceroute approximates the forward path from the local host to a destination. It does not reveal the reverse path from the destination back to the local network.

A looking glass in the remote network can run a traceroute in the opposite direction and provide an approximation of that reverse path.

Different looking-glass services have different interfaces and expose different subsets of router commands.

Distributed measurement platforms such as RIPE Atlas provide probes at many locations and networks.

They can be used to perform measurements from multiple vantage points, including:

• latency measurements;
• traceroutes;
• reachability tests;
• DNS measurements.

This helps reveal how connectivity differs depending on the observer’s location.

Services such as bgp.tools and similar BGP toolkits combine several types of information about ASes and prefixes.

They may show:

• prefixes originated by an AS;
• upstreams, downstreams, and peers;
• observed AS paths;
• BGP communities;
• route collectors that see a route;
• registration and WHOIS information;
• changes in routing history.

For example, AS680 is associated with the German research network DFN, which provides connectivity to universities and research institutes.

Other examples discussed included:

• AS2, an early AS number assigned to the University of Delaware;
• AS701, associated with Verizon Business;
• AS7018, associated with AT&T.

WHOIS and related registry services can identify:

• the organization responsible for an AS number;
• the organization to which an address block was allocated;
• administrative and technical contacts;
• delegation and registration information.

Examples include:

• 12.0.0.0/8, allocated to AT&T;
• private address space such as 10.0.0.0/8, which should not appear as globally routed public space;
• 1.1.1.1, used by Cloudflare’s anycast DNS resolver;
• 8.8.8.8, used by Google Public DNS.

The allocation of a larger address block does not imply that every address in the block is used in the same way. Parts of the block may be sub-allocated or announced by different services.

Many ISPs voluntarily establish BGP sessions with route collectors.

Examples of collector infrastructures include RIPE RIS and RouteViews.

Collectors do not normally forward user traffic. Instead, they receive BGP routing information for measurement and analysis.

The collected data allows operators and researchers to observe:

• route announcements;
• route withdrawals;
• AS-path changes;
• route flapping;
• changes in reachability;
• how an update propagates through different ASes.

BGPlay visualizes BGP routing history.

A user can select an IP prefix and a time interval. BGPlay then shows how different route collectors observed routes to that prefix.

Its animation can display events such as:

• a new announcement;
• a withdrawal;
• a path becoming longer or shorter;
• the selected path changing from one origin or neighbor to another;
• routing instability propagating across the Internet.

A stable prefix may show almost no change for hours or weeks, while an unstable prefix may produce many updates and rapidly changing paths.

BGP communities attach operator-defined labels to routes.

Traditional communities are commonly displayed as:

AS-number:value

Their interpretation depends on the network that defines them.

They may indicate information such as:

• where a route was learned;
• the city or point of presence involved;
• the type of customer or peer;
• how a route should be exported;
• whether local preference should be changed.

Communities are especially useful for routing policy and filtering.

A looking glass may also show:

• the neighbor from which a route was learned;
• whether the route was learned internally or externally;
• when the route was last updated;
• the complete AS path;
• communities associated with the route.

These tools help network operators determine:

• who is announcing a prefix;
• how other networks reach their AS;
• whether policies are being applied as expected;
• where routing instability originates;
• whether a route leak or misconfiguration is occurring.

They are also valuable for researchers studying the global routing system.

The routing system contains many configuration errors. These tools give partial transparency into a system whose policies and topology are otherwise difficult to observe.

• Unicast: one source communicates with one specific destination.
• Multicast: one source sends to a group of destinations.
• Anycast: one source sends to one member of a group, normally the member selected as best or nearest by routing.

An anycast address looks like an ordinary unicast IP address to the client.

The difference is that the same prefix is announced from multiple locations using BGP.

BGP then directs each client toward one of those locations.

“Nearest” does not necessarily mean geographically nearest. It means the location preferred by BGP according to factors such as:

• local preference;
• AS-path length;
• peering and transit policy;
• MED;
• BGP communities;
• IGP cost to the next hop.

Anycast was traditionally considered most suitable for connectionless services, especially UDP.

A DNS query, for example, may be completed with one request and one response. If a later query reaches another replica, little connection state is lost.

A stateful TCP connection is more difficult:

• the connection state is stored at one server;
• if routing changes during the connection, later packets might reach another server;
• the new server does not normally possess the TCP state.

In modern networks, routing changes during short-lived connections are usually uncommon. Therefore, anycast is also widely used for TCP and QUIC-based services, provided that the deployment can tolerate or mitigate occasional replica changes.

A CDN may announce the same prefix from many data centers.

BGP directs the user to one of these sites. Inside the selected site, a load balancer can distribute connections among individual servers or services.

This provides:

• low latency;
• geographic distribution;
• failure tolerance;
• reduced load on the origin;
• simplified addressing, because all sites can use the same service address.

BGP gives the operator only limited control over fine-grained client assignment.

For example, an operator may want:

• one group of clients to use data center A;
• another nearby group to use data center B;
• traffic to move because of server load rather than route length;
• some customers to use different replicas despite similar BGP paths.

Such precise mapping can be difficult when selection is performed only through BGP.

Anycast decisions may also change when routing policy or reachability changes.

DNS-based server selection offers more application-level control.

DNS uses a mixture of recursive and iterative queries.

1. The requesting host sends a recursive query to its resolver.
2. The resolver performs iterative queries toward:
   • a root server;
   • a TLD server;
   • the authoritative server.
3. The resolver returns the final answer to the client.

The root, TLD, and authoritative servers should not perform arbitrary recursive resolution for clients because that would require them to maintain additional state and perform additional work.

The authoritative server normally sees the address of the recursive resolver, not the address of the original client.

A CDN may therefore assume that the client is close to the resolver and return a CDN server close to the resolver.

This works reasonably well when the client uses a nearby ISP resolver.

It may work poorly when:

• the resolver is geographically distant;
• the resolver sends its upstream queries from a distant location;
• a public resolver’s apparent location differs from the client’s location.

For example, the CDN may choose a server 10 ms from the resolver but 100 ms from the client, even though another CDN server is much closer to the client.

This is one reason local ISP resolvers have traditionally been useful for CDN mapping.

EDNS Client Subnet, or ECS, allows a recursive resolver to send information about the client’s network to an authoritative DNS server.

The authoritative server can then choose an answer based on the client’s approximate location rather than only on the resolver’s location.

ECS is an option of the EDNS extension mechanism for DNS.

An ECS option contains information including:

• the address family, such as IPv4 or IPv6;
• the source prefix length;
• the scope prefix length;
• a prefix derived from the client’s IP address.

The resolver normally sends only a truncated client prefix, not necessarily the complete client address.

For example, it might identify the client’s IPv4 network using a prefix rather than identify one exact host.

The source prefix length says how many client-address bits were included in the query.

The scope prefix length in the response says how many bits of the client prefix the answer applies to.

This information helps the resolver determine which later requests may reuse the cached answer.

ECS can improve CDN server selection when the recursive resolver is far from the client.

It allows the authoritative DNS server to select a replica according to the client’s approximate network location.

ECS reduces DNS cache efficiency.

Without ECS, all clients using one resolver may share the same cached answer. With ECS, the resolver may need separate cache entries for many client prefixes.

ECS also reveals information about the client’s network to authoritative servers, creating privacy concerns.

Consequently, ECS represents a tradeoff among:

• accurate CDN mapping;
• DNS cache efficiency;
• client privacy;
• additional DNS complexity.

A router may range from:

• a small home router;
• an enterprise or access router;
• a large modular backbone router.

A home router may be implemented using a relatively conventional CPU, memory, and several Ethernet or Wi-Fi interfaces.

A core router may be a large chassis containing:

• multiple line cards;
• many high-speed ports;
• one or more routing processors;
• a high-capacity switching fabric;
• redundant power, control, and cooling components.

Manufacturers of high-end routers include Cisco, Juniper, Nokia, and Huawei.

The use of particular vendors in critical infrastructure may also become a political and security issue, although the architectural principles are independent of the manufacturer.

A modular router contains three important groups of components:

• routing processor: runs routing protocols and management software;
• line cards: terminate physical links and perform high-speed forwarding;
• switching fabric: transfers packets from input line cards to output line cards.

A line card may expose many ports. Its total port capacity is bounded by the capacity of the line card and its connection to the switching fabric.

For example, a line card with a total capacity of 10 Tb/s might expose:

• ten 1-Tb/s ports;
• one hundred 100-Gb/s ports;
• some other combination whose aggregate rate does not exceed the card’s capacity.

Routing determines which paths should be used.

It includes:

• processing BGP, OSPF, IS-IS, or other routing updates;
• applying routing policy;
• calculating routes;
• selecting next hops;
• constructing routing and forwarding tables.

Routing decisions may take milliseconds, hundreds of milliseconds, or seconds.

Forwarding processes every packet.

For each packet, the router must:

• inspect the relevant header fields;
• find the matching FIB entry;
• determine the output interface and next hop;
• transfer the packet to the output;
• possibly apply an ACL or another forwarding action.

At terabit-per-second rates, forwarding decisions must occur on a nanosecond-scale timescale.

It is therefore impossible to send every ordinary packet through a general-purpose routing CPU.

High-speed forwarding is instead implemented in specialized hardware on the line cards.

The Routing Information Base, or RIB, stores routing information.

A router may maintain separate RIBs or protocol-specific route tables for:

• BGP;
• OSPF;
• IS-IS;
• static routes;
• directly connected networks;
• other routing sources.

The router applies policy and route-selection rules to these candidates to construct a global routing view.

A RIB entry may include:

• destination prefix;
• prefix length or network mask;
• next-hop address;
• outgoing interface;
• metric or cost;
• route source;
• route status;
• policy attributes;
• quality-of-service information;
• access-control information.

A RIB may contain several candidate routes to the same prefix, possibly with different next hops or routing attributes.

The Forwarding Information Base, or FIB, contains the information needed by the data plane to forward packets.

The FIB is derived from the selected routes in the RIB.

The distinction is:

• the RIB may contain multiple competing routes;
• the FIB normally contains the selected forwarding result for each prefix.

The FIB removes information that is unnecessary for per-packet forwarding.

A simplified FIB entry answers:

For a packet matching this destination prefix, which next hop and output interface should be used?

Copies of the FIB are installed on line cards so that each card can perform decentralized forwarding without consulting the routing processor for every packet.

The data plane performs packet forwarding.

Characteristics:

• implemented mainly in ASICs and line-card hardware;
• operates at line rate;
• works on nanosecond-scale timescales;
• performs FIB lookup, packet transfer, queueing, and output scheduling.

The control plane constructs the routing state.

It:

• runs routing protocols;
• receives routing updates;
• applies route-selection policy;
• builds the RIB;
• derives the FIB;
• installs forwarding state in the data plane;
• commonly handles exceptional control traffic such as ICMP errors.

Its timescale ranges from milliseconds to seconds.

The management plane is used by operators to configure and supervise the router.

It includes:

• configuration changes;
• routing-policy changes;
• monitoring;
• traffic engineering;
• capacity planning;
• provisioning new links.

Its timescale can range from seconds to weeks or longer.

Physical expansion is especially slow. Activating an existing optical channel may be relatively quick, but installing new terrestrial fiber or a new submarine cable may take months or years.

When new capacity cannot be added immediately, operators may use traffic engineering to redirect traffic through alternative paths.

Traffic engineering may involve:

• configuring routing policies;
• changing OSPF or IS-IS link weights;
• adjusting BGP attributes;
• changing peering or transit arrangements.

Early routers were conventional computers.

The process was approximately:

1. copy the packet from the input interface into main memory;
2. let the CPU determine the output;
3. copy the packet from memory to the output interface.

Each packet crosses the system bus twice.

The maximum forwarding rate is therefore limited by:

• memory bandwidth;
• system-bus bandwidth;
• CPU processing capacity.

This architecture is simple and is still relevant to software routers and some low-end devices, but it does not scale to very high aggregate rates.

The input port transfers the packet directly to the output port across a shared bus.

The packet carries information identifying its destination output.

All ports share the same bus, so only a limited amount of traffic can cross it simultaneously.

The bus bandwidth therefore limits the aggregate switching rate.

This approach can be sufficient for access or enterprise routers but does not scale indefinitely.

High-end routers use an interconnection network such as:

• a crossbar;
• a Banyan network;
• another multistage switching network.

Such a fabric can transfer several packets in parallel when they use non-conflicting paths and output ports.

This overcomes the single shared-bus bottleneck and supports much higher aggregate throughput.

Network packets vary greatly in size. Ethernet frames, for example, can range from roughly 64 bytes to 1500 bytes.

Allowing one large packet to occupy a switching path for a long time can delay many short packets.

Some routers therefore:

1. split packets into fixed-size internal cells;
2. attach internal metadata identifying the output line card;
3. switch the cells independently through the fabric;
4. reassemble the original packet at the output.

Fixed-size cells simplify scheduling and prevent a single long packet from monopolizing an internal timeslot.

Hosts and routers are nodes.

A communication channel connecting physically adjacent nodes is a link.

Links may be:

• wired;
• wireless;
• point-to-point;
• shared by multiple stations;
• part of a LAN.

The network-layer transfer unit is a packet or datagram.

The link-layer transfer unit is a frame.

A frame encapsulates a network-layer packet by adding a link-layer header and usually a trailer.

The responsibility of the link layer is:

Transfer a frame from one node to a physically adjacent node over one link.

It does not directly provide end-to-end delivery across the entire Internet.

An end-to-end IP path can use a different link-layer protocol on every hop.

For example:

• Wi-Fi from a laptop to an access point;
• Ethernet through a local network;
• an optical provider link;
• PPP on another connection.

The IP packet can remain conceptually the same while being encapsulated in a new link-layer frame on every link.

Different link technologies may provide different services.

A wireless link, for example, normally has a higher bit-error rate than a fiber link and may therefore require stronger local error handling.

A journey from Saarbrücken to Sydney may involve:

• a bus;
• a train;
• an airplane;
• a taxi.

In this analogy:

• the traveler corresponds to the data being transported;
• each travel segment corresponds to one communication link;
• the transportation mode corresponds to the link-layer protocol;
• the travel agent corresponds to the routing algorithm that selects the overall path.

Each local transport mode only moves the traveler across one segment, while the complete sequence forms the end-to-end journey.

The link layer encapsulates a network-layer packet inside a frame.

It adds:

• a header;
• possibly a trailer;
• addressing information;
• error-detection information;
• protocol-specific control fields.

When multiple stations share one medium, the protocol must decide who may transmit and when.

Without a channel-access mechanism, simultaneous transmissions may interfere or collide.

The detailed multiple-access algorithms were postponed to the next lecture.

Frames commonly contain source and destination MAC addresses.

MAC addresses differ from IP addresses:

• IP addresses are hierarchical and support global routing;
• MAC addresses are primarily used for local delivery on a link or LAN;
• MAC addresses do not need to describe a host’s global network location.

MAC address ranges are allocated through the IEEE.

An address may be programmed into a network interface, but software can often change or spoof it.

Therefore, a MAC address alone should not be treated as a secure proof of identity.

Some link protocols can retransmit damaged or lost frames between adjacent nodes.

This is particularly useful on links with relatively high error rates, such as wireless links.

It is less commonly needed on low-error links such as optical fiber.

Link-level reliability does not replace end-to-end reliability.

Even if one link provides reliable delivery:

• another link on the path may not;
• a router may drop a packet because of congestion;
• a route may fail;
• the destination may be unavailable.

Only the end systems can verify that the complete end-to-end operation succeeded.

Link-layer flow control prevents one adjacent sender from overwhelming the receiving node or interface.

It controls the rate between the two endpoints of one link.

Error detection allows the receiver to recognize that a frame was damaged.

It may then:

• discard the frame;
• request retransmission;
• report an error.

Error correction allows the receiver to reconstruct some damaged bits without retransmission.

Some links additionally use Forward Error Correction, where the sender transmits redundant information that allows the receiver to correct certain errors.

In full-duplex communication, both endpoints can transmit at the same time.

In half-duplex communication, both endpoints can transmit, but not simultaneously.

The sending NIC:

1. receives a network-layer packet;
2. encapsulates it in a frame;
3. adds link-layer addresses;
4. adds error-detection bits;
5. applies link-level flow control or reliability if required;
6. converts the frame into physical signals.

The receiving NIC:

1. receives the physical signal;
2. reconstructs the frame;
3. checks for transmission errors;
4. performs link-level reliability and flow-control processing;
5. removes the link-layer header and trailer;
6. passes the enclosed packet to the network layer.

To detect transmission errors, the sender adds Error Detection and Correction bits, abbreviated EDC.

Let:

• \(D\) be the protected data;
• \(EDC\) be the redundant checking information.

The transmitted frame contains both:

\[ D \parallel EDC. \]

More redundancy generally provides stronger error detection or correction, but it consumes link capacity.

If a large fraction of every frame consisted of checking information, less capacity would remain for useful data.

Protocols therefore balance:

• detection strength;
• correction capability;
• computational cost;
• bandwidth overhead.

No finite error-detection code detects every possible corruption with absolute certainty.

Even a very small undetected-error probability matters when a network transmits enormous volumes of data.

A simple scheme adds one parity bit.

For even parity, the parity bit is chosen so that the total number of 1-bits is even.

A single flipped bit changes the parity and is therefore detected.

More generally, parity detects an odd number of bit flips but can miss an even number of bit flips.

It cannot determine which bit was corrupted and therefore cannot normally correct the error.

The data bits can be arranged in a matrix.

The sender adds:

• one parity bit for each row;
• one parity bit for each column.

If exactly one data bit changes:

• one row has incorrect parity;
• one column has incorrect parity;
• their intersection identifies the corrupted bit.

The receiver can therefore flip that bit back and correct a single-bit error.

More complicated combinations of errors may not be uniquely identifiable and can sometimes imitate another valid pattern.