CDN Anycast

At a high level, BGP performs a relatively simple sequence of operations:

1. Receive route advertisements from neighbors.
2. Filter unacceptable routes.
3. Select one best route for each prefix.
4. Install the selected route into the forwarding table.
5. Export selected routes to other neighbors according to export policy.

The protocol mechanism is simple, but the interaction between independently configured policies can produce complicated global behavior.

A simplified BGP route-selection sequence considers attributes such as:

• local preference;
• AS path length;
• Multi-Exit Discriminator (MED);
• additional implementation-specific tie breakers.

The most important point is that different autonomous systems independently define their own import preferences and export policies.

Consequently, the route that one AS selects changes the routes available to other ASes.

Consider an origin AS \(0\) and two neighboring autonomous systems \(1\) and \(2\).

Suppose:

• AS1 prefers the indirect path \(1 \rightarrow 2 \rightarrow 0\) over its direct path \(1 \rightarrow 0\);
• AS2 prefers the indirect path \(2 \rightarrow 1 \rightarrow 0\) over its direct path \(2 \rightarrow 0\).

Both ASes cannot simultaneously use their preferred indirect paths, since that would create a routing loop.

However, two stable outcomes are possible:

• AS1 uses the direct path and AS2 routes through AS1;
• AS2 uses the direct path and AS1 routes through AS2.

Which state is reached can depend on the timing and order of BGP updates.

Therefore, BGP can exhibit history dependence: the final routing state is not determined solely by the physical topology and configured policies.

Such preferences can be implemented, for example, through:

• local preference;
• BGP communities interpreted by import policies.

Suppose one of the stable states uses a high-capacity primary link and another uses a much slower backup link.

If the primary link fails, BGP moves to the state using the backup link.

After the primary link is restored, the network does not necessarily return to the original state. The currently selected routes and advertisements may form another stable solution, so part of the traffic can remain on the slow backup path.

Thus, link restoration does not imply that BGP returns to the same routing state that existed before the failure.

A stronger example uses three autonomous systems surrounding an origin AS.

Each AS has:

• a direct route to the origin;
• an indirect route through its neighboring AS;
• a policy that prefers the indirect route over the direct route.

For example:

• AS1 prefers \(1 \rightarrow 2 \rightarrow 0\);
• AS2 prefers \(2 \rightarrow 3 \rightarrow 0\);
• AS3 prefers \(3 \rightarrow 1 \rightarrow 0\).

When one AS selects its preferred indirect route, it may withdraw or invalidate a route needed by another AS.

This causes the other AS to change its route, which affects a third AS, which eventually causes the first AS to change again.

The system can repeatedly cycle through different routing states without ever reaching a stable solution.

BGP is not guaranteed to converge under arbitrary routing policies.

The problem is not a packet-forwarding loop in one fixed state. Rather, the selected routes themselves continually change.

Observed BGP advertisements are not sufficient to determine whether an oscillation is possible.

A route collector normally sees only routes that are currently selected and exported. It does not see:

• all alternative routes;
• private local-preference settings;
• complete import and export filters;
• routes that would become available after a failure;
• business agreements between autonomous systems.

A complete analysis would require the configuration and policy of every relevant BGP router.

Operators are generally unwilling to publish this information because routing policies can reveal commercially sensitive facts, such as:

• available capacity;
• preferred providers;
• backup arrangements;
• customer and peer relationships.

Even with complete information, deciding stability for arbitrary policies is a computationally difficult problem. In a large and continuously changing Internet, global verification is therefore impractical.

In a customer–provider relationship:

• the customer pays the provider;
• the provider gives the customer connectivity to the rest of the Internet;
• the provider carries traffic between the customer and external networks.

From the customer’s perspective, the provider offers transit.

A provider generally wants to carry customer traffic because the customer pays for the service.

In a peering relationship, two ASes agree to exchange traffic between their own networks and their customers.

Peering is often settlement-free: neither side pays the other for the exchanged traffic.

The motivation is economic. Without peering, both ASes might have to send the same traffic through paid transit providers.

A peer normally does not provide free transit between:

• another peer and a provider;
• two unrelated peers;
• a provider and another provider.

Doing so would consume resources without generating revenue.

Peering often takes place at Internet Exchange Points (IXPs), such as DE-CIX, AMS-IX, or LINX.

At an IXP:

• many networks place or rent routers at the same facility;
• the routers connect to a shared interconnection fabric;
• establishing additional peering sessions becomes relatively inexpensive.

Two large networks commonly interconnect at several geographical locations rather than through only one physical link.

European IXPs became particularly important because, historically, traffic between European networks could otherwise travel through providers in the United States and then return to Europe. Local peering reduced both cost and path length.

A Tier-1 AS does not buy transit from another AS.

To obtain global reachability, Tier-1 networks must peer with one another. Conceptually, the Tier-1 ASes therefore form a fully interconnected peering core.

Below them are customer ASes, which may connect to:

• one provider;
• multiple providers;
• providers and peers.

Commercial relationships produce the typical valley-free AS path structure:

1. zero or more customer-to-provider links, moving “uphill”;
2. optionally one peer-to-peer link;
3. zero or more provider-to-customer links, moving “downhill”.

Therefore, a valid commercial path does not normally contain:

• two peer-to-peer links;
• a provider-to-customer link followed by a customer-to-provider link;
• free transit from one peer to another peer.

The rough path shape is:

customer -> provider -> ... -> provider
                 [optional peer link]
provider -> customer -> ... -> customer

Once the path starts moving downward toward customers, it must not move upward again.

A common route preference is:

1. customer routes;
2. peer routes;
3. provider routes.

A customer route is attractive because carrying the traffic generates revenue.

A peer route avoids paid transit.

A provider route usually costs money.

The exact ordering can differ, but customer routes must generally be preferred over non-customer routes for the commercial-stability argument.

| Route learned from | Export to provider | Export to customer | Export to peer |
|--------------------+--------------------+--------------------+----------------|
| Own AS             | yes                | yes                | yes            |
| Customer           | yes                | yes                | yes            |
| Sibling            | yes                | yes                | yes            |
| Provider           | no                 | yes                | no             |
| Peer               | no                 | yes                | no             |

The central rule is:

Routes learned from providers or peers are exported only to customers.

Exporting such a route to another provider or peer would mean offering unpaid transit.

Customer routes can be exported to everyone because carrying traffic toward a customer is part of the paid customer service.

BGP communities are frequently used to mark where a route was learned, for example:

• customer route;
• peer route;
• provider route;
• internal or sibling route.

Export filters can then make decisions based on these markings.

The Gao–Rexford model restricts routing policies according to commercial relationships.

The main ideas are:

• the customer–provider hierarchy must not contain cycles;
• customer routes are preferred over routes learned from peers or providers;
• customer routes may be exported broadly;
• peer and provider routes are exported only to customers.

These are local restrictions: each AS needs to configure only its own policy.

When the commercial conditions are followed, the resulting global routing system has a stable solution and avoids the arbitrary-policy oscillations shown earlier.

This helps explain why serious BGP oscillations are possible in theory but relatively uncommon in the operational Internet.

Traditional BGP does not cryptographically prove that:

• an AS owns the prefix it originates;
• an AS is authorized to originate that prefix;
• the AS path in an update is genuine;
• the announced route follows real business relationships.

A malicious or misconfigured AS can therefore announce a prefix belonging to another network.

Other ASes may accept the route and redirect traffic toward the false origin.

This is called a prefix hijack.

Hijacks may be:

• intentional attacks;
• accidental configuration errors;
• typographical or “fat-finger” mistakes.

Because each AS applies its own policy, an equally specific false announcement may attract only part of the Internet.

IP forwarding uses longest-prefix matching.

Suppose the legitimate network announces:

12.0.0.0/8

An attacker may announce more-specific subprefixes, such as:

12.0.0.0/9
12.128.0.0/9

Together, the two /9 prefixes cover the entire /8.

Because the /9 routes are more specific, routers prefer them during forwarding, even if the BGP route to the legitimate /8 otherwise has a better AS path.

A more-specific hijack can therefore capture substantially more traffic than an equally specific competing announcement.

The first defense is to reject routes that should not have been announced.

Filtering cannot be perfect because:

• AS relationships are not globally known;
• routing policies are private;
• prefix ownership changes;
• routing databases can be incomplete or outdated.

Nevertheless, operators should filter as accurately as the available information permits.

Static filters reject categories that should normally never be accepted.

Examples include:

• documentation prefixes;
• reserved and special-purpose address ranges;
• currently unallocated prefixes;
• prefixes owned by the receiving AS itself but announced by another AS;
• the default route when it is not expected from that neighbor;
• implausibly specific or implausibly broad prefixes;
• invalid or reserved AS numbers;
• routes whose AS_PATH already contains the local AS.

These lists must be updated as address allocations and operational practices change.

Internet Routing Registries (IRRs) contain routing-policy information submitted by network operators.

Operators can publish Routing Policy Specification Language objects describing:

• which prefixes an AS originates;
• which ASes are its customers;
• which routes may be announced through a particular relationship.

An AS-SET contains:

• AS numbers;
• other nested AS-SETs.

For example, a provider can use an AS-SET to describe its customer cone. Another operator can generate prefix and AS-path filters from that information.

Dynamic IRR-based filtering works particularly well for customer connections.

A provider can require a customer to register its prefixes before accepting them.

It is more difficult for peers because a peer may legitimately announce routes covering a large part of the Internet.

IRR data also has important quality problems:

• records may be missing;
• records may be outdated;
• different registries may disagree;
• operators may fail to maintain their entries.

Consequently, automatically generated filters cannot be trusted blindly.

Secure Inter-Domain Routing (SIDR) is a broader framework for securing global routing.

The Resource Public Key Infrastructure (RPKI) is one important component of SIDR and is often used as the name for the practical deployment.

Two distinct goals are:

• route-origin validation: verify that an AS is authorized to originate a prefix;
• path validation: verify information about the complete AS path.

Origin validation is much more widely deployed than full path validation.

A Route Origin Authorization (ROA) states that a particular AS is authorized to originate a prefix.

A ROA contains:

• an IP prefix;
• an origin AS number;
• a maximum permitted prefix length.

For example, a ROA might authorize AS64499 to originate:

2001:db8::/47

and allow announcements no more specific than /48.

The maximum length is important. It prevents the origin AS, or someone claiming to be that origin, from announcing arbitrary more-specific prefixes beyond what the owner authorized.

The ROA is digitally signed using the resource certificate associated with the prefix owner.

A resource certificate binds a public key to Internet number resources, such as:

• IP address ranges;
• autonomous system numbers.

The certification hierarchy follows the allocation hierarchy, conceptually:

IANA
  -> Regional Internet Registries
     -> National or Local Internet Registries
        -> network operators

For Europe, the relevant Regional Internet Registry is RIPE NCC.

RPKI repositories or publication points distribute ROAs and related signed objects. They may be synchronized using mechanisms such as rsync or RRDP over HTTPS.

A relying-party validator:

1. downloads objects from the RPKI repositories;
2. verifies certificate chains and signatures;
3. checks the validity of ROAs;
4. derives validated prefix-origin information;
5. makes the result available to routers.

A received BGP route is classified as:

• valid: a matching ROA authorizes the origin AS and prefix length;
• unknown: no relevant ROA exists;
• invalid: an existing ROA contradicts the announcement.

The final BGP decision is still policy-based.

A common policy is to reject invalid routes while continuing to accept unknown routes, because many prefixes may not yet have ROAs.

The RPKI infrastructure also uses objects such as:

• Manifest (MFT): a signed list of the objects published by a certification authority, helping detect missing or modified files;
• Certificate Revocation List (CRL): a list of certificates that are no longer valid;
• Ghostbuster Record: signed contact information that allows operators to contact the responsible certification authority during an incident or dispute.

Ideally, every AS should reject an RPKI-invalid route.

However, filtering is especially important close to the source of the invalid announcement.

If the first upstream provider accepts and propagates the false route, the announcement can spread widely before another AS filters it.

Therefore:

All ASes should filter, but the immediate upstream provider is the most important first line of defense.

Origin validation verifies only the final origin AS in the AS_PATH.

An attacker may construct an announcement whose path ends with the legitimate origin AS, even though the attacker has inserted a false relationship earlier in the path.

The origin may therefore appear valid while the path itself is fabricated.

Consequently, ROV prevents many prefix-origin hijacks but does not authenticate the complete AS path.

Autonomous System Provider Authorization (ASPA) introduces additional signed RPKI objects describing provider relationships.

These objects can be used to validate portions of an AS_PATH and detect some forms of:

• forged paths;
• route leaks;
• impossible customer–provider transitions.

Path validation is more difficult to deploy because it requires operators to publish information about their business relationships.

Engineers may see the security benefit, while business departments may oppose revealing commercially sensitive relationships.

The lecturer gave rough, time-dependent deployment estimates indicating that origin validation had reached a substantial fraction of the Internet, while path-validation deployment remained much lower.

Additional routing hygiene includes:

• route flap damping: repeatedly changing routes receive penalties and may be temporarily suppressed;
• maximum-prefix limits: a session is limited to an expected number of routes, preventing an accidental full-table leak;
• AS_PATH checks: reject loops, private AS numbers in inappropriate locations, and other implausible paths;
• prefix-count and prefix-length checks: prevent a neighbor from unexpectedly announcing huge numbers of more-specific routes.

The overall goal is to stop local mistakes from becoming global Internet incidents.

A central computer-science principle is that many problems can be addressed by adding another level of indirection.

Instead of forcing every user to contact one origin server directly, a content provider can place an intermediary system between users and the origin.

The intermediary selects and serves a suitable nearby copy of the content.

CDNs apply this principle to Internet content delivery.

Internet content includes:

• web pages and documents;
• images;
• software and other binary files;
• audio and video;
• live and on-demand streaming.

Historical measurements illustrate how the dominant applications changed:

• before 2000, traditional web traffic formed a large fraction of traffic;
• around 2002, peer-to-peer applications accounted for more than half of some measured traffic;
• by 2009, HTTP again dominated, with video responsible for a large part of HTTP traffic.

Although the application mix changes, the overall amount of Internet content and traffic continues to grow.

The central problem is therefore:

How can popular content be delivered efficiently to extremely large numbers of geographically distributed users?

A single large origin server does not scale well.

A Content Distribution Network stores copies of content at many geographically distributed sites.

A user is directed to a server that is reasonably close or otherwise suitable.

This provides:

• shorter paths;
• lower latency;
• greater aggregate capacity;
• reduced origin-server load;
• resilience against site and link failures;
• better handling of flash crowds.

The fundamental design questions are:

• Where should CDN servers be placed?
• Which content should each server store?
• Which server should handle a particular request?
• At which encoding rate should content be delivered?

Early CDN deployments demonstrated their value during major flash-crowd events, including the enormous demand for news after the September 11 attacks.

Origin sites struggled even after simplifying their pages, while distributed infrastructure could serve the same information from many locations.

This helped establish CDNs as a central component of Internet content delivery.

A load balancer normally distributes requests among servers inside one data center or one local server cluster.

A CDN operates at a much larger scale:

• servers are geographically distributed;
• servers may be located inside many different access networks;
• traffic is directed across the global Internet;
• the CDN monitors network conditions and server health;
• the CDN handles failures across many sites.

A CDN may use load balancers internally, but it is not merely a load balancer.

Akamai, whose name means “intelligent” in Hawaiian, developed from an MIT research effort aimed at handling flash crowds.

The 2023 figures shown in the lecture included approximately:

• 360,000 servers;
• more than 4,200 locations;
• more than 1,350 networks;
• more than 840 cities;
• 135 countries;
• hundreds of terabits per second of peak delivery capacity.

For content consumers, the benefit is faster and more reliable downloads.

For content providers, the benefits include:

• reduced infrastructure cost;
• simpler global deployment;
• protection against flash crowds;
• less dependence on one data center or ISP.

The CDN does not need to create the content. Its task is to distribute and serve content created by its customers.

A content provider can divide a service into:

• cacheable static content;
• dynamic or user-specific content.

Cacheable objects commonly include:

• images;
• video segments;
• JavaScript and style sheets;
• software downloads;
• relatively static HTML fragments.

Dynamic content may remain at the origin or be assembled from fragments at the CDN edge.

For example, the origin may generate a small personalized HTML document while the much larger images, scripts, and video objects are served by the CDN.

The lecturer gave an illustrative case in which only a small fraction of the bytes comes from the origin while almost all transferred data comes from the CDN.

A Time To Live (TTL) determines how long a cached object may be reused:

• rapidly changing content receives a short TTL;
• stable content can receive a much longer TTL.

Consider a user requesting an object under:

www.example.com

A typical CDN request process is:

1. The user loads a web page associated with example.com.
2. The user’s recursive DNS resolver queries the authoritative DNS server for example.com.

3. The authoritative server returns a CNAME pointing to a CDN-controlled name, for example:

   www.example.com CNAME www.example.cdn.net
   

4. The recursive resolver queries the CDN’s authoritative DNS infrastructure.
5. The CDN applies its server-selection algorithm and returns the address of a suitable CDN server.
6. The client connects to that server and fetches the object.

The CNAME introduces a level of indirection: the content provider delegates the final server-selection decision to the CDN.

The geographically closest server is not always the best server.

The CDN must consider several factors.

In traditional DNS-based mapping, the CDN’s authoritative name server normally sees the address of the client’s recursive resolver rather than the end host.

The CDN may use the resolver’s address as an approximation of the client’s location.

This works well when the resolver is near its users, but can produce poor mapping when:

• the user employs a distant public resolver;
• one resolver serves users in many regions;
• the resolver and user have very different network paths.

The detailed DNS solution to this problem, EDNS Client Subnet, was deferred to the next lecture.

Netflix stores copies of video content in its worldwide Open Connect CDN.

A simplified request process is:

1. The subscriber requests a title.
2. The service returns a manifest.
3. The manifest describes available copies, encodings, and bit rates.
4. The client selects an appropriate CDN node and representation.
5. The client retrieves the content at the highest currently supportable rate.
6. If conditions change, the client can switch to another bit rate or copy.

This combines CDN server selection with adaptive streaming.

A congested path may cause the client to choose:

• a lower-rate encoding;
• another CDN server;
• another copy of the same object.

A CDN cannot store every object at every server because edge servers have limited storage capacity.

Therefore:

• highly popular objects are replicated at many locations;
• moderately popular objects are stored at fewer locations;
• rarely requested objects may remain only at regional sites or the origin.

Cache-replacement and popularity-estimation algorithms decide how the limited capacity is used.

CDNs are over-the-top systems.

They are not part of the basic IP forwarding service.

They use ordinary Internet host-to-host communication and build content placement, monitoring, redirection, and adaptation above it.

The underlying Internet offers connectivity, while the CDN must cope with congestion and failures from the network edge.

An anycast address appears to the client like a normal unicast IP address.

The key difference is in routing configuration.

The same IP prefix is announced through BGP from multiple geographical locations.

For example:

• one site announces the prefix in Europe;
• another announces it in North America;
• another announces it in Asia.

Each network selects its preferred BGP route to that prefix.

As a result, users in different parts of the Internet reach different physical servers while using exactly the same destination IP address.

Anycast uses BGP route selection as a distributed server-selection mechanism.

BGP does not necessarily choose the geographically nearest server or the one with the lowest measured RTT.

The selected site depends on attributes such as:

• local preference;
• customer, peer, and provider relationships;
• AS-path length;
• BGP communities;
• MED;
• internal routing cost;
• route advertisements and withdrawals.

Therefore, “nearest” means nearest according to the effective routing policy.

This is one reason traceroutes to an anycast address can reach different locations from different networks.

Anycast provides:

• one stable service address;
• automatic distribution of users among sites;
• short routing paths in many cases;
• resilience when one site withdraws its route;
• no need for the client to know which replica it contacts.

It is widely used by globally distributed infrastructure services.

Well-known anycast DNS services include:

• Google Public DNS: 8.8.8.8;
• Cloudflare DNS: 1.1.1.1;
• Quad9: 9.9.9.9.

These addresses are announced from many locations.

A client normally reaches a nearby instance according to BGP routing.

The lecture showed that traceroutes and RTTs to these addresses differ because the services have different deployments and routing relationships.

At the time represented by the slides, Quad9 operated 228 anycast replicas.

Anycast performs coarse traffic distribution according to routing.

However, BGP does not automatically know:

• real-time server CPU utilization;
• current application-level queue length;
• which server contains a particular object;
• which server would provide the best video throughput.

Consequently, basic anycast cannot perform precise load-aware selection.

Operators can manipulate announcements or withdraw a site, but these are much coarser mechanisms than application-level load balancing.

A client’s selected anycast site can change if BGP routing changes.

For a connectionless request, this is usually acceptable. A later request can reach another replica without depending on previous server state.

For a stateful transport connection, a change is more problematic.

For example:

1. a TCP connection is established with replica A;
2. the route changes;
3. later packets reach replica B;
4. replica B has no TCP state for the connection;
5. the connection fails.

Therefore, connectionless services are especially well suited to anycast.

DNS is a natural use case because many DNS exchanges are short UDP request-response interactions.

Historically, operators were cautious about running TCP services over anycast.

Today, routing is often stable enough that HTTP and other TCP-based anycast services are common, but a routing change can still break an existing connection unless state is shared or another migration mechanism is used.

A CDN can use two broad mechanisms to direct a client to a replica:

• DNS-based server selection;
• BGP-based IP anycast.

DNS-based selection can consider more application and server information, but must infer the client location and is affected by DNS caching.

Anycast is simpler from the client’s perspective and automatically follows BGP reachability, but offers less precise control over server load and individual client assignment.

The two approaches can also be combined.