BGP Convergence Optimization

A large multi-homed content provider has experienced a number of outages and brownouts in the Internet edge of their data center network. The brownouts were caused by high CPU load on the Internet edge routers, leading to unstable forwarding tables and packet loss after EBGP peering session loss.

This document describes the steps the customer could take to improve the BGP convergence and reduce the duration of Internet connectivity brownouts.

The document describes a summary of design challenges sent by readers of ipSpace.net blog and discussed in numerous ExpertExpress engagements. It’s based on real-life queries and network designs but does not represent an actual customer network. Complete document is available as downloadable PDF to ipSpace.net subscribers. You can also buy a digital book with all ExpertExpress case studies  

Brief Network Description

The customer’s data center has two Internet-facing edge routers, each of them connected to a different ISP through a 1GE uplink. Both routers are dual-attached to core switches (see Figure 1). ISP-A is the primary ISP; connection to ISP-B is used only when the uplink to ISP-A fails.

Figure 1: Network core and Internet edge

Edge routers (GW-A and GW-B) have EBGP sessions with ISPs and receive full Internet routing (~450.000 BGP prefixes[1]). GW-A and GW-B exchange BGP routes over an IBGP session to ensure consistent forwarding behavior. GW-A has higher default local preference; GW-B thus always prefers IBGP routes received from GW-A over EBGP routes.

Core routers (Core-1 and Core-2) don’t run BGP; they run OSPF with GW-A and GW-B, and receive default route from both Internet edge routes (the details of default route origination are out of scope).

Solution Overview

The temporary blackout and prolonged brownouts following an Internet uplink loss are caused by BGP convergence issues. Like with any other routing protocol, a router running BGP has to take the following steps to adapt the forwarding tables to link or neighbor loss:

  1. BGP routing process detects a link or neighbor loss.
  2. Invalid routes are removed from the local BGP, routing and forwarding tables. Alternate routes already present in BGP table could be installed at this point.
  3. Updates are sent to other BGP neighbors withdrawing the lost routes.
  4. BGP neighbors process the withdrawal updates, select alternate BGP best routes, and install them in their routing and forwarding tables.
  5. BGP neighbors advertise their new best routes.
  6. The router processes incoming BGP updates, selects new best routes, and installs them in routing and forwarding tables.

Neighbor loss detection can be improved with Bidirectional Forwarding Detection (BFD)[2], fast neighbor failover[3] or BGP next-hop tracking. BGP update propagation can be fine-tuned with BGP update timers. The other elements of the BGP convergence process are harder to tune; they depend primarily on the processing power of routers’ CPU, and the underlying packet forwarding hardware.

Some router vendors offer functionality that can be used to pre-install backup paths in BGP tables (BGP best external paths) and forwarding tables (BGP Prefix Independent Convergence[4]). These features can be used to redirect the traffic to the backup Internet connection even before the BGP convergence process is complete.

Alternatively, you can significantly reduce the CPU load of the Internet edge routers, and improve the BGP convergence time, by reducing the number of BGP prefixes accepted from the upstream ISPs.

Finally, you might need to replace your Internet edge routers with devices that have processing power matching today’s Internet routing table sizes.

Notes

  1. BGP Routing Table Analysis Reports
    http://bgp.potaroo.net/
  2. Bidirectional Forwarding Detection
    http://wiki.nil.com/Bidirectional_Forwarding_Detection_(BFD)
  3. Fast BGP Neighbor Loss Detection
    http://wiki.nil.com/Fast_BGP_neighbor_loss_detection
  4. Prefix Independent Convergence – Fixing the FIB Bottleneck
    http://blog.ioshints.info/2012/01/prefix-independent-convergence-pic.html

Get the complete document

Complete case study, including design and deployment guidelines and sample configuration snippets is available to ipSpace.net subscribers. Select the Case studies tab after logging into the webinar management system.

You can also buy a digital book with all ExpertExpress case studies.