Limiting the Network Bandwidth of a Linux-Based Amazon Lightsail or EC2 Instance

Figure showing an EC2 instance that has wide ingress bandwidth and narrow egress bandwidth.

Last updated: June 12, 2019

The applied rate for data transfer OUT from a Lightsail or EC2 instance to the Internet depends on the AWS region and varies from 90 USD to 170 USD per 1 TB.

For a Lightsail instance, both data transfer IN and data transfer OUT count toward the data transfer allowance of this instance. You can not control what is sent from the Internet to your Lightsail instance. That means, your transfer allowance can be exceeded as a result of multiple malicious IN requests that your instance probably even does not reply. However, if the data transfer allowance is exceeded, charges apply only for data transfer OUT from your Lightsail instance to the Internet (or to another AWS region or to AWS resources in the same region when the public IP address is used).

Amazon EC2 instances do not have a data transfer allowance. Different rates apply only to the data sent OUT from an EC2 instance. Data transfer IN is always free. Also, you can protect your EC2 instances with AWS Shield and AWS WAF, a web application firewall.

This blog post describes how to limit the outbound network bandwidth of your Linux-based Lightsail or EC2 instance and, thus, make the charges predictable.

How to Limit the Network Bandwidth of a Linux-Based Amazon Lightsail or EC2 Instance Using tc

In this procedure, you configure the Token Bucket Filter (TBF) which is a classless queueing discipline (qdisc). It does not just slow down the outbound traffic to the specified fixed rate, but also allows bursts of data to be sent at a higher rate if there are accumulated tokens available in the bucket.

For more information about traffic control in Linux and the TBF, see Overview of Linux Traffic Control and the TBF.

To limit the outbound network bandwidth, perform the following steps:

  1. Define the baseline rate.
    For example, if you want to make sure that data transfer OUT from your Lightsail or EC2 instance is not more than 1 TB per month, divide 1000000000 KB by 31 * 24 * 3600 (days in a month * hours in a day * seconds in an hour). That gives you the rate of 373 KB/second.
  2. Define the token bucket size (burst).
    If you do not transfer data all the time at the baseline rate, the unused tokens are accumulated in the bucket. Later, the accumulated tokens can be used to send bursts of data at a higher rate than the baseline rate.
    • The token bucket is a virtual object. It does not consume system memory or disk space so that you can set the bucket size to a very high value. For example, you can set it to 1000 MB.
    • If you do not want the transfer rate ever to exceed the defined baseline rate, set the bucket size to the MTU value of the corresponding network interface.
  3. Define the latency.
    This is the maximum amount of time a packet is allowed to wait in the queue for available tokens until it is dropped by kernel.
    For example, if you run on your Lightsail or EC2 instance a web server, the reasonable latency is 100 ms.
  4. Set up traffic control using the tc command-line utility.
    For example, to limit the outbound bandwidth of the eth0 interface, execute the following command:
    tc qdisc add dev eth0 root tbf rate 373kbps burst 1000mb latency 100ms
    
    • The TBF is a classless qdisc. It can only be attached at the root of a device.
  5. Check the statistics of sent and dropped packets.
    For example, for the eth0 interface, execute the following command:
    tc -s qdisc show dev eth0
    

When the token bucket is created, it is filled with tokens.
In the above example, the token bucket is filled in the beginning with tokens that correspond to 1000 MB of data. That means you can download the first 1000 MB of data at the maximum rate. Then, when the bucket is empty, the outbound rate is limited to 373 KB/second.

If you do not transmit any data, the token bucket starts accumulating tokens with a speed of 373 KB/second. For example, if the outbound bandwidth is not used for three hours, a big enough token bucket will accumulate the number of tokens that approximately corresponds to 4000 MB of data (373 * 3 * 3600).

In our case, the bucket size is only 1000 MB, so if the outbound bandwidth is not used for three hours, the token bucket will accumulate not 4000 MB but only 1000 MB. Then, 1000 MB of data can be again downloaded at the maximum rate.

  • You can use the iptraf utility to monitor the network bandwidth on your instance.

Appendix A, Overview of Linux Traffic Control and the TBF

The command-line utility tc is used to configure traffic control in the Linux kernel. The processing of traffic is controlled by queuing disciplines (qdisc), classes, and filters.

In the simplest case, traffic control is classless and consists of a single queue that implements pure FIFO (First In, First Out) queuing discipline. In this case, entering packets are collected and then dequeued as quickly as the network interface can accept them.

The default queuing discipline for all network interfaces under Linux is pfifo_fast, which is based on the conventional FIFO algorithm. You can verify it, for example, for the eth0 interface, as follows:

tc qdisc show dev eth0

The Token Bucket Filter (TBF) is a classless qdisc. The following diagram and text describe the main principles of the TBF.

Flow diagram showing the token bucket and the network packets queue, which is FIFO. Network packets are transmitted only when tokens are available in the bucket.

  • The TBF has a baseline rate.
    The TBF transfers packets at a rate that does not exceed the baseline rate but allows bursts in excess of this rate.
  • Tokens in the bucket are replenished at the baseline rate.
    To avoid packet loss, the minimum bucket size on modern tickless Linux kernels must be equal or greater than the MTU of the corresponding network interface.
    On old Linux kernels, each timer tick only rate / HZ tokens are replenished in the bucket. Thus, the minimum bucket size must also be equal to this value. Otherwise, some newly generated tokens are immediately discarded.
  • In order to transmit a packet, this packet must obtain tokens equal to its size in bytes.
  • The token bucket accumulates unused tokens when the actual transfer rate is below the baseline rate. The token bucket has a limited size in bytes which is called burst. The accumulated tokens can be used to send bursts of data at a higher rate than the baseline rate.
  • If there are no available tokens in the bucket, the packets build up a queue and consume kernel memory. You must limit the size of this queue by specifying its size in bytes (limit) or the maximum amount of time a packet can wait in this queue (latency).
  • When the packet queue is full Linux kernel starts dropping the new packets.