Load Balancers: A Deep Dive

Photo by Taylor Vick

Table of contents:

1. What is load balancing
2. Why do we need load balancers
3. Load balancers classification
4. Where does the load balancers fit in today’s systems
5. Load balancing algorithms
6. Defining a load balancer in modern cloud infrastructures
7. Summary

1) What is Load Balancing ?

In the computing world, ‘load balancing’ refers to the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient.

Load balancers help to spread the traffic across a cluster of servers to improve the responsiveness and availability of systems, whether it may be applications, websites or databases etc.

2) Why do we need Load Balancers ? 

In today’s systems, meaning truly distributed systems, it is a common practice to have multiple sets of backend servers, across availability domains/zones, regions in cloud infrastructure.

a) Load balancer acting as a reverse proxy: We don’t want to expose the IPs of backend servers, so we employ load balancers which acts as a “public face” of backend servers, thus providing a Naming abstraction.

b) As a traffic cop: Load balancers also act as a “traffic cop”, by efficiently routing requests/network load to healthy servers backend server pool, sometimes enforcing security rules. Thus providing fault tolerance and high availability.

c) Scalability and Efficiency: It is possible to add/remove servers to the server pool dynamically according to load/traffic since the load balancer effectively takes requests and route them, thus providing scalability and efficiency

d) Health checks: Load balancers do continuous health checks on each of the backend servers, to ensure the requests are only served the healthy servers. The status of the backend server can be (OK, Warn, Critical, Active-Impaired, Unresponsive, etc.) while doing health checks.

There are 2 Main Types of Health Checks:

• Shallow health checks: It involves pinging each of backend servers to make sure they are up and running (available).
• Deep health checks: It Involves checking the health and state of application’s dependencies in-turn, so that the backend server is actually capable of serving the client requests.

e) Service discovery: Load balancers discover devices and services available on that network by talking to the service registry, which is essentially a database that contains the network locations of service instances.

f) Caching and Compression: The load balancer can compress the backend server’s response before returning it to the clients thus saving network bandwidth and can also cache these responses to reduce the load on the backend systems.

g) Incoming request’s decryption: Incase of HTTPS/SSL listener attached to your load balancer, a certificate(SSL server certificate (X.509) enables the load balancer to terminate the connection and decrypt incoming requests before passing them to the backend servers.

3) Load Balancers Classification

Load balancers can be classified based on various criteria.

I) Software Load Balancers Vs Hardware Load Balancers

Traditional load balancing solutions rely on proprietary hardware housed in a data center, and require a team of sophisticated IT personnel to install, tune, and maintain the system. Only large companies with big IT budgets can reap the benefits of improved performance and reliability. In the age of cloud computing, hardware‑based solutions have another serious drawback: they do not support cloud load balancing, because cloud infrastructure vendors typically do not allow customers or proprietary hardware in their environment.

Fortunately, software‑based load balancers can deliver the performance and reliability benefits of hardware‑based solutions at a much lower cost. Because they run on commodity hardware, they are affordable even for smaller companies. And they are ideal for cloud load balancing, as they can run in the cloud like any other software application.

II) Layer 4 Load Balancers Vs Layer 7 Load Balancers

Layer 4 load balancer is often a dedicated hardware device that runs load balancing software. In layer 4 load balancing, load balancer’s public IP is advertised to the clients via DNS service(for a website or service). Layer 4 load balancing usually relies on source and destination IP addresses to make load balancing decisions. It employs NAT(Network address translation) to alter the destination server IP address from its own IP to the IP address of one of the healthy backend servers. Usually, layer 4 load balancers are reachable via TCP, UPD, TLS.

It uses only the first few packets of the request in the TCP stream. Layer 4 load balancing has become obsolete as CPU and memory have become cheap enough to overlook the small performance benefits in layer 4.

Layer 7 load balancer is related to seventh layer of the OSI model, application level and makes use of actual content of the message/request to make more intelligent and informed load balancing decisions. For example, the language indicated in the browser header can be used to redirect visitors to the appropriate content version. The layer 7 load-balancer acts as a proxy, which means it maintains two TCP connections: one with the client and one with the server. Usually layer 7 load balancers are reachable via TCP, SSL/TLS, HTTP, HTTPS.

Some load balancers can be configured to act both as layer 4 and layer 7 load balancers depending on the services and requests.

III) Private Load Balancers Vs Public Load Balancers

Private load balancers have private IP assigned from local private network subnet. This type of load balancer is usually employed to handle traffic with in a private network(VCN, Private cloud), between private networks(between availability zones/domains), between regions, and the network traffic from on-premise cloud.

Public load balancers have public IP address and can be reachable from the Internet. This is usually employed in customer facing applications and systems. for example in facebook, google, financial websites etc.

photo courtesy: NGINX

4) Where does the Load Balancers fit in today's systems

It is a good practice to have two load balancers one as active and another one as standby, to make sure the load balancer doesn’t itself becomes the single point of failure.

We employ floating IP in this case, which is a kind of Virtual IP which can be assigned to multiple servers (load balancers here) at the same time, but it can only be active on one server at any given time.

We can use this floating IP to implement failover, If at all the active(primary) load balancer goes down(becomes unavailable). In this case, the requests will be dynamically routed to the IP address (Anchor IP address) of standby load balancer(secondary).

Typically, Load balancers can be placed:

• Between the user and the web server
• Between web servers and application servers/cache servers
• Between application servers and database

5) Load Balancing Algorithms

There are tens of load balancing algorithms. I am listing few popular ones here:

• Round Robin: Requests are distributed across the group of servers sequentially.
• IP Hash: The IP address of the client along with the particular hashing algorithm is used to determine which server receives the request. This mechanism is used, whenever there is a need that requests from the same user/client to be served by the same backend server every time.
• Least Connections: A new request is sent to the server with the fewest current connections to clients. The relative computing capacity of each server is factored into determining which one has the least connections.
• Least Time: Sends requests to the server selected by a formula that combines the fastest response time and fewest active connections.
• Resource based: combines knowledge from OSI layers 3, 4 and 7 to make more optimized traffic distribution decisions.

6) Defining a Load Balancer in Modern Cloud Infrastructures

To define a load balancer in modern cloud infrastructures, we need the following (physical and logical) entities with servers being the only set of physical entities.

• Two servers (one active, another standby) on which the load balancer software runs with listener’s (ports) defined on them (to check incoming traffic on load balancer’s IP address)
• Virtual Network (VCN/Private cloud network, subnets)
• Health check policy
• Load balancing algorithm(s)
• Optional SSL handling
• Floating IP (to failover, without downtime)

7) Summary

Load balancer has essentially become an important component of every distributed, fault tolerant and high availability systems. It can act as a reverse proxy and can enforce caching and compression techniques too. Load balancer cannot be overlooked because of the ever increasing load and network traffic in todays “active users” hungry applications. They improve the systems responsiveness and overall performance, by distributing traffic based on various algorithms and mechanisms.

Interesting Reads:
1. Futures in Java: CompletableFutures
2. Java Lombok: do we need getters and setters

Happy learning.!!

Resources:

1. https://en.wikipedia.org/wiki/Load_balancing_(computing)
2. https://www.nginx.com/resources/glossary/load-balancing
3. https://docs.oracle.com/en-us/iaas/Content
4. https://www.haproxy.com/blog/loadbalancing-faq
5. https://kb.leaseweb.com/network/floating-ips/using-floating-ips