Reminders

• Final project proposals due next week

Agenda

1. The Journey of a Request
2. Reverse Proxies and Load Balancers
3. nginx Configuration and Routing
4. Internal Traffic Patterns

Why do we need Routing?

Say you've deployed your application on a server somewhere on AWS. Now what?

• How does a user's browser find your server?
• What if you have multiple servers?
• How do you handle thousands of concurrent requests?
• What if one server goes down?

DNS Review

The "phonebook of the internet"

• User types www.example.com in browser
• DNS (Domain Name System) translates domain to IP address
• DNS server returns IP: 123.123.123.123
• Browser now knows where to send the request

Problem: what IP do we use for our domain? What if we just used the IP of a single backend machine?

Problems with Single Server

• All traffic goes to one machine
• No redundancy
• Limited by one machine's capacity
• Single point of failure

Load Balancing

We can still only use a single IP, but we have some use cases to address:

• You might have multiple backend servers
• You need to decide which server handles this request
• You want to check if servers are healthy
• You may want to terminate SSL/TLS here

We need a reverse proxy / load balancer

Forward Proxy vs Reverse Proxy

What is a Reverse Proxy?

• Reverse proxy: Server that sits between clients and backend servers
• Receives all incoming requests and determines where to send them
• Forwards requests to backend servers
• Returns backend responses to clients

From the client's perspective, they're talking directly to your application

Why Use a Reverse Proxy?

• Load balancing: Distribute traffic across multiple servers
• SSL/TLS termination: Handle HTTPS encryption/decryption in one place
• Security: Hide backend server details, add rate limiting
• Unified entry point: Single place to route different services
• Performance: Cache and compress responses

Load Balancing Algorithms

There are various ways to determine how to load balance:

• Round robin: Each server gets requests in turn (A, B, C, A, B, C...)
• Least connections: Send to server with fewest active connections
• Hash-based: Same client IP always goes to same server
• Weighted: Some servers handle more traffic than others
• Random: Pick a server randomly

Health Checks

What if a backend server crashes?

• Health checks: Periodic checks to see if servers are responsive
• Reverse proxy pings each backend (e.g., GET /health every 5 seconds)
• If a server fails health check, stop sending traffic to it
• When it recovers, add it back to the pool

Health checks are also used in Kubernetes to determine if a pod is erroring.

What is nginx?

• nginx: Open-source web server and reverse proxy
• Pronounced "engine-x"
• Created to solve the C10K problem (10,000 concurrent connections)
• Known for high performance and low resource usage
• Most popular reverse-proxy choice

Basic nginx Concepts

• Server block: Configuration for a virtual host
• Location block: How to handle requests with specific URL paths
• Upstream: Group of backend servers
• Proxy pass: Forward request to backend

Simple nginx Configuration

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Load Balancing with nginx

Define an upstream group:

upstream backend {
    server 10.0.1.10:3000;
    server 10.0.1.11:3000;
    server 10.0.1.12:3000;
}

server {
    listen 80;

    location / {
        proxy_pass http://backend;
    }
}

Health Checks in nginx

upstream backend {
    server 10.0.1.10:3000 max_fails=3 fail_timeout=30s;
    server 10.0.1.11:3000 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:3000 max_fails=3 fail_timeout=30s;
}

• max_fails=3: Mark server as down after 3 failed requests
• fail_timeout=30s: Wait 30 seconds before trying again

Routing Different Paths

server {
    listen 80;

    location /api/ {
        proxy_pass http://api-backend;
    }

    location /static/ {
        root /var/www;
    }

    location / {
        proxy_pass http://frontend-backend;
    }
}

External Traffic to Kubernetes

You have services running in Kubernetes pods. How does external traffic reach them?

• Pods have internal IPs that change constantly
• Services provide stable internal endpoints (e.g. http://auth-service), but are only accessible by pods within the cluster
• But how do external users reach your cluster?
• How do you route different domains/paths to different services?

Idea: nginx in Kubernetes

• We deploy nginx pods in our cluster as a deployment
• This allows us to leverage internal kubernetes names and IP mapping!
• We can easily define upstreams like http://auth-service and route traffice accordingly

External Traffic Flow

1. DNS: api.example.com → Load Balancer IP
2. Cloud Load Balancer: AWS NLB (created by Kubernetes)
3. LoadBalancer Service: Routes traffic to nginx pods
4. nginx Deployment: nginx pods with routing configuration
5. Backend Services: Backend application services (api, frontend, etc.)

https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

Kubernetes LoadBalancer Service

How do we expose nginx to the internet?

• LoadBalancer Service: Kubernetes resource that provisions cloud load balancer
• When you create it, Kubernetes automatically creates AWS NLB
• NLB gets a public IP that DNS can point to
• NLB forwards traffic to nginx pods in the cluster

LoadBalancer Service Example

apiVersion: v1
kind: Service
metadata:
  name: nginx-loadbalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

How NLB Connects to nginx Pods

The LoadBalancer Service acts as the bridge:

• You create a LoadBalancer Service with selector app: nginx
• Kubernetes provisions AWS NLB in the same VPC
• Service tracks which pods match the selector
• NLB forwards traffic to those nginx pods
• nginx pods receive external traffic and route it internally

nginx Configuration in Kubernetes

upstream auth {
    server auth-service:80;
}

server {
    listen 80;

    location /auth/ {
        proxy_pass http://auth;
    }
}

Lifecycle of a Request

1. User makes request to my-nlb-url.com/auth/login
2. NLB sees request and forwards it to a pod that matches the selector defined in the service (nginx)
3. Our nginx pod has a configuration to forward this request to auth service based on the /auth/ prefix
4. Auth service recieves request and responds, which flows back out through the way it came