Learn how proxies work by building one using Go

Introduction to proxy servers
- 1. Forward proxy
- 2. Reverse proxy
What are we going to build
Bootstrapping our project
Running demo http servers
Writing the config.yaml file
Loading data from config.yaml to our application
Creating entry point for our application
Creating proxy and its handler
Creating and running http server for our reverse proxy
Updating makefile to run proxy server
Running and testing our reverse proxy
Testing
- Check if demo servers are running
- Check if reverse proxy servers is running

Introduction to proxy servers

It is a server that sits in between client machine and servers. There are different types of proxy servers but all of them can be generalised into two: Forward and Reverse

1. Forward proxy

A forward proxy sits in front of the clients and sends request on behalf of client to web servers. When request is sent by clients, forward proxy intercepts the request and then sends request to web servers on their behalf. It acts on behalf of client. It is generally used when you don’t want server to know your address as forward proxy send request on client behalf.

2. Reverse proxy

A reverse proxy sits in front of the web servers and intercepts all traffic sent by client to web servers. It intercepts request coming from clients and then send those request to appropriate web servers.

It is generally used to improve security, performance and for reliability.

Read more about proxies herre: https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/

What are we going to build

This in article we will be building reverse proxy named TinyRP using Go. We will make reverse configurable using yaml file so that we can add any number of endpoint and destination to our reverse proxy without modifying the code.

This will be an ongoing project but for now we will only focus on creating a working reverse proxy.

Bootstrapping our project

Let’s first create all the required directories so that it will make our life easier.

mkdir -p cmd internal/configs internal/server data

cmd : It will contain entry main.go for our project.

internal/configs: It will hold configuration code to load values from yaml data into our application

internal/server: It will hold code to create and run HTTP proxy server.

data: It will contain the configuration yaml file for our project.

Now again, let’s create all the files that is required in this project

touch Makefile data/config.yaml cmd/main.go internal/configs/config.go internal/server/proxy_handlers.go internal/server/healthcheck.go  internal/server/server.go

Makefile : It will hold recepies to run and stop our application.

config.yaml: It will hold our reverse proxy configuration.

main.go -> It is used at entrypoint to our code

config.go -> It will hold code to load configuration

proxy_handlers.go : It will contain functions to creating new proxy and proxy handler

healthcheck.go: It will be a simple http handler to check if our server is running or not.

server.go : It will be used to create and run HTTP server

All the required files are created, your directory structure should look like this

rev-proxy-dir-structure

Running demo http servers

Open make file and paste following code into Makefile. What we are doing here is we are running three docker container of simple http servers running on port 9001, 9002, 9003

# run: starts demo http services
.PHONY: run
run:
	docker run --rm -d -p 9001:80 --name server1 kennethreitz/httpbin
	docker run --rm -d -p 9002:80 --name server2 kennethreitz/httpbin
	docker run --rm -d -p 9003:80 --name server3 kennethreitz/httpbin

# stop: stops all demo services
.PHONY: stop
stop:
	docker stop server1
	docker stop server2
	docker stop server3

now if you run make run then it will start all three servers which you can view by docker ps.

To stop all the servers, just run make stop. For now let’s just run the server and leave it as it is.

Writing the config.yaml file

Once this is all done, we are ready to start coding.

Let’s first write our configuration yaml file.

What are we doing here is, we are defining some parameters for our reverse proxy so that we can tune it in future without modifying the code.

We are also specifying the reverse proxy where we will have list of resources and each resource will contain following

name: Name of the resource

endpoint: Ednpoint it will be serving at

destination_url: Destination URL where the incoming request at endpoint will be forwarded to.

Copy and pase the fllowing code into data/config.yaml file.

server:
  host: "localhost"
  listen_port: "8080"

resources:
  - name: Server1
    endpoint: /server1
    destination_url: "http://localhost:9001"
  - name: Server2
    endpoint: /server2
    destination_url: "http://localhost:9002"
  - name: Server3
    endpoint: /server3
    destination_url: "http://localhost:9003"

As you can see that we are trying to create a reverse proxy which will accept request at http://localhost:8080/server1 and proxy it to our demo server http://localhost:9001 and same for other endpoints.

Loading data from config.yaml to our application

Now we have created our configuration file, let’s write code to load configuration into our program.

Copy and paste the following code into internal/config/config.go.

package configs

import (
	"fmt"
	"strings"

	"github.com/spf13/viper"
)

type resource struct {
	Name            string
	Endpoint        string
	Destination_URL string
}

type configuration struct {
	Server struct {
		Host        string
		Listen_port string
	}

	Resources []resource
}

var Config *configuration

func NewConfiguration() (*configuration, error) {
	viper.AddConfigPath("data")
	viper.SetConfigName("config")
	viper.SetConfigType("yaml")
	viper.AutomaticEnv()

	viper.SetEnvKeyReplacer(strings.NewReplacer(`.`, `_`))
	err := viper.ReadInConfig()

	if err != nil {
		return nil, fmt.Errorf("error loading config file: %s", err)
	}

	err = viper.Unmarshal(&Config)
	if err != nil {
		return nil, fmt.Errorf("error reading config file: %s", err)
	}
	return Config, nil
}

We are using Viper to load config yaml files into our go program. You can see how its done using this video here: Use configuration file in Go project to manage app settings

What we did above it, we create a custom data type of type struct called configuration.

type resource struct {
	Name            string
	Endpoint        string
	Destination_URL string
}

type configuration struct {
	Server struct {
		Host        string
		Listen_port string
	}

	Resources []resource
}

then we created a variable called Config of type configuration which will hold the data from yaml file once its loaded.

then we created a function called NewConfiguration which will use viper and load configuration from yaml file to our variable Config.

As you noticed, we are using function with first letter uppercase which means it will be exported. We will later call this function get our configuration values.

Creating entry point for our application

let’s create the main function in out cmd/main.go file

Here we will call another function which we will create in server package. We will call server.Run() to run the server. Our Run function will return us some error hence we handle that error as well. Don’t worry we will implement that function next, for now just copy and paste following code into cmd/main.go

package main

import (
	"log"

	"github.com/pgaijin66/lightweight-reverse-proxy/internal/server"
)

func main() {
	if err := server.Run(); err != nil {
		log.Fatalf("could not start the server: %v", err)
	}
}

Creating proxy and its handler

Open internal/server/proxy_handlers.go and paste following code

package server

import (
	"fmt"
	"net/http"
	"net/http/httputil"
	"net/url"
	"strings"
	"time"
)

func NewProxy(target *url.URL) *httputil.ReverseProxy {
	proxy := httputil.NewSingleHostReverseProxy(target)
	return proxy
}

func ProxyRequestHandler(proxy *httputil.ReverseProxy, url *url.URL, endpoint string) func(http.ResponseWriter, *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
		fmt.Printf("[ TinyRP ] Request received at %s at %s\n", r.URL, time.Now().UTC())
		// Update the headers to allow for SSL redirection
		r.URL.Host = url.Host
		r.URL.Scheme = url.Scheme
		r.Header.Set("X-Forwarded-Host", r.Header.Get("Host"))
		r.Host = url.Host

		//trim reverseProxyRoutePrefix
		path := r.URL.Path
		r.URL.Path = strings.TrimLeft(path, endpoint)

		// Note that ServeHttp is non blocking and uses a go routine under the hood
		fmt.Printf("[ TinyRP ] Redirecting request to %s at %s\n", r.URL, time.Now().UTC())
		proxy.ServeHTTP(w, r)

	}
}

Here we are implementing two functions NewProxy and ProxyRequestHandler.

Newproxy: Creates take an url and a new http reverse proxy

ProxyRequestHandler: takes the proxy created by NewProxy, destination URL and endpint and returns a http hander similar to our ping handler. ProxyRequestHandler updates the http headers to redirect request coming at given endpoint to the destination_url.

Creating and running http server for our reverse proxy

Let’s implement the Run function that we called in main.go. Go to internal/server/server.go and paste in following code

package server

import (
	"fmt"
	"net/http"
	"net/url"

	"github.com/pgaijin66/lightweight-reverse-proxy/internal/configs"
)

// Run starts server and listens on defined port
func Run() error {
	// load configurations from config file
	config, err := configs.NewConfiguration()
	if err != nil {
		fmt.Errorf("could not load configuration: %v", err)
	}

	// Creates a new router
	mux := http.NewServeMux()

	// Registering the healthcheck endpoint
	mux.HandleFunc("/ping", ping)

	// Iterating through the configuration resource and registering them
	// into the router.
	for _, resource := range config.Resources {
		url, _ := url.Parse(resource.Destination_URL)
		proxy := NewProxy(url)
		mux.HandleFunc(resource.Endpoint, ProxyRequestHandler(proxy, url, resource.Endpoint))
	}

	// Running proxy server
	if err := http.ListenAndServe(config.Server.Host+":"+config.Server.Listen_port, mux); err != nil {
		return fmt.Errorf("could not start the server: %v", err)
	}
	return nil

}

In above code snippet, we are creating a function called Run which loads configuration by calling NewConfiguration function from config package we created and stores into a variable called config.

We create a new server called mux using net/http package

We register route called /ping to call our http handler ping which we have not created yet.

Then we iterate through the resources endpoints and register them into our router as that we what we want to achieve i.e when someone tries to access http://localhost:8080/server1 then our reverse proxy will intercept that request and forward it to http://localhost:9001 which is their destination_url.

Now we have registered all the routes, we are ready to start the server which we do by calling http.ListenAndServe. As you saw, our Run function returns an error and If everything works out well we return nil otherwise return an error.

Let’s quickly implement our healthcheck handler. Copy and paste following code into internal/server/healthcheck.go.

package server

import "net/http"

// ping returns a "pong" message
func ping(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte("pong"))
}

Before we jump next, let’s understand what this code block is doing

	for _, resource := range config.Resources {
		url, _ := url.Parse(resource.Destination_URL)
		proxy := NewProxy(url)
		mux.HandleFunc(resource.Endpoint, ProxyRequestHandler(proxy, url, resource.Endpoint))
	}

As i said above it iterated through the endpints and registers handler for them. But its doing a bit more than that.

We are iterating through each resources in config.yaml which gives us individual resource on each iteration and each resource holds name, endpoint and destination_url.

Now for each resource, we are taking it’s destination URl and parsing it. The reason we are parsing it is our function to create new proxy will take *url.URL as an argument. since resource.Destination_URL is a string, we are paring it and getting url which is of type *url.URL.

Then we call function called Newproxy with url as our input. This will return us with a new proxy of type *httputil.ReverseProxy. Now we register the resource.Endpoint to the handler returned by our ProxyRequestHandler.

As we saw above, our ProxyRequestHandler function takes three parameters revese proxy, destination URL and endpoint.

Updating makefile to run proxy server

Let’s update our Makefile to add run proxy server command. Append following code snippet into the Makefile

# help: print this help message
.PHONY: help
help:
	@echo 'Usage:'
	@sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' |  sed -e 's/^/ /'

# run: starts demo http services
.PHONY: run-proxy-server
run-proxy-server:
	go run cmd/main.go

We added help command as well which shows all the available commands in a nice way when we run make help

$ make help

Usage:
  help               print this help message
  run-containers     starts demo http services
  run-proxy-server   starts demo http services
  stop               stops all demo services

Running and testing our reverse proxy

That’s it. Go ahead and run the server using make run-proxy-server.

Note: make Sure you already ran make run-containers to start demo servers

Testing

Check if demo servers are running

On another terminal run following commands

Testing server1

$ curl -iv  http://localhost:9001

HTTP/1.1 200 OK
Server: gunicorn/19.9.0
Date: Thu, 01 Dec 2022 21:09:14 GMT
Connection: keep-alive
Content-Type: text/html; charset=utf-8
Content-Length: 9593
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Testing server2

$ curl -iv  http://localhost:9002

HTTP/1.1 200 OK
Server: gunicorn/19.9.0
Date: Thu, 01 Dec 2022 21:09:14 GMT
Connection: keep-alive
Content-Type: text/html; charset=utf-8
Content-Length: 9593
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Testing server3

$ curl -iv  http://localhost:9003

HTTP/1.1 200 OK
Server: gunicorn/19.9.0
Date: Thu, 01 Dec 2022 21:09:14 GMT
Connection: keep-alive
Content-Type: text/html; charset=utf-8
Content-Length: 9593
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Check if reverse proxy servers is running

$ curl -I  http://localhost:8080/server1

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 9593
Content-Type: text/html; charset=utf-8
Date: Thu, 01 Dec 2022 21:11:18 GMT
Server: gunicorn/19.9.0

Let’s look at our server log, we should see something like this

[ TinyRP ] Request received at /server1 at 2022-12-01 21:11:10.922582 +0000 UTC
[ TinyRP ] Redirecting request to http://localhost:9001 at 2022-12-01 21:11:10.922634 +0000 UTC

woo hoo !!!

Our reverse proxy server is serving request and sending to appropriate destination urls.

I hope this information was valuable to you.

You can find the ongoing source code for the project here: https://github.com/pgaijin66/tinyrp