Dynamic configuration (control plane)

This example walks through configuring Envoy using the Go Control Plane reference implementation.

It demonstrates how configuration provided to Envoy persists, even when the control plane is not available, and provides a trivial example of how to update Envoy’s configuration dynamically.

Step 1: Start the proxy container

Change directory to examples/dynamic-config-cp in the Envoy repository.

First build the containers and start the proxy container.

This should also start two upstream HTTP echo servers, service1 and service2.

The control plane has not yet been started.

$ pwd
envoy/examples/dynamic-config-cp
$ docker compose pull
$ docker compose up --build -d proxy
$ docker compose ps

       Name                            Command                State                     Ports
------------------------------------------------------------------------------------------------------------------------
dynamic-config-cp_proxy_1      /docker-entrypoint.sh /usr ... Up      0.0.0.0:10000->10000/tcp, 0.0.0.0:19000->19000/tcp
dynamic-config-cp_service1_1   /bin/echo-server               Up      8080/tcp
dynamic-config-cp_service2_1   /bin/echo-server               Up      8080/tcp

Step 2: Check initial config and web response

As we have not yet started the control plane, nothing should be responding on port 10000.

$ curl http://localhost:10000
curl: (56) Recv failure: Connection reset by peer

Dump the proxy’s static_clusters configuration and you should see the cluster named xds_cluster configured for the control plane:

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].static_clusters'

[
  {
    "cluster": {
      "@type": "type.googleapis.com/envoy.api.v2.Cluster",
      "name": "xds_cluster",
      "type": "STRICT_DNS",
      "connect_timeout": "1s",
      "http2_protocol_options": {},
      "load_assignment": {
        "cluster_name": "xds_cluster",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "go-control-plane",
                      "port_value": 18000
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "last_updated": "2020-10-25T20:20:54.897Z"
  }
]

No dynamic_active_clusters have been configured yet:

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'
null

Step 3: Start the control plane

Start up the go-control-plane service.

You may need to wait a moment or two for it to become healthy.

$ docker compose up --build -d go-control-plane
$ docker compose ps

        Name                                Command                  State                    Ports
-------------------------------------------------------------------------------------------------------------------------------------
dynamic-config-cp_go-control-plane_1  bin/example -debug             Up (healthy)
dynamic-config-cp_proxy_1             /docker-entrypoint.sh /usr ... Up            0.0.0.0:10000->10000/tcp, 0.0.0.0:19000->19000/tcp
dynamic-config-cp_service1_1          /bin/echo-server               Up            8080/tcp
dynamic-config-cp_service2_1          /bin/echo-server               Up            8080/tcp

Step 4: Query the proxy

Once the control plane has started and is healthy, you should be able to make a request to port 10000, which will be served by service1.

$ curl http://localhost:10000
Request served by service1

HTTP/1.1 GET /

Host: localhost:10000
Accept: */*
X-Forwarded-Proto: http
X-Request-Id: 1d93050e-f39c-4602-90f8-a124d6e78d26
X-Envoy-Expected-Rq-Timeout-Ms: 15000
Content-Length: 0
User-Agent: curl/7.72.0

Step 5: Dump Envoy’s `dynamic_active_clusters` config

If you now dump the proxy’s dynamic_active_clusters configuration, you should see it is configured with the example_proxy_cluster pointing to service1, and a version of 1.

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'

[
  {
    "version_info": "1",
    "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "example_proxy_cluster",
      "type": "LOGICAL_DNS",
      "connect_timeout": "5s",
      "dns_lookup_family": "V4_ONLY",
      "load_assignment": {
        "cluster_name": "example_proxy_cluster",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "service1",
                      "port_value": 8080
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "last_updated": "2020-10-25T20:37:05.838Z"
  }
]

Step 6: Stop the control plane

Stop the go-control-plane service:

$ docker compose stop go-control-plane

The Envoy proxy should continue proxying responses from service1.

$ curl http://localhost:10000 | grep "served by"
Request served by service1

Step 7: Edit `go` file and restart the control plane

The example setup starts the go-control-plane service with a custom resource.go file which specifies the configuration provided to Envoy.

Update this to have Envoy proxy instead to service2.

Edit resource.go in the dynamic configuration example folder and change the UpstreamHost from service1 to service2:

const (
	ClusterName  = "example_proxy_cluster"
	RouteName    = "local_route"
	ListenerName = "listener_0"
	ListenerPort = 10000
	UpstreamHost = "service1"
	UpstreamPort = 8080
)

Further down in this file you must also change the configuration snapshot version number from "1" to "2" to ensure Envoy sees the configuration as newer:

func GenerateSnapshot() cache.Snapshot {
	return cache.NewSnapshot(
		"1",
		[]types.Resource{}, // endpoints
		[]types.Resource{makeCluster(ClusterName)},
		[]types.Resource{makeRoute(RouteName, ClusterName)},
		[]types.Resource{makeHTTPListener(ListenerName, RouteName)},
		[]types.Resource{}, // runtimes
		[]types.Resource{}, // secrets
		[]types.Resource{}, // extensions configs
	)
}

Now rebuild and restart the control plane:

$ docker compose up --build -d go-control-plane

You may need to wait a moment or two for the go-control-plane service to become healthy again.

Step 8: Check Envoy uses the updated configuration

Now when you make a request to the proxy it should be served by the service2 upstream service.

$ curl http://localhost:10000 | grep "served by"
Request served by service2

Dumping the dynamic_active_clusters you should see the cluster configuration now has a version of 2, and example_proxy_cluster is configured to proxy to service2:

$ curl -s http://localhost:19000/config_dump  | jq '.configs[1].dynamic_active_clusters'

[
  {
    "version_info": "2",
    "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "example_proxy_cluster",
      "type": "LOGICAL_DNS",
      "connect_timeout": "5s",
      "dns_lookup_family": "V4_ONLY",
      "load_assignment": {
        "cluster_name": "example_proxy_cluster",
        "endpoints": [
          {
            "lb_endpoints": [
              {
                "endpoint": {
                  "address": {
                    "socket_address": {
                      "address": "service2",
                      "port_value": 8080
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "last_updated": "2020-10-26T14:35:17.360Z"
  }
]

Note

In this example we increment the version for simplicity.

Any change to the version will trigger an update in Envoy, and ordering is not significant (see xDS protocol docs for further information about updates).