xDS API endpoints
An xDS management server will implement the below endpoints as required for gRPC and/or REST serving. In both streaming gRPC and REST-JSON cases, a DiscoveryRequest is sent and a DiscoveryResponse received following the xDS protocol.
Below we describe endpoints for the v3 transport API.
gRPC streaming endpoints
- POST /envoy.service.cluster.v3.ClusterDiscoveryService/StreamClusters
See cds.proto for the service definition. This is used by Envoy as a client when
12 cds_config:
13 resource_api_version: V3
14 api_config_source:
15 api_type: GRPC
16 transport_api_version: V3
17 grpc_services:
18 - envoy_grpc:
19 cluster_name: xds_cluster
is set in the dynamic_resources of the Bootstrap config.
- POST /envoy.service.endpoint.v3.EndpointDiscoveryService/StreamEndpoints
See eds.proto for the service definition. This is used by Envoy as a client when
eds_config:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: some_xds_cluster
is set in the eds_cluster_config field of the Cluster config.
- POST /envoy.service.listener.v3.ListenerDiscoveryService/StreamListeners
See lds.proto for the service definition. This is used by Envoy as a client when
20 lds_config:
21 resource_api_version: V3
22 api_config_source:
23 api_type: GRPC
24 transport_api_version: V3
25 grpc_services:
26 - envoy_grpc:
27 cluster_name: xds_cluster
is set in the dynamic_resources of the Bootstrap config.
- POST /envoy.service.route.v3.RouteDiscoveryService/StreamRoutes
See rds.proto for the service definition. This is used by Envoy as a client when
route_config_name: some_route_name
config_source:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: some_xds_cluster
is set in the rds field of the HttpConnectionManager config.
- POST /envoy.service.route.v3.ScopedRoutesDiscoveryService/StreamScopedRoutes
See srds.proto for the service definition. This is used by Envoy as a client when
name: some_scoped_route_name
scoped_rds:
resource_api_version: V3
config_source:
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: some_xds_cluster
is set in the scoped_routes field of the HttpConnectionManager config.
- POST /envoy.service.secret.v3.SecretDiscoveryService/StreamSecrets
See sds.proto for the service definition. This is used by Envoy as a client when
48 token_secret:
49 name: token
50 sds_config:
51 resource_api_version: V3
52 api_config_source:
53 api_type: GRPC
54 transport_api_version: V3
55 grpc_services:
56 - envoy_grpc:
57 cluster_name: sds_server_uds
is set inside a SdsSecretConfig message. This message is used in various places such as the CommonTlsContext.
- POST /envoy.service.runtime.v3.RuntimeDiscoveryService/StreamRuntime
See rtds.proto for the service definition. This is used by Envoy as a client when
name: some_runtime_layer_name
config_source:
resource_api_version: V3
api_config_source:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: some_xds_cluster
is set inside the rtds_layer field.
REST endpoints
- POST /v3/discovery:clusters
See cds.proto for the service definition. This is used by Envoy as a client when
cds_config:
resource_api_version: V3
api_config_source:
api_type: REST
transport_api_version: V3
cluster_names: [some_xds_cluster]
is set in the dynamic_resources of the Bootstrap config.
- POST /v3/discovery:endpoints
See eds.proto for the service definition. This is used by Envoy as a client when
eds_config:
resource_api_version: V3
api_config_source:
api_type: REST
transport_api_version: V3
cluster_names: [some_xds_cluster]
is set in the eds_cluster_config field of the Cluster config.
- POST /v3/discovery:listeners
See lds.proto for the service definition. This is used by Envoy as a client when
lds_config:
resource_api_version: V3
api_config_source:
api_type: REST
transport_api_version: V3
cluster_names: [some_xds_cluster]
is set in the dynamic_resources of the Bootstrap config.
- POST /v3/discovery:routes
See rds.proto for the service definition. This is used by Envoy as a client when
route_config_name: some_route_name
config_source:
resource_api_version: V3
api_config_source:
api_type: REST
transport_api_version: V3
cluster_names: [some_xds_cluster]
is set in the rds field of the HttpConnectionManager config.
Note
The management server responding to these endpoints must respond with a DiscoveryResponse along with a HTTP status of 200. Additionally, if the configuration that would be supplied has not changed (as indicated by the version supplied by the Envoy client) then the management server can respond with an empty body and a HTTP status of 304.
Aggregated Discovery Service
While Envoy fundamentally employs an eventual consistency model, ADS provides an opportunity to sequence API update pushes and ensure affinity of a single management server for an Envoy node for API updates. ADS allows one or more APIs and their resources to be delivered on a single, bidirectional gRPC stream by the management server. Without this, some APIs such as RDS and EDS may require the management of multiple streams and connections to distinct management servers.
ADS will allow for hitless updates of configuration by appropriate sequencing. For example, suppose foo.com was mapped to cluster X. We wish to change the mapping in the route table to point foo.com at cluster Y. In order to do this, a CDS/EDS update must first be delivered containing both clusters X and Y.
Without ADS, the CDS/EDS/RDS streams may point at distinct management servers, or when on the same management server at distinct gRPC streams/connections that require coordination. The EDS resource requests may be split across two distinct streams, one for X and one for Y. ADS allows these to be coalesced to a single stream to a single management server, avoiding the need for distributed synchronization to correctly sequence the update. With ADS, the management server would deliver the CDS, EDS and then RDS updates on a single stream.
ADS is only available for gRPC streaming (not REST) and is described more fully in xDS document. The gRPC endpoint is:
- POST /envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources
See discovery.proto for the service definition. This is used by Envoy as a client when
6 ads_config:
7 api_type: GRPC
8 transport_api_version: V3
9 grpc_services:
10 - envoy_grpc:
11 cluster_name: xds_cluster
is set in the dynamic_resources of the Bootstrap config.
When this is set, any of the configuration sources above can be set to use the ADS channel. For example, a LDS config could be changed from
lds_config:
resource_api_version: V3
api_config_source:
api_type: REST
transport_api_version: V3
cluster_names: [some_xds_cluster]
to
lds_config: {ads: {}}
with the effect that the LDS stream will be directed to some_ads_cluster over the shared ADS channel.
Delta endpoints
The REST, filesystem, and original gRPC xDS implementations all deliver “state of the world” updates: every CDS update must contain every cluster, with the absence of a cluster from an update implying that the cluster is gone. For Envoy deployments with huge amounts of resources and even a trickle of churn, these state-of-the-world updates can be cumbersome.
As of 1.12.0, Envoy supports a “delta” variant of xDS (including ADS), where updates only contain resources added/changed/removed. Delta xDS is a gRPC (only) protocol. Delta uses different request/response protos than SotW (DeltaDiscovery{Request,Response}); see discovery.proto. Conceptually, delta should be viewed as a new xDS transport type: there is static, filesystem, REST, gRPC-SotW, and now gRPC-delta. (Envoy’s implementation of the gRPC-SotW/delta client happens to share most of its code between the two, and something similar is likely possible on the server side. However, they are in fact incompatible protocols. The specification of the delta xDS protocol’s behavior is here.)
To use delta, simply set the api_type field of your ApiConfigSource proto(s) to DELTA_GRPC. That works for both xDS and ADS; for ADS, it’s the api_type field of DynamicResources.ads_config, as described in the previous section.
xDS TTL
When using xDS, users might find themself wanting to temporarily update certain xDS resources. In order to do safely, xDS TTLs can be used to make sure that if the control plane becomes unavailable and is unable to revert the xDS change, Envoy will remove the resource after a TTL specified by the server. See the protocol documentation for more information.
Currently the behavior when a TTL expires is that the resource is removed (as opposed to reverted to the previous version). As such, this feature should primarily be used for use cases where the absence of the resource is preferred instead of the temporary version, e.g. when using RTDS to apply a temporary runtime override.
The TTL is specified on the Resource proto: for Delta xDS this is specified directly within the response, while for SotW xDS the server may wrap individual resources listed in the response within a Resource in order to specify a TTL value.
The server can refresh or modify the TTL by issuing another response for the same version. In this case the resource itself does not have to be included.