Configuring Envoy as an edge proxy¶
Envoy is a production-ready edge proxy, however, the default settings are tailored for the service mesh use case, and some values need to be adjusted when using Envoy as an edge proxy.
TCP proxies should configure:
restrict access to the admin endpoint,
listener buffer limits to 32 KiB,
cluster buffer limits to 32 KiB.
HTTP proxies should additionally configure:
use_remote_address to true (to avoid consuming HTTP headers from external clients, see HTTP header sanitizing for details),
HTTP/2 maximum concurrent streams limit and HTTP/3 maximum concurrent streams limit to 100
HTTP/2 initial stream window size limit to 64 KiB,
headers_with_underscores_action setting to REJECT_REQUEST, to protect upstream services that treat ‘_’ and ‘-‘ as interchangeable.
If Envoy is configured with RBAC filter or makes route selection based on URL path it is recommended to enable the following path normalization options to minimize probability of path confusion vulnerabilities. Path confusion vulnerabilities occur when parties participating in request use different path representations.
Enable normalize_path setting.
Enable merge_slashes setting.
Additionally the path_with_escaped_slashes_action setting should be set according to following recommendations:
REJECT_REQUEST if dowstream clients are expected to use RFC 3986 compliant normalized paths (i.e. gRPC clients).
UNESCAPE_AND_REDIRECT if downstream client supports HTTP redirect (i.e. a browser). This option minimizes possibility of path confusion by forcing request to be re-issued with the same path across all parties: downstream client, Envoy and upstream server. Note that gRPC requests will still be rejected with the INTERNAL (13) error code, as gRPC clients do not support redirect.
KEEP_UNCHANGED for servers that are not RFC 3986 compliant and require encoded slashes.
UNESCAPE_AND_FORWARD for servers that are known to treat escaped and unescaped slashes equivalently. Choosing this option may increase probablity of path confusion vulnerabilities if intermediaries perform path based access control.
The following is a YAML example of the above recommendation (taken from the Google VRP edge server configuration):
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: "envoy.resource_monitors.fixed_heap"
typed_config:
"@type": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
# TODO: Tune for your system.
max_heap_size_bytes: 2147483648 # 2 GiB
actions:
- name: "envoy.overload_actions.shrink_heap"
triggers:
- name: "envoy.resource_monitors.fixed_heap"
threshold:
value: 0.95
- name: "envoy.overload_actions.stop_accepting_requests"
triggers:
- name: "envoy.resource_monitors.fixed_heap"
threshold:
value: 0.98
admin:
address:
socket_address:
address: 127.0.0.1
port_value: 9090
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 443
listener_filters:
- name: "envoy.filters.listener.tls_inspector"
# Uncomment if Envoy is behind a load balancer that exposes client IP address using the PROXY protocol.
# - name: envoy.filters.listener.proxy_protocol
# typed_config:
# "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
per_connection_buffer_limit_bytes: 32768 # 32 KiB
filter_chains:
- filter_chain_match:
server_names: ["example.com", "www.example.com"]
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain: {filename: "certs/servercert.pem"}
private_key: {filename: "certs/serverkey.pem"}
alpn_protocols: ["h2,http/1.1"]
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
use_remote_address: true
normalize_path: true
merge_slashes: true
path_with_escaped_slashes_action: UNESCAPE_AND_REDIRECT
common_http_protocol_options:
idle_timeout: 3600s # 1 hour
headers_with_underscores_action: REJECT_REQUEST
http2_protocol_options:
max_concurrent_streams: 100
initial_stream_window_size: 65536 # 64 KiB
initial_connection_window_size: 1048576 # 1 MiB
stream_idle_timeout: 300s # 5 mins, must be disabled for long-lived and streaming requests
request_timeout: 300s # 5 mins, must be disabled for long-lived and streaming requests
http_filters:
- name: envoy.filters.http.router
route_config:
virtual_hosts:
- name: default
domains: ["*"]
routes:
- match: {prefix: "/"}
route:
cluster: service_foo
idle_timeout: 15s # must be disabled for long-lived and streaming requests
clusters:
- name: service_foo
per_connection_buffer_limit_bytes: 32768 # 32 KiB
load_assignment:
cluster_name: some_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options:
initial_stream_window_size: 65536 # 64 KiB
initial_connection_window_size: 1048576 # 1 MiB
layered_runtime:
layers:
- name: static_layer_0
static_layer:
envoy:
resource_limits:
listener:
example_listener_name:
connection_limit: 10000
overload:
global_downstream_max_connections: 50000