1.20.0 (October 5, 2021)

Incompatible Behavior Changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • config: due to the switch to using work-in-progress annotations and warnings to indicate APIs subject to change, the following API packages have been force migrated from v3alpha to v3: envoy.extensions.access_loggers.open_telemetry.v3, envoy.extensions.cache.simple_http_cache.v3, envoy.extensions.filters.http.admission_control.v3, envoy.extensions.filters.http.bandwidth_limit.v3, envoy.extensions.filters.http.cache.v3, envoy.extensions.filters.http.cdn_loop.v3, envoy.extensions.filters.http.ext_proc.v3, envoy.extensions.filters.http.oauth2.v3, envoy.extensions.filters.network.sni_dynamic_forward_proxy.v3, envoy.extensions.filters.udp.dns_filter.v3, envoy.extensions.transport_sockets.s2a.v3, envoy.extensions.watchdog.profile_action.v3, envoy.service.ext_proc.v3, and envoy.watchdog.v3. If your production deployment was using one of these APIs, you will be forced to potentially vendor the old proto file to continue serving old versions of Envoy. The project realizes this is unfortunate because some of these are known to be used in production, however the project does not have the resources to undergo a migration in which we support v3alpha and v3 at the same time. The switch to using work-in-progress annotations with clear and explicit warnings will avoid any such issue in the future. We apologize again for any difficulty this change causes, though it is for the best. Additionally, some of the above namespaces have had their work-in-progress annotations removed due to known production usage. Thus, they will not warn and are offered full API stability support by the project from this point forward.

  • config: the --bootstrap-version CLI flag has been removed, Envoy has only been able to accept v3 bootstrap configurations since 1.18.0.

  • contrib: the squash filter has been moved to contrib images.

  • contrib: the kafka broker filter has been moved to contrib images.

  • contrib: the RocketMQ proxy filter has been moved to contrib images.

  • contrib: the Postgres proxy filter has been moved to contrib images.

  • contrib: the MySQL proxy filter has been moved to contrib images.

  • dns_filter: dns_filter protobuf fields have been renumbered to restore compatibility with Envoy 1.18, breaking compatibility with Envoy 1.19.0 and 1.19.1. The new field numbering allows control planes supporting Envoy 1.18 to gracefully upgrade to dns_resolution_config, provided they skip over Envoy 1.19.0 and 1.19.1. Control planes upgrading from Envoy 1.19.0 and 1.19.1 will need to vendor the corresponding protobuf definitions to ensure that the renumbered fields have the types expected by those releases.

  • extensions: deprecated extension names now default to triggering a configuration error. The previous warning-only behavior may be temporarily reverted by setting the runtime key envoy.deprecated_features.allow_deprecated_extension_names to true.

Minor Behavior Changes

Changes that may cause incompatibilities for some users, but should not for most

  • client_ssl_auth filter: now sets additional termination details and UAEX response flag when the client certificate is not in the allowed-list.

  • config: configuration files ending in .yml now load as YAML.

  • config: configuration file extensions now ignore case when deciding the file type. E.g., .JSON files load as JSON.

  • config: reduced log level for “Unable to establish new stream” xDS logs to debug. The log level for “gRPC config stream closed” is now reduced to debug when the status is Ok or has been retriable (DeadlineExceeded, ResourceExhausted, or Unavailable) for less than 30 seconds.

  • config: use of work-in-progress API files, messages, or fields will now generate an explicit warning. Please read the text about (xds.annotations.v3.file_status).work_in_progress, (xds.annotations.v3.message_status).work_in_progress, and (xds.annotations.v3.field_status).work_in_progress here for more information. Some APIs that are known to be implicitly not work-in-progress have been force migrated and are individually indicated elsewhere in the release notes. A server-wide wip_protos counter has also been added in server statistics to track this.

  • ext_authz: fixed skipping authentication when returning either a direct response or a redirect. This behavior can be temporarily reverted by setting the envoy.reloadable_features.http_ext_authz_do_not_skip_direct_response_and_redirect runtime guard to false.

  • grpc: gRPC async client can be cached and shared across filter instances in the same thread, this feature is turned off by default, can be turned on by setting runtime guard envoy.reloadable_features.enable_grpc_async_client_cache to true.

  • http: correct the use of the x-forwarded-proto header and the :scheme header. Where they differ (which is rare) :scheme will now be used for serving redirect URIs and cached content. This behavior can be reverted by setting runtime guard correct_scheme_and_xfp to false.

  • http: reject requests with #fragment in the URI path. The fragment is not allowed to be part of the request URI according to RFC3986 (3.5), RFC7230 (5.1) and RFC 7540 (8.1.2.3). Rejection of requests can be changed to stripping the #fragment instead by setting the runtime guard envoy.reloadable_features.http_reject_path_with_fragment to false. This behavior can further be changed to the deprecated behavior of keeping the fragment by setting the runtime guard envoy.reloadable_features.http_strip_fragment_from_path_unsafe_if_disabled. This runtime guard must only be set to false when existing non-compliant traffic relies on #fragment in URI. When this option is enabled, Envoy request authorization extensions may be bypassed. This override and its associated behavior will be decommissioned after the standard deprecation period.

  • http: set the default lazy headermap threshold to 3, which defines the minimal number of headers in a request/response/trailers required for using a dictionary in addition to the list. Setting the envoy.http.headermap.lazy_map_min_size runtime feature to a non-negative number will override the default value.

  • http: stop processing pending H/2 frames if connection transitioned to a closed state. This behavior can be temporarily reverted by setting the envoy.reloadable_features.skip_dispatching_frames_for_closed_connection to false.

  • listener: added the enable_reuse_port field and changed the default for reuse_port from false to true, as the feature is now well supported on the majority of production Linux kernels in use. The default change is aware of the hot restart, as otherwise, the change would not be backward compatible between restarts. This means that hot restarting onto a new binary will retain the default of false until the binary undergoes a full restart. To retain the previous behavior, either explicitly set the new configuration field to false, or set the runtime feature flag envoy.reloadable_features.listener_reuse_port_default_enabled to false. As part of this change, the use of reuse_port for TCP listeners on both macOS and Windows has been disabled due to suboptimal behavior. See the field documentation for more information.

  • listener: destroy per network filter chain stats when a network filter chain is removed during the listener in-place update.

  • quic: enables IETF connection migration. This feature requires a stable UDP packet routine in the L4 load balancer with the same first-4-bytes in connection id. It can be turned off by setting runtime guard envoy.reloadable_features.FLAGS_quic_reloadable_flag_quic_connection_migration_use_new_cid_v2 to false.

  • thrift_proxy: allow Framed and Header transport combinations to perform payload passthrough.

Bug Fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • access log: fix %UPSTREAM_CLUSTER% when used in http upstream access logs. Previously, it was always logging as an unset value.

  • aws request signer: fix the AWS Request Signer extension to correctly normalize the path and query string to be signed according to AWS’ guidelines, so that the hash on the server side matches. See AWS SigV4 documentation.

  • cluster: delete pools when they’re idle to fix unbounded memory use when using PROXY protocol upstream with tcp_proxy. This behavior can be temporarily reverted by setting the envoy.reloadable_features.conn_pool_delete_when_idle runtime guard to false.

  • cluster: finish cluster warming even if hosts are removed before health check initialization. This only affected clusters with ignore_health_on_host_removal.

  • compressor: fix a bug where if trailers were added and a subsequent filter paused the filter chain, the request could be stalled. This behavior can be reverted by setting envoy.reloadable_features.fix_added_trailers to false.

  • dynamic forward proxy: fixing a validation bug where san and sni checks were not applied setting http_protocol_options via typed_extension_protocol_options.

  • ext_authz: fix the ext_authz filter to correctly merge multiple same headers using the ‘,’ as separator in the check request to the external authorization service.

  • ext_authz: fix the use of append field of response_headers_to_add to set or append encoded response headers from a gRPC auth server.

  • ext_authz: fix the HTTP ext_authz filter to respond with 403 Forbidden when a gRPC auth server sends a denied check response with an empty HTTP status code.

  • ext_authz: the network ext_authz filter now correctly sets dynamic metadata returned by the authorization service for non-OK responses. This behavior now matches the http ext_authz filter.

  • hcm: remove deprecation for xff_num_trusted_hops and forbid mixing ip detection extensions with old related knobs.

  • http: limit use of deferred resets in the http2 codec to server-side connections. Use of deferred reset for client connections can result in incorrect behavior and performance problems.

  • listener: fixed an issue on Windows where connections are not handled by all worker threads.

  • lua: fix BodyBuffer setting a Lua string and printing Lua string containing hex characters. Previously, BodyBuffer setting a Lua string or printing strings with hex characters will be truncated.

  • xray: fix the AWS X-Ray tracer bug where span’s error, fault and throttle information was not reported properly as per the AWS X-Ray documentation. Before this fix, server error was reported under the ‘annotations’ section of the segment data.

Removed Config or Runtime

Normally occurs at the end of the deprecation period

  • http: removed envoy.reloadable_features.http_upstream_wait_connect_response runtime guard and legacy code paths.

  • http: removed envoy.reloadable_features.allow_preconnect runtime guard and legacy code paths.

  • listener: removed envoy.reloadable_features.disable_tls_inspector_injection runtime guard and legacy code paths.

  • ocsp: removed envoy.reloadable_features.check_ocsp_policy deprecation runtime guard and legacy code paths.

  • ocsp: removed envoy.reloadable_features.require_ocsp_response_for_must_staple_certs deprecation and legacy code paths.

  • quic: removed envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing runtime guard.

New Features

  • access_log: added METADATA token to handle all types of metadata (DYNAMIC, CLUSTER, ROUTE).

  • bootstrap: added inline_headers in the bootstrap to make custom inline headers bootstrap configurable.

  • contrib: added new contrib images which contain contrib extensions.

  • dns: added V4_PREFERRED option to return V6 addresses only if V4 addresses are not available.

  • ext_authz: added dynamic_metadata_from_headers to support emitting dynamic metadata from headers returned by an external authorization service via HTTP.

  • grpc reverse bridge: added a new option to support streaming response bodies when withholding gRPC frames from the upstream.

  • grpc_json_transcoder: added support to unescape ‘+’ in query parameters to space with a new config field query_param_unescape_plus.

  • http: added cluster_header in weighted_clusters to allow routing to the weighted cluster specified in the request_header.

  • http: added alternate_protocols_cache_options for enabling HTTP/3 connections to servers which advertise HTTP/3 support via HTTP Alternative Services and caching the advertisements to disk.

  • http: added string_match in the header matcher.

  • http: added x-envoy-upstream-stream-duration-ms that allows configuring the max stream duration via a request header.

  • http: added support for max_requests_per_connection for both upstream and downstream connections.

  • http: sanitizing the referer header as documented here. This feature can be temporarily turned off by setting runtime guard envoy.reloadable_features.sanitize_http_header_referer to false.

  • http: validating outgoing HTTP/2 CONNECT requests to ensure that if :path is set that :protocol is present. This behavior can be temporarily turned off by setting runtime guard envoy.reloadable_features.validate_connect to false.

  • jwt_authn: added support for Jwt Cache and its size can be specified by jwt_cache_size.

  • jwt_authn: added support for extracting JWTs from request cookies using from_cookies.

  • jwt_authn: added support for setting the extracted headers from a successfully verified JWT using header_in_metadata to dynamic metadata.

  • listener: new listener metric downstream_cx_transport_socket_connect_timeout to track transport socket timeouts.

  • lua: added header:getAtIndex() and header:getNumValues() methods to header object for retrieving the value of a header at certain index and get the total number of values for a given header.

  • matcher: added invert for inverting the match result in the metadata matcher.

  • overload: add a new overload action that resets streams using a lot of memory. To enable the tracking of allocated bytes in buffers that a stream is using we need to configure the minimum threshold for tracking via buffer_factory_config. We have an overload action Envoy::Server::OverloadActionNameValues::ResetStreams that takes advantage of the tracking to reset the most expensive stream first.

  • rbac: added destination_port_range for matching range of destination ports.

  • rbac: added matcher along with extension category extension_category_envoy.rbac.matchers for custom RBAC permission matchers. Added reference implementation for matchers envoy.rbac.matchers.upstream_ip_port.

  • route config: added dynamic_metadata for routing based on dynamic metadata.

  • router: added retry options predicate extensions configured via retry_options_predicates. These extensions allow modification of requests between retries at the router level. There are not currently any built-in extensions that implement this extension point.

  • router: added per_try_idle_timeout timeout configuration.

  • router: added an optional override_auto_sni_header to support setting SNI value from an arbitrary header other than host/authority.

  • sxg_filter: added filter to transform response to SXG package to contrib images. This can be enabled by setting SXG configuration.

  • thrift_proxy: added support for mirroring requests.

  • udp: allows updating filter chain in-place through LDS, which is supported by Quic listener. Such listener config will be rejected in other connection-less UDP listener implementations. It can be reverted by envoy.reloadable_features.udp_listener_updates_filter_chain_in_place.

  • udp: disallow L4 filter chain in config which configures connection-less UDP listener. It can be reverted by envoy.reloadable_features.udp_listener_updates_filter_chain_in_place.

  • upstream: added support for slow start mode, which allows to progresively increase traffic for new endpoints.

  • upstream: extended Round Robin load balancer configuration with slow start support.

  • upstream: extended Least Request load balancer configuration with slow start support.

  • windows: added a new container image based on Windows Nanoserver 2022.

  • xray: request direction (ingress or egress) is recorded as X-Ray trace segment’s annotation by name direction.

Deprecated