1.32.0 (October 15, 2024)

Incompatible behavior changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • eds: Enabling caching of EDS assignments when used with ADS by default (introduced in Envoy v1.28). Prior to this change, Envoy required that EDS assignments were sent after an EDS cluster was updated. If no EDS assignment was received for the cluster, it ended up with an empty assignment. Following this change, after a cluster update, Envoy waits for an EDS assignment until initial_fetch_timeout times out, and will then apply the cached assignment and finish updating the warmed cluster. This change can be temporarily disabled by setting the runtime flag envoy.restart_features.use_eds_cache_for_ads to false.

  • golang: Change OnLogDownstreamStart, OnLogDownstreamPeriodic and OnLog methods so that user can get the request/response’s headers and trailers when producing access log.

  • http: Access log handlers added by filters will now be evaluated before access log handlers that configured in the access_log configuration. This change can be disabled by setting the runtime guard flag envoy.reloadable_features.filter_access_loggers_first to false.

  • http: Added HTTP/1-safe option for max_connection_duration in HttpConnectionManager. When enabled, max_connection_duration will only drain downstream HTTP/1 connections by adding the Connection: close response header; it will never cause the HttpConnectionManager to close the connection itself. Defaults to off (allows “unsafe” connection closing) but is configurable via http1_safe_max_connection_duration.

  • http: Allow HTTP/2 and HTTP/3 upstream servers to half close the stream before the downstream. This enables bidirectional gRPC streams where server completes streaming before the client. Behavior of HTTP/1 or TCP proxy upstream servers is unchanged and the stream is reset if the upstream server completes response before the downstream. The stream is also reset if the upstream server responds with an error status before the downstream. This behavior is disabled by default and can be enabled by setting the envoy.reloadable_features.allow_multiplexed_upstream_half_close runtime key to true.

  • http: The default configuration of Envoy will continue to trust internal addresses while in the future it will not trust them by default. If you have tooling such as probes on your private network which need to be treated as trusted (e.g. changing arbitrary x-envoy headers) please explictily include those addresses or CIDR ranges into internal_address_config See the config examples from the above internal_address_config link. You can turn up or test the upcoming internal address defaults by setting runtime guard envoy.reloadable_features.explicit_internal_address_config to true.

  • monitoring: Removed runtime feature flag envoy.restart_features.enable_execution_context. The execution context feature can instead be enabled by setting compile option --define=execution_context=enabled.

  • stats scoped_rds: Added new tag extraction so that scoped rds stats have their scope_route_config_name and stat prefix extracted.

  • tracing: Removed support for (long deprecated) opentracing. See issue 27401 for details.

Minor behavior changes

Changes that may cause incompatibilities for some users, but should not for most

  • access_log: Sanitize SNI for potential log injection. The invalid character will be replaced by _ with an invalid: marker. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.sanitize_sni_in_access_log to false.

  • aws: AWS request signing common code now uses the HTTP async client by default, moving curl to the deprecation path. This behavior change can be reverted by setting the envoy_reloadable_features_use_http_client_to_fetch_aws_credentials runtime flag to false.

  • command line options: --enable-fine-grain-logging and --component-log-level were incompatible in that one would make the other ineffective. Setting both options at once is now an error, to reduce potential confusion.

  • conn_handler: Enhanced listener filter chain execution to handle the case that listener filter has maxReadBytes() of 0, but may return StopIteration in onAccept to wait for asynchronous callback.

  • ext_proc: Added support for send_body_without_waiting_for_header_response.

  • http: Modified the authority header value validator to allow the same characters as oghttp2 plus the @ character. This is compliant with nghttp2, and supports the HTTP/1 use-cases that allow user-info@ as part of the authority. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.internal_authority_header_validator to false.

  • http2: Changed the default value of envoy.reloadable_features.http2_use_oghttp2 to false. This changes the codec used for HTTP/2 requests and responses to address to address stability concerns. This behavior can be reverted by setting the feature to true.

  • http3: HTTP/3 alt-svc headers will now be respected from IP-address-based hostnames. This change is guarded by runtime guard envoy.reloadable_features.allow_alt_svc_for_ips.

  • http3: The ACCEPT_UNTRUSTED option now works more consistently for HTTP/3 requests. This change is guarded by envoy.reloadable_features.extend_h3_accept_untrusted.

  • http_11_proxy: Made the inner transport_socket field optional in the proto configuration.

  • lua: When Lua scripts execute httpCall, backpressure is now exercised when receiving body from downstream client. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.lua_flow_control_while_http_call to false.

  • quic: UDP client connection sockets are now connected before use and sockets will only bind if the local address is specified. This behavior change can be reverted by setting the envoy_reloadable_features_quic_connect_client_udp_sockets runtime flag to false.

  • sni: When computing SNI and SAN value for the auto-sni and auto-san verification feature, route host manipulations are now taken into account. This behavior can be reverted by setting the runtime guard envoy_reloadable_features_use_route_host_mutation_for_auto_sni_san to false.

  • tcp: Added support for connection_pool_per_downstream_connection flag in tcp connection pool.

  • tracers: Set status code based on gRPC status code for OpenTelemetry tracers (previously unset).

  • udp: Envoy now sets the Don’t Fragment (DF) flag bit on IP packet header on UDP listener sockets and QUIC upstream connection sockets. This behavior can be reverted by setting envoy.reloadable_features.udp_set_do_not_fragment to false.

  • xds-failover: Add the ability to stick with either the primary or the failover xDS sources once Envoy connects to one of them. To use the prior behavior, set the runtime flag envoy.reloadable_features.xds_failover_to_primary_enabled to false.

Bug fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • c-ares: Applying a C-ares patch to fix DNS resoultion by the Google gRPC library.

  • dispatcher: Update approximate now after polling instead of before polling. This is only used by QUIC. The behavior can be reverted by setting envoy.restart_features.fix_dispatcher_approximate_now to false.

  • dns: The DNS filter no longer returns FORMERR if a message has an ID of 0.

  • dynamic_forward_proxy: Fixed a bug where DFP sub-cluster gets removed due to CDS update and doesn’t gets recreated. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.avoid_dfp_cluster_removal_on_cds_update to false.

  • ext_proc: Add runtime guard for timeout error code 504 Gateway Timeout that is returned to downstream. If runtime flag envoy.reloadable_features.ext_proc_timeout_error is set to false, old error code 500 Internal Server Error will be returned.

  • http: Fixed a bug where an incomplete request (missing body or trailers) may be proxied to the upstream when the limit on the number of requests per I/O cycle is configured and an HTTP decoder filter that pauses filter chain is present. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.use_filter_manager_state_for_downstream_end_stream to false.

  • http2: Fixed bug where an upstream that sent a GOAWAY and gracefully closed a connection would result in an increment of the cluster stat upstream_cx_protocol_error and setting the UpstreamProtocolError response flag. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.http2_no_protocol_error_upon_clean_close to false.

  • http3: Fixed a bug in the CONNECT-UDP forwarding mode where Envoy reset the upstream stream when it received HTTP/3 datagrams before receiving the SETTINGS frame from the upstream peer. Envoy now drops the datagrams in this case instead of resetting the stream.

  • http3: Fixed a bug where an empty trailers block could be sent. This would occur if a filter removed the last trailer - a likely occurrence with the grpc_web_filter. This change makes HTTP/3 behave the same way HTTP/2 does, converting an empty trailers block to no trailers. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.http3_remove_empty_trailers to false.

  • http_async_client: Fixed the local reply and destroy order crashes when using the HTTP async client for websocket handshake.

  • jwt: Fixed a bug where using clear_route_cache with remote JWKs works incorrectly and may cause a crash when the modified request does not match any route.

  • oauth: Set token cookies in response regardless of forward_bearer_token config option.

  • proxy_filter: Fixed a bug in the CONNECT implementation that would cause the CONNECT request created to be invalid when the hostname contains a port number. When the port number is not specified, the port 443 will be automatically added. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.proxy_ssl_port to false.

  • quic: Fixes access log formatter %CONNECTION_ID% for QUIC connections.

  • rbac: RBAC will now allow stat prefixes configured in per-route config to override the base config’s stat prefix.

  • runtime: Fixed an inconsistency in how boolean values are loaded in RTDS, where they were previously converted to 1/0 instead of true/false. The correct string representation (true/false) will now be used. This change can be reverted by setting the runtime guard envoy.reloadable_features.boolean_to_string_fix to false.

  • tracing: Fixed a bug where the OpenTelemetry tracer exports the OTLP request even when no spans are present.

  • upstream: Fixed a bug using hard coded drop category when reporting drop_overload stats to the load report service. It is changed to use drop category that is set in category.

  • upstream: Use previously calculated healthcheck when endpoints move priority levels repeatedly.

  • wasm: Fixed a bug where a body received in chunks is not correctly dumped, resulting in an incomplete dump and loss of the last chunk. This issue is manifested in HTTP/2.

  • websocket: Fixed a bug where the websocket upgrade filter would not take into account per-filter configs.

Removed config or runtime

Normally occurs at the end of the deprecation period

  • DNS: Removed envoy.reloadable_features.dns_cache_set_first_resolve_complete runtime flag and legacy code paths.

  • dynamic forward proxy: Removed envoy.reloadable_features.normalize_host_for_preresolve_dfp_dns runtime flag and legacy code paths.

  • dynamic_foward_proxy: Removed runtime flag envoy.reloadable_features.dfp_mixed_scheme and legacy code path.

  • ext_proc: Removed runtime flag envoy_reloadable_features_immediate_response_use_filter_mutation_rule and legacy code path.

  • ext_proc: Removed runtime flag envoy_reloadable_features_send_header_raw_value and legacy code path.

  • grpc reverse bridge: Removed envoy.reloadable_features.grpc_http1_reverse_bridge_change_http_status runtime flag and legacy code paths.

  • grpc reverse bridge: Removed envoy.reloadable_features.grpc_http1_reverse_bridge_handle_empty_response runtime flag and legacy code paths.

  • http: Removed envoy.reloadable_features.use_http3_header_normalisation runtime flag and legacy code paths.

  • http: Removed runtime flag envoy.reloadable_features.abort_filter_chain_on_stream_reset and legacy code path.

  • http: Removed runtime flag envoy.reloadable_features.http1_connection_close_header_in_redirect and legacy code paths.

  • http: Removed runtime flag envoy.reloadable_features.no_downgrade_to_canonical_name and legacy code path.

  • http: Removed the envoy.reloadable_features.http2_validate_authority_with_quiche runtime flag and its legacy code paths.

  • quic: Removed envoy.reloadable_features.quic_fix_filter_manager_uaf runtime flag and legacy code paths.

  • stateful_session: Removed envoy.reloadable_features.stateful_session_encode_ttl_in_cookie runtime flag and legacy code paths.

  • tls: Removed runtime flag envoy.reloadable_features.ssl_transport_failure_reason_format.

  • udp: Removed envoy.restart_features.udp_read_normalize_addresses runtime flag and legacy code paths.

  • upstream: Removed runtime flag envoy.reloadable_features.avoid_zombie_streams and legacy code paths.

  • upstream: Removed runtime flag envoy.reloadable_features.upstream_allow_connect_with_2xx and legacy code paths.

  • upstream flow control: Removed envoy.reloadable_features.upstream_wait_for_response_headers_before_disabling_read runtime flag and legacy code paths.

New features

  • access log: Added support for %DOWNSTREAM_PEER_CHAIN_FINGERPRINTS_1%, %DOWNSTREAM_PEER_CHAIN_FINGERPRINTS_256%, and %DOWNSTREAM_PEER_CHAIN_SERIALS%, as access log formatters.

  • access_log: Added new JSON access log formatter implementation which provides 16-25x performance improvement over the old implementation. The legacy implementation will be enabled by default for now but will be deprecated and removed in a future release. The sort_properties field will be ignored in the new implementation because the new implementation always sorts properties. And the new implementation will always keep the value type in the JSON output. For example, the duration field will always be rendered as a number instead of a string. The new JSON formatter could be enabled by setting the runtime guard envoy.reloadable_features.logging_with_fast_json_formatter to true.

  • access_log: Added new access log command operators %START_TIME_LOCAL% and %EMIT_TIME_LOCAL%, similar to %START_TIME% and %EMIT_TIME%, but use local time zone.

  • access_log: added %UPSTREAM_CLUSTER_RAW% access log formatter to log the original upstream cluster name, regardless of whether alt_stat_name is set.

  • access_logs: Added support for %UPSTREAM_CONNECTION_ID% access log substitution string in TCP and UDP tunneling flows.

  • apple_dns_impl: For the apple DNS resolver, prefer using IPv6 address when addresses from both families are available. Can be reverted by setting envoy.reloadable_features.prefer_ipv6_dns_on_macos to false.

  • basic_auth: Added support to provide an override authentication_header to load Basic Auth credential.

  • cluster: Allows customizing the happy eyeballs algorithm for an upstream cluster by configuring happy_eyeballs_config. A default configuration will be used if not provided. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.use_config_in_happy_eyeballs to false.

  • dns: for the strict DNS and logical DNS cluster types, the new dns_jitter field, if provided, will causes the cluster to refresh DNS entries later by a random amount of time as to avoid stampedes of DNS requests. This field sets the upper bound (exclusive) for the random amount.

  • ext_authz: Added emit_filter_state_stats which enables filter state stats for access logging.

  • ext_authz: Added config field filter_metadata for injecting arbitrary data to the filter state for logging.

  • ext_proc: Added allow list allowed_override_modes for mode_override.

  • extension_discovery_service: added ECDS support for UDP session filters.

  • formatter: Added full feature absl::FormatTime() support to the DateFormatter. This allows the timepoint formatters (like %START_TIME%) to use %E#S, %E*S, %E#f and %E*f to format the subsecond part of the timepoint.

  • geoip: Added envoy.reloadable_features.mmdb_files_reload_enabled runtime flag that enables reload of mmdb files by default.

  • getaddrinfo: Added num_retries to configure the number of DNS resolution retries. If this field is not provided, the getaddrinfo resolver will retry indefinitely until it succeeds or the DNS query times out. This behavior can be reverted by setting the runtime guard envoy.reloadable_features.getaddrinfo_num_retries to false.

  • getaddrinfo: Added envoy.enable_dfp_dns_trace runtime flag to enable DNS query trace in the DNS resolution details.

  • grpc_field_extraction: Added map<string, string> support: Target fields of type map<string, string> can be extracted and added to dynamic metadata.

  • http: Added configuration setting for the maximum size of response headers in responses. The default can be overridden with runtime key envoy.reloadable_features.max_response_headers_size_kb.

  • http3: http3_protocol_options in HttpConnectionManager has been upgraded to general access.

  • http_11_proxy: Added the option to configure the transport socket via locality or endpoint metadata.

  • jwt_authn: Added missing implementation to jwt_authn matchers to allow glob pattern matching.

  • load_balancing: Added a new WIP implementation of client_side_weighted_round_robin load balancing policy that uses OrcaLoadReport provided by the upstream host to calculate host load balancing weight.

  • local_ratelimit: Add the rate_limits field to generate rate limit descriptors. If this field is set, the VirtualHost.rate_limits or RouteAction.rate_limits fields will be ignored.

  • lua: Added two new methods oidsPeerCertificate() and oidsLocalCertificate() to SSL connection object API SSL connection info object.

  • matching: Added dynamic metadata matcher support Dynamic metadata input and Dynamic metadata input matcher.

  • oauth: Added disable_access_token_set_cookie to disable setting the Access Token cookie.

  • oauth: Added disable_refresh_token_set_cookie to disable setting the Refresh Token cookie.

  • oauth: Added retry_policy to reconnect to the auth server when a connection fails to be established. Added cookie_domain field to OAuth2 filter to allow setting the domain of cookies. Added a nonce to the state parameter in the authorization request to mitigate CSRF attacks. The nonce is generated by the OAuth2 filter and stored in a cookie. This feature is enabled by defaut starting from this release.

  • original_ip_detection extension: The xff original IP detection method now supports using a list of trusted CIDRs when parsing x-forwarded-for.

  • quic: Added QUIC protocol option reject_new_connections to reject connection-establishing packets at the QUIC layer.

  • quic: Added QUIC protocol option save_cmsg_config to optionally specify a CMSG header type to be propagated from the first packet on the connection to QuicListenerFilter.

  • quic: Added max_packet_length to the QUIC protocol options to allow users to change the largest size of a QUIC packet that can be transmitted over the QUIC connection.

  • quic: QUIC server and client support certificate compression, which can in some cases reduce the number of round trips required to setup a connection. This change temporarily disabled by setting the runtime flag envoy.reloadable_features.quic_support_certificate_compression to false.

  • ratelimit: Added the ability to modify hits_addend by setting by setting filter state value envoy.ratelimit.hits_addend to the desired value.

  • rbac: Added delay_deny to support deny connection after the configured duration.

  • redis: Added support for publish.

  • redis_proxy: Added external_auth_provider to support external authentication for redis proxy.

  • resource_monitors: Added possibility to monitor CPU utilization in Linux based systems via cpu utilization monitor in overload manager.

  • sockets: Added socket type field for specifying a socket type to apply the socket option to under SocketOption. If not specified, the socket option will be applied to all socket types.

  • tls: Added prefer_client_ciphers to support enabling client cipher preference instead of server’s for TLS handshakes.

  • tls: Added an extension point custom_tls_certificate_selector to allow overriding TLS certificate selection behavior. An extension can select certificate base on the incoming SNI, in both sync and async mode.

  • udp_access_logs: Added support for %BYTES_RECEIVED%, %BYTES_SENT%, %UPSTREAM_HEADER_BYTES_SENT%, %UPSTREAM_HEADER_BYTES_RECEIVED%, %UPSTREAM_WIRE_BYTES_SENT%, %UPSTREAM_WIRE_BYTES_RECEIVED% access log substitution strings for UDP tunneling flows.