1.29.0 (January 16, 2024)

Incompatible behavior changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • access_log: Added issuer in certificate_properties to the gRPC access log service(AlS).

  • grpc reverse bridge: Handle empty response bodies in grpc_http1_reverse_bridge. This may cause problems for clients expecting the filter to crash for empty responses. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.grpc_http1_reverse_bridge_handle_empty_response to false.

  • http: Flip runtime flag envoy.reloadable_features.no_downgrade_to_canonical_name to true. Name downgrading in the per filter config searching will be disabled by default. This behavior can be temporarily reverted by setting the flag to false explicitly. See doc Http filter route specific config or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples.

  • http: Remove the hop by hop TE header from downstream request headers. This change can be temporarily reverted by setting envoy.reloadable_features.sanitize_te to false.

  • http2: Changes the default value of envoy.reloadable_features.http2_use_oghttp2 to true. This changes the codec used for HTTP/2 requests and responses. This behavior can be reverted by setting the feature to false.

  • http2: Discard the Host header if the :authority header was received to bring Envoy into compliance with https://www.rfc-editor.org/rfc/rfc9113#section-8.3.1 This behavioral change can be reverted by setting runtime flag envoy.reloadable_features.http2_discard_host_header to false.

  • listener: Undeprecated runtime key overload.global_downstream_max_connections until downstream connections monitor extension becomes stable.

  • stats connection_limit: Fixed tag extraction so that stat_prefix is properly extracted. This changes the Prometheus name from envoy_connection_limit_myprefix_limited_connections{} to envoy_connection_limit_limited_connections{envoy_connection_limit_prefix="myprefix"}.

  • stats dns_filter: Fixed tag extraction so that stat_prefix is properly extracted. This changes the Prometheus name from dns_filter_myprefix_local_a_record_answers{} to dns_filter_local_a_record_answers{envoy.dns_filter_prefix="myprefix"}.

  • stats rbac: Fixed tag extraction so that stat_prefix is properly extracted. This changes the Prometheus name from envoy_myprefixrbac_allowed{} to envoy_rbac_allowed{envoy_rbac_prefix="myprefix"}.

Minor behavior changes

Changes that may cause incompatibilities for some users, but should not for most

  • admin: Switch no admin warning -> info.

  • aws: Added support to use HTTP async client to fetch the credentials from EC2 instance metadata and ECS task metadata providers instead of libcurl which is deprecated. By default this behavior is disabled. To enable set envoy.reloadable_features.use_http_client_to_fetch_aws_credentials to true.

  • config parsing, http cache filter: Replaces Protobuf hashing by human-readable string with a dedicated deterministic hashing algorithm. The performance of the hash operation is improved by 2-10x depending on the structure of the message, which is expected to reduce config update time or startup time by 10-25%. The new algorithm is also used for http_cache_filter hashing, which will effectively cause a one-time cache flush on update for users with a persistent cache. To enable this behavior set envoy.restart_features.use_fast_protobuf_hash to true.

  • generic_proxy: Update the stats prefix of generic proxy from <stats_prefix> to generic_proxy.<stats_prefix>.

  • golang: Remove Protocol method from RequestHeaderMap. To get the protocol, please use GetProperty("request.protocol") instead.

  • header_mutation: If per route configuration is configured at multiple levels (route, virtual host, and route table), all specified levels’ mutations are applied. Default order is from least to most specific level (i.e. most specific level wins).

  • http2: Flip the runtime guard envoy.reloadable_features.defer_processing_backedup_streams to be on by default. This feature improves flow control within the proxy by deferring work on the receiving end if the other end is backed up.

  • local_rate_limit: Added new configuration field rate_limited_as_resource_exhausted to allow for setting if rate limit gRPC response should be RESOURCE_EXHAUSTED instead of the default UNAVAILABLE.

Bug fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • DNS: Fixed a race condition that when multiple requests with the same authority header are sent to Envoy, sometimes some requests may receive 503 response with no_healthy_upstream from Envoy. The fix is guarded by runtime guard envoy.reloadable_features.dns_cache_set_first_resolve_complete, which defaults to true.

  • access log: Fixed a bug where the omit_empty_values field was not honored for access logs specifying formats via text_format_source.

  • aws: Default credentials file location was relying on incorrect ~ expansion. This fix now resolves to the correct path by using HOME environment variable, if the credentials file location is not explicitly specified.

  • buffer: Fixed a bug (https://github.com/envoyproxy/envoy/issues/28760) that the internal listener causes an undefined behavior due to the unintended release of the buffer memory.

  • docker: Updated base image to ubuntu:22.04 to fix Redis memory issue (https://github.com/envoyproxy/envoy/issues/31248).

  • dynamic_forward_proxy: Fixed a bug where the preresolved hostnames specified in the Dynamic Forward Proxy cluster config would not use the normalized hostname as the DNS cache key, which is the same key used for retrieval. This caused cache misses on initial use, even though the host DNS entry was pre-resolved. The fix is guarded by runtime guard envoy.reloadable_features.normalize_host_for_preresolve_dfp_dns, which defaults to true.

  • ext_authz: Fixed a bug to ensure the proper functioning of the with_request_body feature within the per-route ExtAuthZ filter.

  • ext_proc: Fixed content_length related issues when body mutation by external processor is enabled. The ext_proc filter removes the content length header in 1)``STREAMED BodySendMode`` 2) BUFFERED_PARTIAL BodySendMode and 3) BUFFERED BodySendMode + SKIP HeaderSendMode. This will enable chunked-encoding whenever feasible in HTTP1.1. Besides, ext_proc filter keep content length header in BUFFERED BodySendMode + SEND HeaderSendMode. It is now external processor’s responsibility to set the content length correctly matched to the mutated body. if those two doesn’t match, the mutation will be rejected and local reply with error status will be returned.

  • grpc: Fixed a bug in gRPC async client cache which intermittently causes CPU spikes due to busy loop in timer expiration.

  • oauth: Add default_expires_in configuration setting. The OAuth spec does not dictate that an authorization server must respond with an expiry. Envoy currently fails any OAuth flow if the expiry is not set. This setting allows you to provide a default in this case to ensure the OAuth flow can succeed.

  • otlp_stat_sink: Fixed a bug where histogram bucket counts were wrong. Additionally, the number of buckets is fixed and is now one element larger than the explicit bounds elements, as required by the specification.

  • quic: Fixed a bug in QUIC and HCM interaction which could cause use-after-free during asynchronous certificates retrieval. The fix is guarded by runtime envoy.reloadable_features.quic_fix_filter_manager_uaf.

  • quic: Fixed a bug in QUIC upstream port migration which could cause use-after-free upon STATELESS_RESET packets.

  • redis: Fixed a bug causing crash if incoming redis key does not match against a prefix_route and catch_all_route is not defined.

  • tracing: Fixed a bug that caused the Datadog tracing extension to drop traces that should be kept on account of an extracted sampling decision.

  • tracing: Fixed a bug where Datadog spans tagged as errors would not have the appropriate error property set.

  • tracing: Fixed a bug where child spans produced by the Datadog tracer would have an incorrect operation name.

  • upstream: Fixed a bug that the subset load balancer will always be used even if the subset load balancer config does not contain any subset selector.

  • upstream: Fixed a bug with upstream auto-config with HTTP/3 where certain network configurations would result in zombie streams left in the grid. Guarded by envoy.reloadable_features.avoid_zombie_streams.

  • xds: Fixed a bug (https://github.com/envoyproxy/envoy/issues/27702) that caused ADS initialization to fail on the first attempt and set a back-off retry interval of up to 1 second, if ADS is using an Envoy Cluster for the backend. The issue was fixed to ensure that ADS initialization happens after the Envoy Cluster it depends upon has been properly initialized. ADS that does not depend on an Envoy Cluster (i.e. GoogleGrpc) is not affected by this change.

Removed config or runtime

Normally occurs at the end of the deprecation period

  • access log: Removed the deprecated envoy.reloadable_features.format_ports_as_numbers runtime flag and legacy code path.

  • grpc: Removed the deprecated envoy.reloadable_features.service_sanitize_non_utf8_strings runtime flag and legacy code path.

  • http: Removed envoy.reloadable_features.expand_agnostic_stream_lifetime and legacy code paths.

  • http: Removed envoy.reloadable_features.sanitize_original_path and legacy code paths.

  • maglev: Removed envoy.reloadable_features.allow_compact_maglev and legacy code paths.

  • router: Removed the deprecated envoy.reloadable_features.ignore_optional_option_from_hcm_for_route_config runtime flag and legacy code path.

  • router: Removed the deprecated envoy.reloadable_features.prohibit_route_refresh_after_response_headers_sent runtime flag and legacy code path.

  • upstream: Removed the deprecated envoy.reloadable_features.validate_detailed_override_host_statuses runtime flag and legacy code path.

New features

  • CEL-attributes: Added attribute connection.transport_failure_reason for looking up connection transport failure reason.

  • access_log: Added new access log command operator %EMIT_TIME% to get the time when the log entry is emitted.

  • access_log: Added support for %CONNECTION_ID% command operator for UDP session access log.

  • access_log: Added support for listener metadata in %METADATA% formatter.

  • attributes: Added support for listener metadata and listener direction in xDS attributes.

  • attributes: Added support for node data in %CEL% formatter.

  • aws: Added support for AWS common utility to fetch metadata credentials from AWS STS by using WebIdentityToken. To enable you need to set envoy.reloadable_features.use_http_client_to_fetch_aws_credentials to true so that web identity credentials provider can use http async client to fetch credentials. Web identity credentials provider cannot use current default libcurl credentials fetcher which is under deprecation and will soon be removed. Web identity credentials provider is not compatible with Grpc Credentials AWS IAM plugin which can only support deprecated libcurl credentials fetcher, see https://github.com/envoyproxy/envoy/pull/30626.

  • aws_request_signing: Added support for specifying the aws signing algorithm, either AWS_SIGV4 or AWS_SIGV4A via signing_algorithm config API.

  • composite filter: Added ExtensionConfiguration discovery service support for composite filter.

  • composite filter: Added composite filter’s matched action name to filter state with key envoy.extensions.filters.http.composite.matched_actions.

  • compression: Added qatzip compressor.

  • ext_authz: Forward filter_metadata selected by route_metadata_context_namespaces and typed_filter_metadata selected by route_typed_metadata_context_namespaces from the metadata of the selected route to external auth service. This metadata propagation is independent from the dynamic metadata from connection and request.

  • ext_authz: New config parameter charge_cluster_response_stats for not incrementing cluster statistics on ext_authz response. Default true, no behavior change.

  • ext_authz_filter: Added with_request_body to optionally override the default behavior of sending the request body to the authorization server from the per-route filter.

  • filter state: Added config name of filter sending a local reply in filter state with key envoy.filters.network.http_connection_manager.local_reply_owner. See the well-known filter state keys for more detail.

  • filters: Added the Basic Auth filter, which can be used to authenticate user credentials in the HTTP Authentication heaer defined in RFC7617.

  • grpc async client: Added max_cached_entry_idle_duration to control the cached gRPC client eviction time in the cache.

  • grpc_http_bridge: Added ignore_query_parameters option for automatically stripping query parameters in request URL path.

  • internal_redirects: Added support to copy headers from the redirect response to the triggered request. See response_headers_to_copy.

  • jwt: The JWT authentication filter supports changing the routes when either the headers or the dynamic metadata are modified.

  • jwt: The JWT filter can now serialize non-primitive custom claims when maping claims to headers. These claims will be serialized as JSON and encoded as Base64.

  • lua: Added Lua extension of router cluster specifier plugin to support selecting cluster dynamically by Lua code.

  • ratelimit: Ratelimit supports optional additional prefix to use when emitting statistics with stat_prefix configuration flag.

  • ratelimit: Ratelimit supports setting the HTTP status that is returned to the client when the ratelimit server returns an error or cannot be reached with status_on_error configuration flag.

  • rbac filter: allow listed HttpAttributesCelMatchInput to be used with the xDS matcher in the RBAC filter.

  • redis: Added support for the getdel command.

  • redis: Added support for the watch command (aborts multi transactions if watched keys change).

  • set_metadata: Added support for injecting typed and untyped dynamic metadata with this filter, also adds the ability to add multiple namespaces with one filter and config to overwrite existing metadata is opt-in. untyped_metadata may now be used to configure the set_metadata filter.

  • stateful_session: Added strict mode to cookie and header based stateful session. If a destination encoded in cookie or in specified header respectively is not available, Envoy will return 503 instead of selecting another destination from the cluster.

  • stats: Added per_endpoint_stats to get some metrics for each endpoint in a cluster.

  • stream info: Added time spent reading request headers to DownstreamTiming.

  • tcp_proxy: Added support to TCP Proxy for recording the latency in UpstreamTiming from when the first initial connection to the upstream cluster was attempted to when either the connection was successfully established or the filiter failed to initialize any connection to the upstream.

  • tracing: Added support for configuring resource detectors on the OpenTelemetry tracer.

  • tracing: Added support to configure a Dynatrace resource detector for the OpenTelemetry tracer.

  • tracing: Added support to configure a sampler for the OpenTelemetry tracer.

  • tracing: Provide initial span attributes to a sampler used in the OpenTelemetry tracer.

  • udp_proxy: Add access log options to allow recording an access log entry periodically for the UDP session, and allow recording an access log entry on the connection tunnel created successfully to upstream when UDP tunneling is configured.

  • udp_proxy: Added support for propagating the response headers in UdpTunnelingConfig and response trailers in UdpTunnelingConfig to the downstream info filter state.

  • upstream: Implmented API drop_overloads which can be used to drop certain percentage of traffic from Envoy.

  • zookeeper: Added support for emitting per opcode decoder error metrics via enable_per_opcode_decoder_error_metrics.

Deprecated

  • set_metadata: metadata_namespace and value are deprecated. Please use the new field untyped_metadata to configure static metadata to inject.

  • wasm: Wasm-specific configuration attributes are deprecated in favor of xds attributes.