1.27.0 (July 26, 2023)

Incompatible behavior changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • active health check: Preserve the active-health check status of a host after a cluster/assignment update. This is now preserved in cases where the assignment updates a host’s locality. This behavioral change can be temporarily reverted by setting the runtime flag envoy.reloadable_features.keep_endpoint_active_hc_status_on_locality_update to false.

  • build: Moved the subset, ring_hash, and maglev LB code into extensions. If you use these load balancers and override bazel/extensions_build_config.bzl you will need to include them explicitly.

  • build: Moved xDS code extensions. If you use the xDS and override bazel/extensions_build_config.bzl you will need to include the new config_subscriptions explicitly.

  • ext_proc: Apply header mutation rules from the ext_proc config to the ImmediateResponse. This behavior change can be temporarily reverted by setting the runtime guard envoy_reloadable_features_immediate_response_use_filter_mutation_rule to false.

  • ext_proc: The proto field value type is string. This make it unable to support enconding non-utf8 characters in the HeaderValue message. To support sending header value with non-utf8 characters, a new proto field is added in the HeaderValue message: raw_value. The header values are now encoded in this raw_value field when Envoy ext_proc filter sending and receiving messages from the ext_proc server. This behavioral change can be temporarily reverted by setting the runtime guard envoy_reloadable_features_send_header_raw_value to false.

  • http: When append_x_forwarded_host is enabled for a given route action it is now only appended iff it is different from the last value in the list. This resolves issues where a retry caused the same value to be appended multiple times. This behavioral change can be temporarily reverted by setting runtime guard envoy_reloadable_features_append_xfh_idempotent to false.

  • quic: Add a default false runtime flag envoy.reloadable_features.quic_reject_all to disable QUIC listener if needed.

  • stats tls: Fixed metric tag extraction so that TLS parameters are properly extracted from the stats, both for listeners and clusters. This changes the Prometheus names from envoy_listener_ssl_ciphers_ECDHE_RSA_AES128_GCM_SHA256{envoy_listener_address="0.0.0.0_10000"} to envoy_listener_ssl_ciphers{envoy_listener_address="0.0.0.0_10000", envoy_ssl_cipher="ECDHE_RSA_AES128_GCM_SHA256"}, and similar for envoy_listener_ssl_versions_TLSv1_2, envoy_cluster_ssl_versions_TLSv1_2, envoy_listener_ssl_curves_P_256, envoy_cluster_ssl_curves_P_256, envoy_listener_ssl_sigalgs_rsa_pss_rsae_sha256.

Minor behavior changes

Changes that may cause incompatibilities for some users, but should not for most

  • aws: Added support for fetching credentials from the AWS credentials file, which only happens if credentials cannot be fetched from environment variables. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.enable_aws_credentials_file to false.

  • connection pool: Increase granularity mapping connection pool failures to specific stream failure reasons to make it more transparent why the stream is reset when a connection pool’s connection fails.

  • custom response: The filter now traverses matchers from most specific to least specific per filter config till a match is found for the response.

  • dns: Changing the DNS cache to use host:port as the cache key rather than host. This allows a downstream DFP filter to serve both secure and insecure clusters. This behavioral change can be reverted by setting runtime flag envoy.reloadable_features.dfp_mixed_scheme to false.

  • eds: Added the ability to specify mulitple addresses for a host in an EDS cluster. Connections to the host with more than one address will be established using the Happy Eyeballs algorithm.

  • ext_authz: Forward typed_filter_metadata selected by typed_metadata_context_namespaces and filter_metadata selected by metadata_context_namespaces from connection metadata to external auth service. This is addition to the current behavior of forwarding request metadata. In the event of both connection and request metadata containing the requested metadata the request value will be provided.

  • ext_proc: Filter metadata containing ext proc stats has been moved from ext-proc-logging-info to a namespace corresponding to the name of the ext_proc filter.

  • ext_proc: When clear_route_cache is set, ext_proc will check for header mutations beforce clearing the route cache. Failures due to this check will be counted under the clear_route_cache_ignored stat.

  • http: Envoy will now lower case scheme values by default. This behaviorial change can be temporarily reverted by setting runtime guard envoy.reloadable_features.lowercase_scheme to false.

  • http: Round trip time will not be refreshed for every request by default. And if this is necessary, it can be enabled by setting runtime guard envoy.reloadable_features.refresh_rtt_after_request to true.

  • http cookies: Changed internal format of http cookie to protobuf and added expiry timestamp. Processing expired cookie results in selection of a new upstream host and sending a new cookie to the client. Previous format of the cookie is still accepted, but is planned to be obsoleted in the future. This behavior change can be reverted by setting envoy.reloadable_features.stateful_session_encode_ttl_in_cookie to false.

  • http1: Allowing mixed case schemes in absolute urls (e.g. HtTp://www.google.com). Mixed case schemes will be normalized to the lower cased equivalents before being forwarded upstream. This behavior can be reverted by setting runtime flag envoy.reloadable_features.allow_absolute_url_with_mixed_scheme to false.

  • http1: The HTTP1 server-side codec no longer considers encoding 1xx headers as starting the response. This allows the codec to raise protocol errors, sending detailed local replies instead of just closing the connection. This behavior can be reverted by setting runtime flag envoy.reloadable_features.http1_allow_codec_error_response_after_1xx_headers to false.

  • overload manager: Changed behavior of the overload manager to error on unknown overload manager actions. Prior it would silently fail. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.overload_manager_error_unknown_action to false.

  • resource_monitors: Changed behavior of the fixed heap monitor to count unused mapped pages as free memory. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.count_unused_mapped_pages_as_free to false.

  • router: Added check for existing metadata before setting metadata due to auto_sni, auto_san_validation, or override_auto_sni_header to prevent triggering ENVOY_BUG when an earlier filter has set the metadata.

  • stats: Added new type of gauge with type hidden. These stats are hidden from admin/stats-sinks but can shown with a query-parameter of /stats?hidden=include or /stats?hidden=showonly.

  • uhv: Allow malformed URL encoded triplets in the default header validator. This behavior can be reverted by setting runtime flag envoy.reloadable_features.uhv_allow_malformed_url_encoding to false, in which case requests with malformed URL encoded triplets in path are rejected. This setting is only applicable when the Unversal Header Validator is enabled and has no effect otherwise.

  • uhv: Preserve case of %-encoded triplets in the default header validator. This behavior can be reverted by setting runtime flag envoy.reloadable_features.uhv_preserve_url_encoded_case to false, in which case %-encoded triplets are normalized to uppercase characters. This setting is only applicable when the Unversal Header Validator is enabled and has no effect otherwise.

  • upstream: Changed behavior of the unpausing connect with 2xx status codes. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.upstream_allow_connect_with_2xx to false.

Bug fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • boringssl: Fixed the crash that occurs when contrib is compiled with boringssl=fips defined.

  • cors: Fix a use-after-free bug that occurs in the CORS filter if the origin header is removed between request header decoding and response header encoding.

    Fix CVE-2023-35943.

  • dependency: Update C-ares -> 1.91.1 to resolve:

  • dependency: Update Wasmtime and related deps -> 9.0.3 to resolve CVE-2023-30624.

  • ext_authz: Fix a bug where the ext_authz filter will ignore the request body when the pack_as_bytes is set to true and HTTP authorization service is configured.

  • ext_authz: Fix a bug where the ext_authz filter will remove non UTF-8 characters from the body of a request when configured to use http_service, if configured to send the body.

  • http: Switched Envoy internal scheme checks from case sensitive to case insensitive. This behaviorial change can be temporarily reverted by setting runtime guard envoy.reloadable_features.handle_uppercase_scheme to false.

    Fix CVE-2023-35944.

  • http: The is_optional field of HTTP filter can only be used for configuration loading of HTTP filter and will be ignored for loading of route or virtual host level filter config. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.ignore_optional_option_from_hcm_for_route_config to false. You can also use route/virtual host optional flag as a replacement of the feature.

  • http2: Fix memory leak in nghttp2 when scheduled requests are cancelled due to the GOAWAY frame being received from the upstream service.

  • logging: Do not display GRPC_STATUS_NUMBER for non gRPC requests. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.validate_grpc_header_before_log_grpc_status to false.

  • maglev loadbalancer: Fixes maglev stability problem. Previously, maglev returns slightly different backend assignment from the same backends and keys.

  • oauth2: Fixed a cookie validator bug that meant the HMAC calculation could be the same for different payloads.

    This prevents malicious clients from constructing credentials with permanent validity in some specific scenarios.

  • oauth2: The Max-Age attribute of Set-Cookie HTTP response header was being assigned a value representing Seconds Since the Epoch, causing cookies to expire in ~53 years. This was fixed an now it is being assigned a value representing the number of seconds until the cookie expires. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.oauth_use_standard_max_age_value to false.

  • oauth2: The httpOnly attribute for Set-Cookie for tokens in HTTP response header was missing, causing tokens to be accessible from the JavaScript making the apps vulnerable. This was fixed now by marking the cookie as httpOnly. This behavioral change can be temporarily reverted by setting runtime guard envoy.reloadable_features.oauth_make_token_cookie_httponly to false.

    Fix CVE-2023-35941.

  • opentelemetry/grpc/access log: Fixed a bug in the open telemetry access logger. This logger now uses the server scope for stats instead of the listener’s global scope. This fixes a use-after-free that can occur if the listener is drained but the cached gRPC access logger uses the listener’s global scope for stats.

    Fix CVE-2023-35942.

  • original_dst: Fixes an issue with the ORIGINAL_DST cluster cleanup timer lifetime, which can occur if the cluster is removed while the timer is armed.

  • postgres: Enable parsing when using upstream SSL.

  • redis: Fixes a bug where redis transactions do not work properly when redis traffic is mirrored.

  • redis_proxy: Fixes a bug where route properties such as key_formatter, prefix and remove_prefix do not take effect when configured for catch_all_route.

  • router: Fixed the bug that updating scope_key_builder of SRDS config doesn’t work and multiple HCM share the same scope_key_builder.

  • tcp_proxy: Fixed assert crash when multiple readDisable are called for TCP tunneling scenarios, by allowing multiple calls. This will also cause stats that indicate disable or enable of downstream read to be flushed only once per actual disabling or enabling.

  • tls: Fix build FIPS compliance when using both FIPS mode and Wasm extensions (--define boringssl=fips and --define wasm=v8).

  • upstream: Fixes a bug where the healthStatus() method of host return incorrect health status when the host status is updated by the EDS.

  • upstream: Fixes a bug where the healthStatus() method of host return unmatched health status with the coarseHealth() method.

Removed config or runtime

Normally occurs at the end of the deprecation period

  • config: Removed runtime key envoy.reloadable_features.delta_xds_subscription_state_tracking_fix and legacy code paths.

  • grpc_stats: Removed runtime key envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default and legacy code paths.

  • header_formatters: Removed runtime key envoy.reloadable_features.unified_header_formatter and legacy code paths.

  • http: Removed runtime key envoy.reloadable_features.allow_upstream_filters and legacy code paths.

  • http: Removed runtime key envoy.reloadable_features.closer_shadow_behavior and legacy code paths.

  • http: Removed runtime key envoy.reloadable_features.http_response_half_close and legacy code paths.

  • http: Removed runtime key envoy.reloadable_features.http_strip_fragment_from_path_unsafe_if_disabled and legacy code paths.

  • logging: Removed runtime key envoy.reloadable_features.correct_remote_address and legacy code paths.

  • quic: Removed runtime key envoy.reloadable_features.quic_defer_send_in_response_to_packet and legacy code paths.

  • tls: Remove runtime key envoy.reloadable_features.tls_async_cert_validation and legacy code paths.

  • udp: Removed runtime key envoy.reloadable_features.udp_proxy_connect and legacy code paths.

  • upstream: Removed runtime key envoy.reloadable_features.fix_hash_key and legacy code paths.

New features

  • access_log: (QUIC only) Added support for %BYTES_RETRANSMITTED% and %PACKETS_RETRANSMITTED%.

  • access_log: Added CEL access log formatter to print CEL expression.

  • access_log: Added DisableBuiltinLables to disable envoy builtin resource labels.

  • access_log: Added %ACCESS_LOG_TYPE% substitution string, to help distinguishing between access log records and when they are being recorded. Please refer to the access log configuration documentation for more information.

  • access_log: Added access log filter log_type_filter to filter access log records based on the type of the record.

  • access_log: Added additional HCM access log option flush_log_on_tunnel_successfully_established. Enabling this option will write a log to all access loggers when HTTP tunnels (e.g. Websocket and CONNECT) are successfully established.

  • access_log: Added bytes snapshotting for upstream and downstream logging that will be reset after every periodic log. Downstream periodic loggers should read BytesMeter::bytesAtLastDownstreamPeriodicLog(), and upstream periodic loggers should read BytesMeter::bytesAtLastUpstreamPeriodicLog().

  • admin: Adds a new admin stats html bucket-mode detailed to generate all recorded buckets and summary percentiles.

  • application_logs: Added bootstrap option application_log_format to enable setting application log format as JSON structure.

  • application_logs: Added bootstrap option application_log_format to enable setting application log text format from config.

  • config: added a statistic warming_state to indicate the current warming state of a cluster.

  • dynamic_forward_proxy: Added sub_clusters_config to enable independent sub cluster for each host:port, with STRICT_DNS cluster type.

  • ext_proc: forward_rules to only allow headers matching the rules to be forwarded to the external processing server.

  • ext_proc: Added new configuration field allow_mode_override If set to true, the filter config processing_mode can be overridden by the mode_override in the response message from the external processing server. If not set, the mode_override API in the response message will be ignored.

  • ext_proc: Added new configuration field disable_clear_route_cache to force the ext_proc filter from clearing the route cache. Failures to clear from setting this field will be counted under the clear_route_cache_disabled stat.

  • ext_proc: Added new field filter_metadata <envoy_v3_api_field_extensions.filters.http.ext_proc.v3.ExtProc.filter_metadata to aid in logging. Metadata will be stored in StreamInfo filter metadata under a namespace corresponding to the name of the ext proc filter.

  • fault: Added new field envoy.extensions.filters.http.fault.v3.HTTPFault.filter_metadata to aid in logging. Metadata will be stored in StreamInfo dynamic metadata under a namespace corresponding to the name of the fault filter.

  • golang: Added new l4 golang network filter.

  • health_check: Added host related information metadata and locality to the health check event definition.

  • http: Add support to the route/virtual host level is_optional field. A route/virtual host level per filter config can be marked as optional, which means that if the filter fails to load, the configuration will no be rejected.

  • http: Added outbound_control_frames_active and outbound_frames_active statistic.

  • http: Added OVERWRITE_IF_EXISTS header manipulation keyword to overwrite a header only when it exists before manipulation.

  • http: Added runtime feature envoy.reloadable_features.max_request_headers_size_kb to override the default value of max request headers size.

  • http: Added support for CONNECT-UDP (RFC 9298). Can be disabled by setting runtime feature envoy.reloadable_features.enable_connect_udp_support to false.

  • http: Added support for configuring additional cookie attributes.

  • http: Added support for the route/virtual host level disabled field. A route/virtual host level per filter config can be marked as disabled, which means that the filter will be disabled in a specific route/virtual host.

  • lds: Pause SRDS when LDS is updated.

  • listeners: Added max_connections_to_accept_per_socket_event that sets the maximum number of new connections to be accepted per socket event on a listener. If there are more connections to be accepted beyond the maximum, the remaining connections would be processed in later dispatcher loop iterations. Added listener histogram connections_accepted_per_socket_event to allow users to empirically determine an appropriate configuration for their deployment.

  • load shed point: Added load shed point envoy.load_shed_points.http1_server_abort_dispatch that rejects HTTP1 server processing of requests.

  • load shed point: Added load shed point envoy.load_shed_points.http2_server_go_away_on_dispatch that sends GOAWAY for HTTP2 server processing of requests. When a GOAWAY frame is submitted by this the counter http2.goaway_sent will be incremented.

  • load shed point: Added load shed point envoy.load_shed_points.http_connection_manager_decode_headers that rejects new http streams by sending a local reply.

  • load_balancing: Added new option weighted_priority_health to compute the health of a priority level by using load balancing weight instead of the count of healthy hosts.

  • match_delegate: Added per route configuration to the ExtensionWithMatcher filter. Which allows the associated matcher to be defined on a per route basis.

  • match_delegate: If no matcher is set the ExtensionWithMatcher filter is now set to skip rather than erroring out.

  • matchers: Added RuntimeFraction input matcher. It allows matching hash of the input on a runtime key.

  • matching: Added CEL(Common Expression Language) matcher support CEL data input and CEL input matcher.

  • original_dst: Filter state is pulled from request context first (if available), then falls back to connection context. Added ability to pick host from dynamic metadata using metadata_key. Same behavior - looks in request context first (if available), falls back to connection context.

  • ratelimit: Added new configuration field domain to allow for setting rate limit domains on a per-route basis.

  • redis_proxy: Added new configuration field key_formatter to format redis key. The field supports using %KEY% as a formatter command for substituting the redis key as part of the substitution formatter expression.

  • redis_proxy: Added new field connection_rate_limit to limit reconnection rate to redis server to avoid reconnection storm.

  • stat_sinks: Added envoy.stat_sinks.open_telemetry stats_sink, that supports flushing metrics by the OTLP protocol, for supported Open Telemetry collectors.

  • stats: Added config enable_deferred_creation_stats. When set to true, enables deferred instantiation on supported stats structures.

  • tls: Added FIPS compliant build for arm64.

  • tls: Added support for hot-reloading CRL file when the file changes on disk. This works with dynamic secrets when CertificateValidationContext is delivered via SDS.

  • tls: Added support to configure the new config option enforce_rsa_key_usage. This can be used to override its configuration in BoringSSL. It is currently default to false but expected to be changed to true by default in a future release. ssl.was_key_usage_invalid is added to listener metrics and will be incremented for certificate configurations that would fail if this option were set to true.

  • tls_inspector: Added histogram bytes_processed which records the number of bytes of the tls_inspector processed while analyzing for tls usage. In cases where the connection uses tls this records the tls client hello size. In cases where the connection doesn’t use tls this records the amount of bytes the tls_inspector processed until it realized the connection was not using tls.

  • tls_inspector: Added new configuration field initial_read_buffer_size to allow users to tune the buffer size requested by the filter. If configured, and the filter needs additional bytes, the filter will double the number of bytes requested up to the default 64KiB maximum.

  • upstream: Added cluster provided extension to suppport the load balancer policy.

  • zookeeper: Added the addWatch opcode support to the ZooKeeper proxy filter.

Deprecated