1.16.0 (October 8, 2020)

Incompatible Behavior Changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • build: added visibility rules for upstream. If these cause visibility related breakage, see notes in BUILD.

  • build: tcmalloc changes require Clang 9. This requirement change can be avoided by building with –define tcmalloc=gperftools to use the older tcmalloc code.

  • config: additional warnings have been added for the use of v2 APIs. These appear as log messages and are also captured in the deprecated_feature_use counter after server initialization.

  • dns: envoy.restart_features.use_apple_api_for_dns_lookups is on by default. This flag only affects Apple platforms (macOS, iOS). It is incompatible to have the runtime flag set to true at the same time as specifying the use_tcp_for_dns_lookups option or custom dns resolvers. Doing so will cause failure.

  • watchdog: added two guarddogs, breaking the aggregated stats for the single guarddog system. The aggregated stats for the guarddogs will have the following prefixes: main_thread and workers. Concretely, anything monitoring server.watchdog_miss and server.watchdog_mega_miss will need to be updated.

Minor Behavior Changes

Changes that may cause incompatibilities for some users, but should not for most

  • adaptive concurrency: added a response body / grpc-message header for rejected requests.

  • async_client: minor change to handling header only responses more similar to header-with-empty-body responses.

  • build: an Ubuntu based debug image is built and published in DockerHub.

  • build: the debug information will be generated separately to reduce target size and reduce compilation time when build in compilation mode dbg and opt. Users will need to build dwp file to debug with gdb.

  • compressor: always insert Vary headers for compressible resources even if it’s decided not to compress a response due to incompatible Accept-Encoding value. The Vary header needs to be inserted to let a caching proxy in front of Envoy know that the requested resource still can be served with compression applied.

  • decompressor: headers-only requests were incorrectly not advertising accept-encoding when configured to do so. This is now fixed.

  • ext_authz filter: request timeout will now count from the time the check request is created, instead of when it becomes active. This makes sure that the timeout is enforced even if the ext_authz cluster’s circuit breaker is engaged. This behavior can be reverted by setting runtime feature envoy.reloadable_features.ext_authz_measure_timeout_on_check_created to false. When enabled, a new ext_authz.timeout stat is counted when timeout occurs. See stats.

  • grpc reverse bridge: upstream headers will no longer be propagated when the response is missing or contains an unexpected content-type.

  • http: added contains, a new string matcher type which matches if the value of the string has the substring mentioned in contains matcher.

  • http: added contains, a new header matcher type which matches if the value of the header has the substring mentioned in contains matcher.

  • http: added headers_to_add to local reply mapper to allow its users to add/append/override response HTTP headers to local replies.

  • http: added HCM level configuration of error handling on invalid messaging which substantially changes Envoy’s behavior when encountering invalid HTTP/1.1 defaulting to closing the connection instead of allowing reuse. This can temporarily be reverted by setting envoy.reloadable_features.hcm_stream_error_on_invalid_message to false, or permanently reverted by setting the HTTP/1 configuration override_stream_error_on_invalid_http_message to true to restore prior HTTP/1.1 behavior (i.e. connection isn’t terminated) and to retain prior HTTP/2 behavior (i.e. connection is terminated).

  • http: added HCM level configuration of error handling on invalid messaging which substantially changes Envoy’s behavior when encountering invalid HTTP/1.1 defaulting to closing the connection instead of allowing reuse. This can temporarily be reverted by setting envoy.reloadable_features.hcm_stream_error_on_invalid_message to false, or permanently reverted by setting the HCM option to true to restore prior HTTP/1.1 beavior and setting the new HTTP/2 configuration override_stream_error_on_invalid_http_message to false to retain prior HTTP/2 behavior.

  • http: applying route level header modifications to local replies sent on that route. This behavior may be temporarily reverted by setting envoy.reloadable_features.always_apply_route_header_rules to false.

  • http: changed Envoy to send GOAWAY to HTTP2 downstreams when the disable_keepalive overload action is active. This behavior may be temporarily reverted by setting envoy.reloadable_features.overload_manager_disable_keepalive_drain_http2 to false.

  • http: changed Envoy to send error headers and body when possible. This behavior may be temporarily reverted by setting envoy.reloadable_features.allow_response_for_timeout to false.

  • http: changed empty trailers encoding behavior by sending empty data with end_stream true (instead of sending empty trailers) for HTTP/2. This behavior can be reverted temporarily by setting runtime feature envoy.reloadable_features.http2_skip_encoding_empty_trailers to false.

  • http: changed how local replies are processed for requests which transform from grpc to not-grpc, or not-grpc to grpc. Previously the initial generated reply depended on which filter sent the reply, but now the reply is consistently generated the way the downstream expects. This behavior can be temporarily reverted by setting envoy.reloadable_features.unify_grpc_handling to false.

  • http: clarified and enforced 1xx handling. Multiple 100-continue headers are coalesced when proxying. 1xx headers other than {100, 101} are dropped.

  • http: fixed a bug in access logs where early stream termination could be incorrectly tagged as a downstream disconnect, and disconnects after partial response were not flagged.

  • http: fixed the 100-continue response path to properly handle upstream failure by sending 5xx responses. This behavior can be temporarily reverted by setting envoy.reloadable_features.allow_500_after_100 to false.

  • http: the per-stream FilterState maintained by the HTTP connection manager will now provide read/write access to the downstream connection FilterState. As such, code that relies on interacting with this might see a change in behavior.

  • logging: added fine-grain logging for file level log control with logger management at administration interface. It can be enabled by option --enable-fine-grain-logging.

  • logging: changed default log format to “[%Y-%m-%d %T.%e][%t][%l][%n] [%g:%#] %v” and default value of --log-format-prefix-with-location to 0.

  • logging: nghttp2 log messages no longer appear at trace level unless ENVOY_NGHTTP2_TRACE is set in the environment.

  • lua: changed the response body returned by httpCall() API to raw data. Previously, the returned data was string.

  • memory: switched to the new tcmalloc for linux_x86_64 builds. The old tcmalloc can still be enabled with the –define tcmalloc=gperftools option.

  • postgres: changed log format to tokenize fields of Postgres messages.

  • router: added transport failure reason to response body when upstream reset happens. After this change, the response body will be of the form upstream connect error or disconnect/reset before headers. reset reason:{}, transport failure reason:{}.This behavior may be reverted by setting runtime feature envoy.reloadable_features.http_transport_failure_reason_in_body to false.

  • router: now consumes all retry related headers to prevent them from being propagated to the upstream. This behavior may be reverted by setting runtime feature envoy.reloadable_features.consume_all_retry_headers to false.

  • stats: the fake symbol table implemention has been removed from the binary, and the option –use-fake-symbol-table is now a no-op with a warning.

  • thrift_proxy: special characters {‘0’, ‘r’, ‘n’} will be stripped from thrift headers.

  • watchdog: replaced single watchdog with separate watchdog configuration for worker threads and for the main thread configured via Watchdogs. It works with watchdog by having the worker thread and main thread watchdogs have same config.

Bug Fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • csrf: fixed issues with regards to origin and host header parsing.

  • dynamic_forward_proxy: only perform DNS lookups for routes to Dynamic Forward Proxy clusters since other cluster types handle DNS lookup themselves.

  • fault: fixed an issue with active_faults gauge not being decremented for when abort faults were injected.

  • fault: made the HeaderNameValues::prefix() method const.

  • grpc-web: fixed an issue with failing HTTP/2 requests on some browsers. Notably, WebKit-based browsers (https://bugs.webkit.org/show_bug.cgi?id=210108), Internet Explorer 11, and Edge (pre-Chromium).

  • http: fixed CVE-2020-25018 by rolling back the GURL dependency to previous state (reverted: 2d69e30, d828958, and c9c4709 commits) due to potential of crashing when Unicode URIs are present in requests.

  • http: fixed bugs in datadog and squash filter’s handling of responses with no bodies.

  • http: made the HeaderValues::prefix() method const.

  • jwt_authn: supports jwt payload without “iss” field.

  • listener: fixed crash at listener inplace update when connetion load balancer is set.

  • rocketmq_proxy: fixed an issue involving incorrect header lengths. In debug mode it causes crash and in release mode it causes underflow.

  • thrift_proxy: fixed crashing bug on request overflow.

  • udp_proxy: fixed a crash due to UDP packets being processed after listener removal.

Removed Config or Runtime

Normally occurs at the end of the deprecation period

  • http: removed legacy header sanitization and the runtime guard envoy.reloadable_features.strict_header_validation.

  • http: removed legacy transfer-encoding enforcement and runtime guard envoy.reloadable_features.reject_unsupported_transfer_encodings.

  • http: removed configurable strict host validation and runtime guard envoy.reloadable_features.strict_authority_validation.

  • http: removed the connection header sanitization runtime guard envoy.reloadable_features.connection_header_sanitization.

New Features

  • access log: added a dynamic metadata filter for access logs, which filters whether to log based on matching dynamic metadata.

  • access log: added support for %DOWNSTREAM_PEER_FINGERPRINT_1% as a response flag.

  • access log: added support for %CONNECTION_TERMINATION_DETAILS% as a log command operator about why the connection is terminated by Envoy.

  • access log: added support for nested objects in JSON logging mode.

  • access log: added omit_empty_values option to omit unset value from formatted log.

  • access log: added support for %CONNECTION_ID% for the downstream connection identifier.

  • admin: added circuit breakers settings information to GET /clusters?format=json cluster status.

  • admin: added node information to GET /server_info response object.

  • admin: added the ability to dump init manager unready targets information /init_dump and /init_dump?mask={}.

  • admission control: added the admission control filter for client-side request throttling.

  • build: enable building envoy arm64 images by buildx tool in x86 CI platform.

  • cluster: added new connection_pool_per_downstream_connection flag, which enable creation of a new connection pool for each downstream connection.

  • decompressor filter: reports compressed and uncompressed bytes in trailers.

  • dns: added support for doing DNS resolution using Apple’s DnsService APIs in Apple platforms (macOS, iOS). This feature is ON by default, and is only configurable via the envoy.restart_features.use_apple_api_for_dns_lookups runtime key. Note that this value is latched during server startup and changing the runtime key is a no-op during the lifetime of the process.

  • dns_filter: added support for answering service record queries.

  • dynamic_forward_proxy: added use_tcp_for_dns_lookups option to use TCP for DNS lookups in order to match the DNS options for Clusters.

  • ext_authz filter: added support for emitting dynamic metadata for both HTTP and network filters. The emitted dynamic metadata is set by dynamic metadata field in a returned CheckResponse.

  • ext_authz filter: added stat_prefix as an optional additional prefix for the statistics emitted from ext_authz HTTP filter.

  • ext_authz filter: added support for enabling the filter based on dynamic metadata.

  • ext_authz filter: added support for letting the authorization server instruct Envoy to remove headers from the original request by setting the new field headers_to_remove before forwarding it to the upstream.

  • ext_authz filter: added support for sending raw bytes as request body of a gRPC check request by setting pack_as_bytes to true.

  • ext_authz_filter: added disable_request_body_buffering to disable request data buffering per-route.

  • grpc-json: support specifying response_body field in for google.api.HttpBody message.

  • hds: added cluster_endpoints_health to HDS responses, keeping endpoints in the same groupings as they were configured in the HDS specifier by cluster and locality instead of as a flat list.

  • hds: added transport_socket_matches to HDS cluster health check specifier, so the existing match filter transport_socket_match_criteria in the repeated field health_checks has context to match against. This unblocks support for health checks over HTTPS and HTTP/2.

  • hot restart: added --socket-path and --socket-mode to configure UDS path in the filesystem and set permission to it.

  • http: added HTTP/2 support for connection keepalive via PING.

  • http: added support for %DOWNSTREAM_PEER_FINGERPRINT_1% as custom header.

  • http: added allow_chunked_length configuration option for HTTP/1 codec to allow processing requests/responses with both Content-Length and Transfer-Encoding: chunked headers. If such message is served and option is enabled - per RFC Content-Length is ignored and removed.

  • http: added CDN Loop filter and documentation.

  • http: added MaxStreamDuration proto for configuring per-route downstream duration timeouts.

  • http: introduced new HTTP/1 and HTTP/2 codec implementations that will remove the use of exceptions for control flow due to high risk factors and instead use error statuses. The old behavior is used by default for HTTP/1.1 and HTTP/2 server connections. The new codecs can be enabled for testing by setting the runtime feature envoy.reloadable_features.new_codec_behavior to true. The new codecs will be in development for one month, and then enabled by default while the old codecs are deprecated.

  • http: modified the HTTP header-map data-structure to use an underlying dictionary and a list (no change to the header-map API). To conform with previous versions, the use of a dictionary is currently disabled. It can be enabled by setting the envoy.http.headermap.lazy_map_min_size runtime feature to a non-negative number which defines the minimal number of headers in a request/response/trailers required for using a dictionary in addition to the list. Our current benchmarks suggest that the value 3 is a good threshold for most workloads.

  • load balancer: added RingHashLbConfig to configure the table size of Maglev consistent hash.

  • load balancer: added a configuration option to specify the active request bias used by the least request load balancer.

  • load balancer: added an option to optimize subset load balancing when there is only one host per subset.

  • load balancer: added support for bounded load per host for consistent hash load balancers via hash_balance_factor.

  • local_ratelimit: added new HTTP local ratelimit filter.

  • local_reply config: added content_type field to set content-type.

  • lua: added Lua APIs to access SSL connection info object.

  • lua: added Lua API for base64 escaping a string.

  • lua: added Lua API for setting the current buffer content.

  • lua: added new source_code field to support the dispatching of inline Lua code in per route configuration of Lua filter.

  • overload management: add scaling trigger for OverloadManager actions.

  • postgres network filter: metadata is produced based on SQL query.

  • proxy protocol: added support for generating the header upstream using Proxy Protocol Transport Socket.

  • ratelimit: added enable_x_ratelimit_headers option to enable X-RateLimit-* headers as defined in draft RFC.

  • ratelimit: added per route config for rate limit filter.

  • ratelimit: added support for optional descriptor_key to Generic Key action.

  • rbac filter: added the name of the matched policy to the response code detail when a request is rejected by the RBAC filter.

  • rbac filter: added a log action to the RBAC filter which sets dynamic metadata to inform access loggers whether to log.

  • redis: added fault injection support fault injection for redis proxy, described further in configuration documentation.

  • router: added a new rate limited retry back off strategy that uses headers like Retry-After or X-RateLimit-Reset to decide the back off interval.

  • router: added new envoy-ratelimited retry policy, which allows retrying envoy’s own rate limited responses.

  • router: added new host_rewrite_path_regex option, which allows rewriting Host header based on path.

  • router: added support for DYNAMIC_METADATA header formatter.

  • router_check_tool: added support for request_header_matches, response_header_matches to router check tool.

  • signal: added support for calling fatal error handlers without envoy’s signal handler, via FatalErrorHandler::callFatalErrorHandlers().

  • stats: added optional histograms to cluster stats that track headers and body sizes of requests and responses.

  • stats: allow configuring histogram buckets for stats sinks and admin endpoints that support it.

  • tap: added generic body matcher to scan http requests and responses for text or hex patterns.

  • tcp_proxy: added max_downstream_connection_duration for downstream connection. When max duration is reached the connection will be closed.

  • tcp_proxy: allow earlier network filters to set metadataMatchCriteria on the connection StreamInfo to influence load balancing.

  • tls: added OCSP stapling support through the ocsp_staple and ocsp_staple_policy configuration options. See OCSP Stapling for usage and runtime flags.

  • tls: introduce new extension point for overriding TLS handshaker behavior.

  • tls: switched from using socket BIOs to using custom BIOs that know how to interact with IoHandles. The feature can be disabled by setting runtime feature envoy.reloadable_features.tls_use_io_handle_bio to false.

  • tracing: added ability to set some optional segment fields in the AWS X-Ray tracer.

  • udp_proxy: added hash_policies to support hash based routing.

  • udp_proxy: added use_original_src_ip option to replicate the downstream remote address of the packets on the upstream side of Envoy. It is similar to original source filter.

  • watchdog: support randomizing the watchdog’s kill timeout to prevent synchronized kills via a maximium jitter parameter max_kill_timeout_jitter.

  • watchdog: supports an extension point where actions can be registered to fire on watchdog events such as miss, megamiss, kill and multikill. See watchdog actions.

  • watchdog: watchdog action extension that does cpu profiling. See Profile Action.

  • watchdog: watchdog action extension that sends SIGABRT to the stuck thread to terminate the process. See Abort Action.

  • xds: added extension config discovery support for HTTP filters.

  • xds: added support for mixed v2/v3 discovery response, which enable type url downgrade and upgrade. This feature is disabled by default and is controlled by runtime guard envoy.reloadable_features.enable_type_url_downgrade_and_upgrade.

  • zlib: added option to use zlib-ng as zlib library.

Deprecated