Traffic tapping

Envoy currently provides two experimental extensions that can tap traffic:

Tap transport socket configuration

Attention

The tap transport socket is experimental and is currently under active development. There is currently a very limited set of match conditions, output configuration, output sinks, etc. Capabilities will be expanded over time and the configuration structures are likely to change.

Tapping can be configured on Listener and Cluster transport sockets, providing the ability to interpose on downstream and upstream L4 connections respectively.

To configure traffic tapping, add an envoy.transport_sockets.tap transport socket configuration to the listener or cluster. For a plain text socket this might look like:

traffic_tapping_plain_text.yaml
29      transport_socket:
30        name: envoy.transport_sockets.tap
31        typed_config:
32          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tap.v3.Tap
33          common_config:
34            static_config:
35              match:
36                any_match: true
37              output_config:
38                sinks:
39                - format: PROTO_BINARY
40                  file_per_tap:
41                    path_prefix: /some/tap/path
42          transport_socket:
43            name: envoy.transport_sockets.raw_buffer
44            typed_config:
45              "@type": type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer

For a TLS socket, this will be:

traffic_tapping_ssl.yaml
44    transport_socket:
45      name: envoy.transport_sockets.tap
46      typed_config:
47        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tap.v3.Tap
48        common_config:
49          static_config:
50            match:
51              any_match: true
52            output_config:
53              sinks:
54              - format: PROTO_BINARY
55                file_per_tap:
56                  path_prefix: /some/tap/path
57        transport_socket:
58          name: envoy.transport_sockets.tls
59          typed_config:
60            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext

where the TLS context configuration replaces any existing downstream or upstream TLS configuration on the listener or cluster, respectively.

Each unique socket instance will generate a trace file prefixed with path_prefix. E.g. /some/tap/path_0.pb.

Buffered data limits

For buffered socket taps, Envoy will limit the amount of body data that is tapped to avoid OOM situations. The default limit is 1KiB for both received and transmitted data. This is configurable via the max_buffered_rx_bytes and max_buffered_tx_bytes settings. When a buffered socket tap is truncated, the trace will indicate truncation via the read_truncated and write_truncated fields as well as the body truncated field.

Streaming

The tap transport socket supports both buffered and streaming, controlled by the streaming setting. When buffering, SocketBufferedTrace messages are emitted. When streaming, a series of SocketStreamedTraceSegment are emitted.

See the HTTP tap filter streaming documentation for more information. Most of the concepts overlap between the HTTP filter and the transport socket.

PCAP generation

The generated trace file can be converted to libpcap format, suitable for analysis with tools such as Wireshark with the tap2pcap utility, e.g.:

bazel run @envoy_api//tools:tap2pcap /some/tap/path_0.pb path_0.pcap
tshark -r path_0.pcap -d "tcp.port==10000,http2" -P
  1   0.000000    127.0.0.1 → 127.0.0.1    HTTP2 157 Magic, SETTINGS, WINDOW_UPDATE, HEADERS
  2   0.013713    127.0.0.1 → 127.0.0.1    HTTP2 91 SETTINGS, SETTINGS, WINDOW_UPDATE
  3   0.013820    127.0.0.1 → 127.0.0.1    HTTP2 63 SETTINGS
  4   0.128649    127.0.0.1 → 127.0.0.1    HTTP2 5586 HEADERS
  5   0.130006    127.0.0.1 → 127.0.0.1    HTTP2 7573 DATA
  6   0.131044    127.0.0.1 → 127.0.0.1    HTTP2 3152 DATA, DATA