BPF Compiler Collection (BCC) for Observability, Performance, Monitoring, and Tracing
The BPF Compiler Collection (BCC) is a set of powerful tracing and performance analysis tools for Linux systems that are based on the eBPF (extended Berkeley Packet Filter) virtual machine. These tools allow developers and system administrators to trace and profile kernel and user-space code in a non-intrusive manner, without requiring any special instrumentation or recompilation of the target code.
BCC provides a rich set of pre-built tools that can be used for various purposes, such as tracing system calls, monitoring network traffic, profiling CPU and memory usage, and analyzing performance bottlenecks in real-time.
Some of the most popular BCC tools include:
Tracepoint: A tool for tracing kernel events and functions.
tcptracer: A tool for tracing TCP connections and packets.
opensnoop: A tool for tracing file system events and operations.
execsnoop: A tool for tracing process creation and execution.
funccount: A tool for profiling function calls in user-space code.
biolatency: A tool for measuring block I/O latency.
profile: A tool for profiling CPU usage by sampling the stack trace of running threads.
BCC tools are typically run from the command line and provide real-time output that can be filtered and analyzed using various command-line tools, such as awk and grep. They are widely used in production environments for performance troubleshooting, debugging, and monitoring.
yum install -y kernel-devel-$(uname -r) bcc-tools
The tools provided by the bcc-tools package are installed in the
Active Workload Observability
BCC tools can be launched by running /usr/share/bcc/tools/tool_name
The gethostlatency bcc-tool is showing latency statistics on the getaddrinfo() and gethostbyname() system calls. Effectively, you get to see how long these system calls take and which hostnames the applications need IP addresses for.
Next, important tool is tcplife.
tcplife will tell you data about established tcp connections. PID is the process ID of the process opening the connection. COMM, the command that corresponds to the PID. LADDR and LPORT reference the local address and port of the connection. RADDR and RPORT the remote address and port. TX_KB and RX_KB are the amount of data transmitted (in Kilobytes) and Received (in Kilobytes). MS is the time of the connection, in milliseconds.
File Operations Observability
File operations observability refers to the ability to monitor and analyze file system activities such as file reads, writes, opens, closes, and other related operations. This can be useful for various purposes, such as performance monitoring, security auditing, and troubleshooting.
filetop - This tool will track the count of READS and WRITES as well as the size, R_Kb, and W_Kb, respectively. Additionally, it includes the type, T, of the file and the FILE itself that is interacted with by the command, COMM.
Image: Applications accessing files in real-time
The BCC toolkit provides several tools that can be used for file operations observability. Some of these tools include:
opensnoop: This tool traces file open operations and can be used to monitor which files are being accessed by which processes.
execsnoop: This tool traces process execution and can be used to monitor which processes are accessing which files.
vfsstat: This tool provides statistics on the number of file system operations, such as reads, writes, and syncs, as well as the number of errors encountered.
biolatency: This tool measures block I/O latency and can be used to identify performance bottlenecks related to file system operations.
ext4dist: This tool provides a distribution of file sizes and can be used to monitor file growth and identify large files that may be causing performance issues.
In summary, by using these BCC tools, developers and system administrators can gain deep insights into file system activities and identify issues that may be impacting system performance or security. Additionally, the real-time output and filtering capabilities of these tools can be extremely useful for troubleshooting and identifying the root cause of issues.