Merge remote-tracking branch 'upstream/master' into merge

Author: pzakha

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "src/cc/libbpf"]
+	path = src/cc/libbpf
+	url = https://github.com/libbpf/libbpf.git

CMakeLists.txt

@@ -9,6 +9,12 @@ endif()
 
 enable_testing()
 
+# populate submodules (libbpf)
+if(NOT EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/src/cc/libbpf/src)
+  execute_process(COMMAND git submodule update --init --recursive
+                  WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+endif()
+
 include(cmake/GetGitRevisionDescription.cmake)
 include(cmake/version.cmake)
 include(CMakeDependentOption)

CONTRIBUTING-SCRIPTS.md

@@ -9,7 +9,7 @@ _(Written by Brendan Gregg.)_
 bcc has 2 types of scripts, in different directories:
 
 - **/examples**: intended as short examples of bcc & eBPF code. You should focus on keeping it short, neat, and documented (code comments). A submission can just be the example code.
-- **/tools**: intended as production safe performance and troubleshooting tools. You should focus on it being useful, tested, low overhead, documented (incl. all caveats), and easy to use. A submission should involve 4 changes: the tool, a man page, an example file, and an addition to README.md. Follow [my lead](https://github.com/brendangregg/bcc/commit/9fa156273b395cfc5505f0fff5d6b7b1396f7daa), and see the checklist below. These will be run in mission critical environments as root, so if spending hours testing isn't for you, please submit your idea as an issue instead, or chat with us on irc.
+- **/tools**: intended as production safe performance and troubleshooting tools. You should focus on it being useful, tested, low overhead, documented (incl. all caveats), and easy to use. A submission should involve 4 changes: the tool, a man page, an example file, and an addition to README.md. Follow [my lead](https://github.com/brendangregg/bcc/commit/9fa156273b395cfc5505f0fff5d6b7b1396f7daa), and see the checklist below. These are run in mission critical environments as root (tech companies, financial institutions, government agencies), so if spending hours testing isn't for you, please submit your idea as an issue instead, or chat with us on irc.
 
 More detail for each below.
 
@@ -31,7 +31,9 @@ A checklist for bcc tool development:
 1. **Measure the overhead of the tool**. If you are running a micro-benchmark, how much slower is it with the tool running. Is more CPU consumed? Try to determine the worst case: run the micro-benchmark so that CPU headroom is exhausted, and then run the bcc tool. Can overhead be lowered?
 1. **Test again, and stress test**. You want to discover and fix all the bad things before others hit them.
 1. **Consider command line options**. Should it have -p for filtering on a PID? -T for timestamps? -i for interval? See other tools for examples, and copy the style: the usage message should list example usage at the end. Remember to keep the tool doing one thing and doing it well. Also, if there's one option that seems to be the common case, perhaps it should just be the first argument and not need a switch (no -X). A special case of this is *stat tools, like iostat/vmstat/etc, where the convention is [interval [count]].
-1. **Concise, intuitive, self-explanatory output**. The default output should meet the common need concisely. Leave much less useful fields and data to be shown with options: -v for verbose, etc. Consider including a startup message that's self-explanatory, eg "Tracing block I/O. Output every 1 seconds. Ctrl-C to end.". Also, try hard to keep the output less than 80 characters wide, especially the default output of the tool. That way, the output not only fits on the smallest reasonable terminal, it also fits well in slide decks, blog posts, articles, and printed material, all of which help education and adoption. Publishers of technical books often have templates they require books to conform to: it may not be an option to shrink or narrow the font to fit your output.
+1. **Concise, intuitive, self-explanatory output**. The default output should meet the common need concisely. Leave much less useful fields and data to be shown with options: -v for verbose, etc. Consider including a startup message that's self-explanatory, eg "Tracing block I/O. Output every 1 seconds. Ctrl-C to end.".
+1. **Default output <80 chars wide**. Try hard to keep the output less than 80 characters wide, especially the default output of the tool. That way, the output not only fits on the smallest reasonable terminal, it also fits well in slide decks, blog posts, articles, and printed material, all of which help education and adoption. Publishers of technical books often have templates they require books to conform to: it may not be an option to shrink or narrow the font to fit your output.
+1. **Short tool name**. Follow the style of the other tools, which follow the style of other /usr/bin utilities. They are short and easy to type. No underscores.
 1. **Use pep8 to check Python style**: pep8 --show-source --ignore=E123,E125,E126,E127,E128,E302 filename . Note that it misses some things, like consistent usage, so you'll still need to double check your script.
 1. **Make sure your script is Python3-ready**: Adding `from __future__ import absolute_import, division, print_function, unicode_literals` helps make your script Python3-ready.
 1. **Write an _example.txt file**. Copy the style in tools/biolatency_example.txt: start with an intro sentence, then have examples, and finish with the USAGE message. Explain everything: the first example should explain what we are seeing, even if this seems obvious. For some people it won't be obvious. Also explain why we are running the tool: what problems it's solving. It can take a long time (hours) to come up with good examples, but it's worth it. These will get copied around (eg, presentations, articles).

INSTALL.md

@@ -8,11 +8,13 @@
   - [Gentoo](#gentoo---portage)
   - [openSUSE](#opensuse---binary)
   - [RHEL](#rhel---binary)
+  - [Amazon Linux 1](#Amazon-Linux-1---Binary)
 * [Source](#source)
   - [Debian](#debian---source)
   - [Ubuntu](#ubuntu---source)
   - [Fedora](#fedora---source)
   - [openSUSE](#opensuse---source)
+  - [Centos](#centos---source)
   - [Amazon Linux](#amazon-linux---source)
 * [Older Instructions](#older-instructions)
 
@@ -30,7 +32,10 @@ CONFIG_NET_CLS_BPF=m
 # [optional, for tc actions]
 CONFIG_NET_ACT_BPF=m
 CONFIG_BPF_JIT=y
+# [for Linux kernel versions 4.1 through 4.6]
 CONFIG_HAVE_BPF_JIT=y
+# [for Linux kernel versions 4.7 and later]
+CONFIG_HAVE_EBPF_JIT=y
 # [optional, for kprobes]
 CONFIG_BPF_EVENTS=y
 ```
@@ -54,7 +59,32 @@ Kernel compile flags can usually be checked by looking at `/proc/config.gz` or
 
 The stable and the nightly packages are built for Ubuntu Xenial (16.04), Ubuntu Artful (17.10) and Ubuntu Bionic (18.04). The steps are very straightforward, no need to upgrade the kernel or compile from source!
 
-**Stable and Signed Packages**
+**Ubuntu Packages**
+
+As of Ubuntu Bionic (18.04), versions of bcc are available in the standard Ubuntu
+multiverse repository. The Ubuntu packages have slightly different names: where iovisor
+packages use `bcc` in the name (e.g. `bcc-tools`), Ubuntu packages use `bpfcc` (e.g.
+`bpfcc-tools`). Source packages and the binary packages produced from them can be
+found at [packages.ubuntu.com](https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=bpfcc&searchon=sourcenames).
+
+```bash
+sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
+```
+
+The tools are installed in `/sbin` (`/usr/sbin` in Ubuntu 18.04) with a `-bpfcc` extension. Try running `sudo opensnoop-bpfcc`.
+
+**_Note_**: the Ubuntu packages have different names but the package contents, in most cases, conflict
+and as such _cannot_ be installed alongside upstream packages. Should one choose to use
+Ubuntu's packages instead of the upstream iovisor packages (or vice-versa), the
+conflicting packages will need to be removed.
+
+The iovisor packages _do_ declare they provide the Ubuntu packages and as such may be
+used to satisfy dependencies. For example, should one attempt to install package `foo`
+which declares a dependency on `libbpfcc` while the upstream `libbcc` package is installed,
+`foo` should install without trouble as `libbcc` declares that it provides `libbpfcc`.
+That said, one should always test such a configuration in case of version incompatibilities.
+
+**Upstream Stable and Signed Packages**
 
 ```bash
 sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
@@ -64,7 +94,7 @@ sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
 ```
 (replace `xenial` with `artful` or `bionic` as appropriate). Tools will be installed under /usr/share/bcc/tools.
 
-**Nightly Packages**
+**Upstream Nightly Packages**
 
 ```bash
 echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list
@@ -73,16 +103,6 @@ sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
 ```
 (replace `xenial` with `artful` or `bionic` as appropriate)
 
-**Ubuntu Packages**
-
-The previous commands will install the latest bcc from the iovisor repositories. It is also available from the standard Ubuntu multiverse repository, under the package name `bpfcc-tools`.
-
-```bash
-sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
-```
-
-The tools are installed in /sbin with a -bpfcc extension. Try running `sudo opensnoop-bpfcc`.
-
 ## Fedora - Binary
 
 Ensure that you are running a 4.2+ kernel with `uname -r`. If not, install a 4.2+ kernel from
@@ -164,6 +184,23 @@ For RHEL 7.6, bcc is already included in the official yum repository as bcc-tool
 yum install bcc-tools
 ```
 
+## Amazon Linux 1 - Binary
+Use case 1. Install BCC for latest kernel available in repo:
+   Tested on Amazon Linux AMI release 2018.03 (kernel 4.14.88-72.73.amzn1.x86_64)
+```
+sudo yum update kernel
+sudo yum install bcc
+sudo reboot
+```
+
+Use case 2. Install BCC for your AMI's default kernel (no reboot required):
+   Tested on Amazon Linux AMI release 2018.03 (kernel 4.14.77-70.59.amzn1.x86_64)
+```
+sudo yum install kernel-headers-$(uname -r | cut -d'.' -f1-5)
+sudo yum install kernel-devel-$(uname -r | cut -d'.' -f1-5)
+sudo yum install bcc
+```
+
 # Source
 
 ## Debian - Source
@@ -355,13 +392,62 @@ sudo make install
 popd
 ```
 
+## Centos - Source
+
+For Centos 7.6 only
+
+### Install build dependencies
+
+```
+sudo yum install -y epel-release
+sudo yum update -y
+sudo yum groupinstall -y "Development tools"
+sudo yum install -y elfutils-libelf-devel cmake3
+sudo yum install -y luajit luajit-devel  # for Lua support
+```
+
+### Install and compile LLVM
+
+```
+curl  -LO  http://releases.llvm.org/7.0.1/llvm-7.0.1.src.tar.xz
+curl  -LO  http://releases.llvm.org/7.0.1/cfe-7.0.1.src.tar.xz
+tar -xf cfe-7.0.1.src.tar.xz
+tar -xf llvm-7.0.1.src.tar.xz
+
+mkdir clang-build
+mkdir llvm-build
+
+cd llvm-build
+cmake3 -G "Unix Makefiles" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
+  -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr ../llvm-7.0.1.src
+make
+sudo make install
+
+cd ../clang-build
+cmake3 -G "Unix Makefiles" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
+  -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr ../cfe-7.0.1.src
+make
+sudo make install
+cd ..
+```
+
+### Install and compile BCC
+
+```
+git clone https://github.com/iovisor/bcc.git
+mkdir bcc/build; cd bcc/build
+cmake .. -DCMAKE_INSTALL_PREFIX=/usr
+make
+sudo make install
+```
+
 ## Amazon Linux - Source
 
 Tested on Amazon Linux AMI release 2018.03 (kernel 4.14.47-56.37.amzn1.x86_64)
 
 ### Install packages required for building
 ```
-# enable epel to get iperf, luajit, luajit-devel, cmake3 (cmake3 is required to support c++11) 
+# enable epel to get iperf, luajit, luajit-devel, cmake3 (cmake3 is required to support c++11)
 sudo yum-config-manager --enable epel
 
 sudo yum install -y bison cmake3 ethtool flex git iperf libstdc++-static python-netaddr gcc gcc-c++ make zlib-devel elfutils-libelf-devel

README.md

@@ -102,7 +102,8 @@ pair of .c and .py files, and some are directories of files.
 - tools/[dbstat](tools/dbstat.py): Summarize MySQL/PostgreSQL query latency as a histogram. [Examples](tools/dbstat_example.txt).
 - tools/[dcsnoop](tools/dcsnoop.py): Trace directory entry cache (dcache) lookups. [Examples](tools/dcsnoop_example.txt).
 - tools/[dcstat](tools/dcstat.py): Directory entry cache (dcache) stats. [Examples](tools/dcstat_example.txt).
-- tools/[deadlock_detector](tools/deadlock_detector.py): Detect potential deadlocks on a running process. [Examples](tools/deadlock_detector_example.txt).
+- tools/[deadlock](tools/deadlock.py): Detect potential deadlocks on a running process. [Examples](tools/deadlock_example.txt).
+- tools/[drsnoop](tools/drsnoop.py): Trace direct reclaim events with PID and latency. [Examples](tools/drsnoop_example.txt).
 - tools/[execsnoop](tools/execsnoop.py): Trace new processes via exec() syscalls. [Examples](tools/execsnoop_example.txt).
 - tools/[ext4dist](tools/ext4dist.py): Summarize ext4 operation latency distribution as a histogram. [Examples](tools/ext4dist_example.txt).
 - tools/[ext4slower](tools/ext4slower.py): Trace slow ext4 operations. [Examples](tools/ext4slower_example.txt).
@@ -118,8 +119,9 @@ pair of .c and .py files, and some are directories of files.
 - tools/[killsnoop](tools/killsnoop.py): Trace signals issued by the kill() syscall. [Examples](tools/killsnoop_example.txt).
 - tools/[llcstat](tools/llcstat.py): Summarize CPU cache references and misses by process. [Examples](tools/llcstat_example.txt).
 - tools/[mdflush](tools/mdflush.py): Trace md flush events. [Examples](tools/mdflush_example.txt).
-- tools/[mysqld_qslower](tools/mysqld_qslower.py): Trace MySQL server queries slower than a threshold. [Examples](tools/mysqld_qslower_example.txt).
 - tools/[memleak](tools/memleak.py): Display outstanding memory allocations to find memory leaks. [Examples](tools/memleak_example.txt).
+- tools/[mountsnoop](tools/mountsnoop.py): Trace mount and umount syscalls system-wide. [Examples](tools/mountsnoop_example.txt).
+- tools/[mysqld_qslower](tools/mysqld_qslower.py): Trace MySQL server queries slower than a threshold. [Examples](tools/mysqld_qslower_example.txt).
 - tools/[nfsslower](tools/nfsslower.py): Trace slow NFS operations. [Examples](tools/nfsslower_example.txt).
 - tools/[nfsdist](tools/nfsdist.py): Summarize NFS operation latency distribution as a histogram. [Examples](tools/nfsdist_example.txt).
 - tools/[offcputime](tools/offcputime.py): Summarize off-CPU time by kernel stack trace. [Examples](tools/offcputime_example.txt).

SPECS/bcc.spec

@@ -88,6 +88,10 @@ popd
 %install
 pushd build
 make install/strip DESTDIR=%{buildroot}
+# mangle shebangs
+find %{buildroot}/usr/share/bcc/{tools,examples} -type f -exec \
+    sed -i -e '1 s|^#!/usr/bin/python$|#!'%{__python}'|' \
+           -e '1 s|^#!/usr/bin/env python$|#!'%{__python}'|' {} \;
 
 %package -n libbcc
 Summary: Shared Library for BPF Compiler Collection (BCC)

cmake/FindCompilerFlag.cmake

@@ -15,3 +15,16 @@ else()
 	endif()
 	set(CMAKE_REQUIRED_FLAGS "${_backup_c_flags}")
 endif()
+
+# check whether reallocarray availability
+# this is used to satisfy reallocarray usage under src/cc/libbpf/
+CHECK_CXX_SOURCE_COMPILES(
+"
+#define _GNU_SOURCE
+#include <stdlib.h>
+
+int main(void)
+{
+        return !!reallocarray(NULL, 1, 1);
+}
+" HAVE_REALLOCARRAY_SUPPORT)

debian/changelog

@@ -1,3 +1,11 @@
+bcc (0.9.0-1) unstable; urgency=low
+
+  * Adds support for BTF
+  * Uses libbpf common library to wrap syscall API
+  * Many bugfixes and new tools
+
+ -- Brenden Blanco <bblanco@gmail.com>  Thu, 07 Mar 2019 17:00:00 +0000
+
 bcc (0.8.0-1) unstable; urgency=low
 
   * Support for kernel up to 5.0

debian/control

@@ -16,6 +16,8 @@ Homepage: https://github.com/iovisor/bcc
 
 Package: libbcc
 Architecture: all
+Provides: libbpfcc, libbpfcc-dev
+Conflicts: libbpfcc, libbpfcc-dev
 Depends: libc6, libstdc++6, libelf1
 Description: Shared Library for BPF Compiler Collection (BCC)
  Shared Library for BPF Compiler Collection to control BPF programs
@@ -28,20 +30,28 @@ Description: Examples for BPF Compiler Collection (BCC)
 
 Package: python-bcc
 Architecture: all
+Provides: python-bpfcc
+Conflicts: python-bpfcc
 Depends: libbcc (= ${binary:Version}), python, binutils
 Description: Python wrappers for BPF Compiler Collection (BCC)
 
 Package: python3-bcc
 Architecture: all
+Provides: python3-bpfcc
+Conflicts: python3-bpfcc
 Depends: libbcc (= ${binary:Version}), python3, binutils
 Description: Python3 wrappers for BPF Compiler Collection (BCC)
 
 Package: bcc-tools
 Architecture: all
+Provides: bpfcc-tools
+Conflicts: bpfcc-tools
 Depends: python-bcc (= ${binary:Version})
 Description: Command line tools for BPF Compiler Collection (BCC)
 
 Package: bcc-lua
 Architecture: all
+Provides: bpfcc-lua
+Conflicts: bpfcc-lua
 Depends: libbcc (= ${binary:Version})
 Description: Standalone tool to run BCC tracers written in Lua