[TOC]
本文介绍如何在 Ubuntu 22.04 系统上，通过 IntelliJ IDEA 对 containerd 进行源码级调试。我们将从 containerd 的安装、源码编译、验证调试信息的存在，到最终的调试过程中，每一步骤都进行详细讲解。
1 安装 containerd 📦
首先，按照以下链接中的指引完成 containerd 的安装过程：Ubuntu 22.04 安装 containerd 教程 👈
2 源码编译 🔨
源码编译是调试的前提，具体步骤如下：
克隆 containerd 的 GitHub 仓库：🌐
git clone git@github.com:containerd/containerd.git 🌐
切换到目标分支或版本标签，例如切换到 1.7.2 版本：🔀
git branch v1.7.2 v1.7.2 && git checkout v1.7.2
下载 containerd 的依赖项：📚
cd containerd && go mod tidy
编译 containerd 源码：🏗️
make build GODEBUG=true all
root@containerd:~/workspace/containerd# root@containerd:~/workspace/containerd# root@containerd:~/workspace/containerd# root@containerd:~/workspace/containerd# make build GODEBUG=true all+ build+ bin/ctrgo build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie -o bin/ctr -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd ' -tags "urfave_cli_no_docs static_build" ./cmd/ctr+ bin/containerdgo build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie -o bin/containerd -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd ' -tags "urfave_cli_no_docs static_build" ./cmd/containerd+ bin/containerd-stressgo build -gcflags=all="-N -l" -gcflags=-trimpath=/root/go/src -buildmode=pie -o bin/containerd-stress -ldflags '-X github.com/containerd/containerd/version.Version=v1.7.2-2-gbe3ad13c1 -X github.com/containerd/containerd/version.Revision=be3ad13c14e0e1da2840fc6496f2bcefefb99764 -X github.com/containerd/containerd/version.Package=github.com/containerd/containerd ' -tags "urfave_cli_no_docs static_build" ./cmd/containerd-stress+ bin/containerd-shim+ bin/containerd-shim-runc-v1+ bin/containerd-shim-runc-v2+ binariesroot@containerd:~/workspace/containerd# root@containerd:~/workspace/containerd# root@containerd:~/workspace/containerd# ls -lhtr bin/total 158M-rwxr-xr-x 1 root root 31M Jul 29 14:34 ctr-rwxr-xr-x 1 root root 61M Jul 29 14:34 containerd-rwxr-xr-x 1 root root 29M Jul 29 14:34 containerd-stress-rwxr-xr-x 1 root root 9.4M Jul 29 14:34 containerd-shim-rwxr-xr-x 1 root root 12M Jul 29 14:34 containerd-shim-runc-v1-rwxr-xr-x 1 root root 17M Jul 29 14:34 containerd-shim-runc-v2root@containerd:~/workspace/containerd#
复制代码
注意：编译过程中可能会因为缺少 btrfs 文件系统而报错（linux/btrfs_tree.h: No such file or directory）。遵循本文第一步的安装指导可避免此问题。⚠️
3 验证编译的二进制文件是否包含调试信息 🔍
验证编译是否成功并包含调试信息，可以通过以下任一方法：
3.1 使用 objdump 工具
成功编译的输出示例，会显示含调试信息的输出objdump --syms bin/containerd ✅
root@containerd:~/workspace/containerd# objdump --syms bin/containerd
bin/containerd: file format elf64-x86-64
SYMBOL TABLE:0000000000000000 l df *ABS* 0000000000000000 Scrt1.o00000000000003b4 l O .note.ABI-tag 0000000000000020 __abi_tag0000000000000000 l df *ABS* 0000000000000000 go.go00000000003ad4c0 l F .text 0000000000000000 runtime.text00000000003ad4c0 l F .text 0000000000000059 internal/cpu.Initialize00000000003ad520 l F .text 0000000000000537 internal/cpu.processOptions00000000003ada60 l F .text 0000000000000026 internal/cpu.indexByte00000000003adaa0 l F .text 0000000000000925 internal/cpu.doinit00000000003ae3e0 l F .text 0000000000000006 internal/cpu.isSet00000000003ae400 l F .text 000000000000001b internal/cpu.cpuid.abi000000000003ae420 l F .text 0000000000000011 internal/cpu.xgetbv.abi000000000003ae440 l F .text 0000000000000009 internal/cpu.getGOAMD64level.abi000000000003ae460 l F .text 000000000000007a type:.eq.internal/cpu.option00000000003ae4e0 l F .text 00000000000000e6 type:.eq.[6]internal/cpu.option00000000003ae5e0 l F .text 0000000000000003 runtime/internal/atomic.(*Int32).Load00000000003ae600 l F .text 0000000000000003 runtime/internal/atomic.(*Int32).Store00000000003ae620 l F .text 000000000000000d runtime/internal/atomic.(*Int32).CompareAndSwap00000000003ae640 l F .text 000000000000000a runtime/internal/atomic.(*Int32).Add00000000003ae660 l F .text 0000000000000004 runtime/internal/atomic.(*Int64).Load00000000003ae680 l F .text 0000000000000004 runtime/internal/atomic.(*Int64).Store00000000003ae6a0 l F .text 000000000000000f runtime/internal/atomic.(*Int64).CompareAndSwap00000000003ae6c0 l F .text 0000000000000007 runtime/internal/atomic.(*Int64).Swap00000000003ae6e0 l F .text 000000000000000d runtime/internal/atomic.(*Int64).Add00000000003ae700 l F .text 0000000000000003 runtime/internal/atomic.(*Uint8).Load00000000003ae720 l F .text 0000000000000003 runtime/internal/atomic.(*Uint8).Store00000000003ae740 l F .text 0000000000000004 runtime/internal/atomic.(*Uint8).And00000000003ae760 l F .text 0000000000000004 runtime/internal/atomic.(*Uint8).Or00000000003ae780 l F .text 0000000000000024 runtime/internal/atomic.(*Bool).Load00000000003ae7c0 l F .text 000000000000001f runtime/internal/atomic.(*Bool).Store00000000003ae7e0 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).Load00000000003ae800 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).LoadAcquire00000000003ae820 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).Store00000000003ae840 l F .text 0000000000000003 runtime/internal/atomic.(*Uint32).StoreRelease00000000003ae860 l F .text 000000000000000d runtime/internal/atomic.(*Uint32).CompareAndSwap00000000003ae880 l F .text 000000000000000d runtime/internal/atomic.(*Uint32).CompareAndSwapRelease00000000003ae8a0 l F .text 0000000000000005 runtime/internal/atomic.(*Uint32).Swap00000000003ae8c0 l F .text 0000000000000004 runtime/internal/atomic.(*Uint32).And00000000003ae8e0 l F .text 0000000000000004 runtime/internal/atomic.(*Uint32).Or
复制代码
若输出提示 no symbols，则表示该二进制文件不包含调试信息。❌
root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd# objdump --syms /usr/local/bin/containerd
/usr/local/bin/containerd: file format elf64-x86-64
SYMBOL TABLE:no symbols
root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd#
复制代码
3.2 使用 file 工具
含调试信息的文件将显示 with debug_info, not stripped。✅
root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd# file bin/containerdbin/containerd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9bb291e61e1eceb23359dc29100845e5c1edf763, for GNU/Linux 3.2.0, with debug_info, not strippedroot@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd#
复制代码
若显示 stripped，则不包含调试信息。❌
root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd# file /usr/local/bin/containerd/usr/local/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=322f89b7e351fe2ccfaa0fe30de79c76d49d6e26, for GNU/Linux 3.2.0, strippedroot@containerd:~/workspace/containerd#
复制代码
3.3 dlv 工具验证
正确的调试信息如下所示：✅
root@containerd:~/workspace/containerd# dlv exec bin/containerdType 'help' for list of commands.(dlv)(dlv)(dlv)(dlv)(dlv)(dlv) exitroot@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd#
复制代码
错误的调试信息如下，会提示：no debug info found，如下所示：❌
root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd# dlv exec /usr/local/bin/containerdWarning: no debug info found, some functionality will be missing such as stack traces and variable evaluation.Type 'help' for list of commands.(dlv)(dlv)(dlv)(dlv)(dlv) exit
复制代码
4 debug
debug containerd步骤如下：
1、利用ps -ef|gerp containerd查看containerd启动所需要的参数，这一步特别重要，尤其是在debug k8s源码的时候，k8s的每一个组件都带了很多命令行参数，想要调试这些组件，必须把这些组件的启动参数原封不动的加入到dlv调试命令当中
注意，实际上执行上述命令之后会发现，containerd并没有启动参数，因此无需关心。
2、通过dlv命令启动containerd，启动命令我们可以从IDEA remote debug功能拷贝过来🔍
dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd📡
注意：这里在启动contaienrd的时候没有指定任何参数，实际上也可以根据自己的情况加入containerd参数，譬如指定containerd的配置文件的位置（containerd默认配置文件为：/etc/containerd/config.toml），也可以指定调试的debug级别，譬如：
dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --config=/root/mycontainerd/config.toml --log-level=debug
dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug
3、在IDEA启动debug，连接到远程调试📍
注意：在启动IDEA调试之前，你需要在想要debug的位置增加断点，否则程序启动会直接运行起来，等你这个时候打断点，很可能就晚了。
注意：在启动IDEA调试之前，你需要修改IDEA操作系统标识为Linux
执行dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debug命令之后，此时会阻塞在这里，千万不要使用ctrl + c，只有当IDEA连接上来的时候才会开始执行🚫⌛
root@containerd:~/workspace/containerd#root@containerd:~/workspace/containerd# dlv --listen=:12345 --headless=true --api-version=2 --accept-multiclient exec bin/containerd -- --log-level=debugAPI server listening at: [::]:123452023-07-29T15:06:45+08:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
复制代码
评论