In Part 2, we learned the anatomy of an ELF core dump — the header, the program headers, and the four types of notes that GDB needs. We know what to write. Now the question is: where do we get the data?

The answer is five files in /proc/<pid>/. That's it. Five virtual files, and we can reconstruct everything GDB needs.

The /proc Filesystem as Our Backdoor

Let's be very precise about why this works. When you read /proc/<pid>/maps, the kernel doesn't send a signal to the process. It doesn't ask the process to cooperate. It doesn't use ptrace. It simply walks its own internal data structures — the mm_struct, the vm_area_struct linked list, the task list — and formats the results as text.

The process doesn't even know you're reading its data. It can't block you because it's not involved. This is fundamentally different from ptrace, which requires the target to stop and acknowledge.

This is why /proc works on D-state processes. The kernel is reading its own bookkeeping, not asking the process to do anything.

Memory Maps: /proc/<pid>/maps

This file gives us the complete memory layout of the process. Every loaded library, every heap allocation, every stack — it's all here:

$ cat /proc/1234/maps
55a8b1000000-55a8b1025000 r--p 00000000 08:01 1234567  /usr/bin/cvmfs2
55a8b1025000-55a8b10a0000 r-xp 00025000 08:01 1234567  /usr/bin/cvmfs2
55a8b10a0000-55a8b10b0000 r--p 000a0000 08:01 1234567  /usr/bin/cvmfs2
55a8b10b0000-55a8b10b2000 rw-p 000af000 08:01 1234567  /usr/bin/cvmfs2
7f4a10000000-7f4a10021000 rw-p 00000000 00:00 0        [heap]
7f4a12340000-7f4a12380000 r--p 00000000 08:01 2345678  /usr/lib64/libc.so.6
7f4a12380000-7f4a12500000 r-xp 00040000 08:01 2345678  /usr/lib64/libc.so.6
...
7ffc8a000000-7ffc8a021000 rw-p 00000000 00:00 0        [stack]
7ffc8a1f0000-7ffc8a1f4000 r--p 00000000 00:00 0        [vvar]
7ffc8a1f4000-7ffc8a1f6000 r-xp 00000000 00:00 0        [vdso]

Each line is one memory region. The format is:

start-end  perms  offset  dev  inode  pathname

We parse this to build our mem_region_t array:

int parse_maps(int pid, mem_region_t **regions_out) {
    char path[64];
    snprintf(path, sizeof(path), "/proc/%d/maps", pid);
    FILE *f = fopen(path, "r");

    char line[512];
    while (fgets(line, sizeof(line), f)) {
        unsigned long start, end, offset;
        char perms[5], name[256] = "";

        sscanf(line, "%lx-%lx %4s %lx %*s %*s %255[^\n]",
               &start, &end, perms, &offset, name);

        regions[count].start      = start;
        regions[count].end        = end;
        regions[count].offset     = offset;
        regions[count].readable   = (perms[0] == 'r');
        regions[count].writable   = (perms[1] == 'w');
        regions[count].executable = (perms[2] == 'x');
        strncpy(regions[count].name, name, 255);

        count++;
    }
    return count;
}

What we learn from each region:

Field What It Tells Us
start–end Virtual address range → becomes p_vaddr in PT_LOAD
perms (rwxp) Permissions → becomes p_flags in PT_LOAD
offset File offset → goes into NT_FILE entries
pathname Which file is mapped → goes into NT_FILE, also used for dump level filtering

The First Region Trick

The first file-backed region in /proc/pid/maps is always the main executable. We save this path so that in standard dump mode, we can always keep the executable's memory while skipping other libraries:

if (g_exe_path[0] == '\0' && name[0] == '/') {
    strncpy(g_exe_path, name, sizeof(g_exe_path) - 1);
}

Regions We Skip

Some kernel-mapped regions cause EIO (Input/Output error) when you try to read them via /proc/pid/mem. We filter these out:

static int should_dump_region(const mem_region_t *r) {
    if (!r->readable) return 0;
    if (strstr(r->name, "[vvar]"))        return 0;  // kernel clock data
    if (strstr(r->name, "[vvar_vclock]")) return 0;  // variant clock
    if (strstr(r->name, "[vsyscall]"))    return 0;  // legacy syscall page
    // ... dump level logic follows
}

[vvar] is a kernel-mapped page that provides fast clock access. It's readable in userspace, but reading it through /proc/pid/mem from another process triggers EIO. Same for [vsyscall]. We skip these because they contain no useful diagnostic data anyway.

CPU Registers: /proc/<pid>/task/<tid>/syscall

This is the most interesting data source. When a thread is blocked in a system call, the kernel records the syscall state in /proc/<pid>/task/<tid>/syscall:

$ cat /proc/1234/task/1234/syscall
0 0x7 0x7f4a12345678 0x1000 0x0 0x0 0x0 0x7ffc89ffee80 0x7f4a12367890

The format is:

syscall_nr  arg0  arg1  arg2  arg3  arg4  arg5  RSP  RIP

Those last two values are gold: RSP (stack pointer) and RIP (instruction pointer) — the two registers GDB needs most.

But wait — there are 27 registers in the x86_64 user_regs_struct, and we only get 9 values. Where do the rest come from?

Mapping Syscall Arguments to Registers

The Linux x86_64 syscall calling convention defines which register carries which argument:

// /proc/pid/syscall format:
// syscall_nr  a0   a1   a2   a3   a4   a5   RSP  RIP
//
// Linux x86_64 syscall convention:
// a0 = RDI, a1 = RSI, a2 = RDX, a3 = R10, a4 = R8, a5 = R9

memset(regs, 0, 27 * sizeof(unsigned long));

regs[15] = nr;   // ORIG_RAX — which syscall was called
regs[10] = nr;   // RAX — return value (set to syscall nr while blocked)
regs[14] = a0;   // RDI — 1st argument
regs[13] = a1;   // RSI — 2nd argument
regs[12] = a2;   // RDX — 3rd argument
regs[7]  = a3;   // R10 — 4th argument
regs[9]  = a4;   // R8  — 5th argument
regs[8]  = a5;   // R9  — 6th argument
regs[19] = rsp;  // RSP — stack pointer
regs[16] = rip;  // RIP — instruction pointer

// User-mode segment selectors (constant for all userspace code)
regs[17] = 0x33; // CS — code segment
regs[20] = 0x2b; // SS — stack segment

The registers we can't get from /proc/pid/syscall (like R11, R12, R13, R14, R15, RBX) are left as zero. GDB handles this gracefully — it just shows <unavailable> for those values. The important ones (RIP, RSP, and the syscall arguments) are all present.

But there's one critical register missing: RBP.

The RBP Problem and Heuristic Recovery

Why RBP Matters

When you compile C code with frame pointers (the default on most Linux distributions), every function creates a frame pointer chain on the stack:

+-----------------+  <-- higher addresses (stack bottom)
|   saved RBP     |  --- points to caller's frame
|   return addr   |  --- where to return after this function
|   local vars    |
+-----------------+
|   saved RBP     |  --- points to parent's frame
|   return addr   |  --- where parent was executing
|   local vars    |
+-----------------+
|   saved RBP     |  --- points to grandparent's frame
|   return addr   |  --- where grandparent was executing
|   local vars    |
+-----------------+
|       ...       |
+-----------------+  <-- lower addresses (RSP points here)

GDB walks this chain to produce a backtrace:

  1. Read RBP → that's the current frame
  2. At address RBP, read the saved RBP → that's the caller's frame
  3. At address RBP + 8, read the return address → that's where the caller was executing
  4. Repeat until saved RBP is 0 (top of the call chain)

Without RBP, GDB can only show you frame 0 (the current function from RIP) and frame 1 (by scanning the stack for the return address). The rest of the backtrace is lost.

The Problem

/proc/<pid>/task/<tid>/syscall gives us RSP and RIP but not RBP. The kernel simply doesn't expose it through this interface.

For some binaries, this doesn't matter — if they were compiled with DWARF .eh_frame unwind information (which most modern code is), GDB can use the CFI (Call Frame Information) tables to unwind the stack without RBP. But for code compiled with -fomit-frame-pointer and without DWARF info, or in complex cases where DWARF and frame pointers interact, having the real RBP makes backtraces significantly more reliable.

The Heuristic

Our approach: scan the stack for values that look like valid frame pointers, and pick the one with the longest valid chain.

Here's the algorithm:

1. Find the stack region that contains RSP
2. Read 4KB of stack above RSP into a local buffer
3. For every 8-byte-aligned value on the stack:
   a. Does it point to a valid stack address? If not, skip.
   b. Read 16 bytes at that address from /proc/pid/mem:
      - First 8 bytes = saved_rbp
      - Next 8 bytes = return_address
   c. Is the return_address inside an executable memory region? If not, skip.
   d. Walk the chain: does saved_rbp also point to a valid frame?
   e. Count the chain depth
4. The candidate with the deepest valid chain wins

Let's look at the key functions:

Validation: Is This a Real Return Address?

static int is_executable_addr(unsigned long addr,
                              const mem_region_t *regions,
                              int num_regions) {
    for (int i = 0; i < num_regions; i++) {
        if (regions[i].executable &&
            addr >= regions[i].start &&
            addr < regions[i].end)
            return 1;
    }
    return 0;
}

If *(candidate + 8) falls inside an executable region (like .text of libc or cvmfs2), it's likely a real return address. Random data on the stack almost never happens to fall inside executable pages.

Chain Walking: How Deep Does It Go?

static int walk_fp_chain(unsigned long rbp_candidate,
                         int mem_fd,
                         unsigned long stack_lo,
                         unsigned long stack_hi,
                         const mem_region_t *regions,
                         int num_regions,
                         int max_depth) {
    unsigned long fp = rbp_candidate;
    int depth = 0;

    while (depth < max_depth) {
        unsigned long frame[2];  // [saved_rbp, return_addr]
        ssize_t n = pread(mem_fd, frame, 16, (off_t)fp);
        if (n < 16) break;

        unsigned long saved_rbp = frame[0];
        unsigned long ret_addr  = frame[1];

        // Return address must be in executable memory
        if (!is_executable_addr(ret_addr, regions, num_regions))
            break;

        depth++;

        if (saved_rbp == 0) break;           // Top of chain (main)
        if (saved_rbp <= fp) break;           // Must move upward in stack
        if (saved_rbp < stack_lo ||
            saved_rbp >= stack_hi) break;     // Must stay on stack

        fp = saved_rbp;  // Follow the chain
    }

    return depth;
}

The beauty of this approach is that false positives are extremely rare. For a random value on the stack to pass all these checks, it would need to:

  1. Point to another stack address
  2. Where the next 8 bytes happen to be inside executable memory
  3. And the 8 bytes before that point to yet another valid stack+code pair
  4. And this chain would need to continue for multiple levels

The chance of this happening with garbage data is astronomically small. In practice, the heuristic finds the correct RBP on the first try for frame-pointer-compiled code.

flowchart TD A["Read 4KB of stack above RSP"] --> B["For each 8-byte value"] B --> C{"Points to\nstack address?"} C -->|No| B C -->|Yes| D{"*(val+8) in\nexecutable memory?"} D -->|No| B D -->|Yes| E["Walk chain:\nfollow saved_rbp pointers"] E --> F{"Chain depth\n> best so far?"} F -->|No| B F -->|Yes| G["New best candidate"] G --> B B -->|"Done scanning"| H["Return best RBP\n(deepest chain wins)"] style G fill:#cfc,stroke:#393 style H fill:#cfc,stroke:#393

The Result

When RBP recovery succeeds, the difference is dramatic:

Without RBP recovery:

(gdb) bt
#0  __internal_syscall_cancel (...) at cancellation.c:44
#1  __pthread_cond_wait_common (cond=..., mutex=...) at pthread_cond_wait.c:421

Two frames. We know it's stuck in a condition wait, but we have no idea why it's waiting, or what the full call chain looks like.

With RBP recovery:

(gdb) bt
#0  __internal_syscall_cancel (...) at cancellation.c:44
#1  __pthread_cond_wait_common (cond=..., mutex=...) at pthread_cond_wait.c:421
#2  ___pthread_cond_wait (...) at pthread_cond_wait.c:453
#3  Tube<download::DataTubeElement>::PopFront (...) at .../cvmfs/util/tube.h:125
#4  download::DownloadManager::Fetch (...) at .../cvmfs/network/download.cc:2045
#5  cvmfs::Fetcher::Fetch (...) at .../cvmfs/fetch.cc:177
#6  catalog::ClientCatalogManager::StageNestedCatalogByHash (...) at .../cvmfs/catalog_mgr_client.cc:379
#7  catalog::AbstractCatalogManager::StageNestedCatalogAndUnlock (...) at .../cvmfs/catalog_mgr_impl.h:865
#8  catalog::AbstractCatalogManager::LookupPath (...) at .../cvmfs/catalog_mgr_impl.h:287
#9  cvmfs::GetDirentForPath (path=...) at .../cvmfs/cvmfs.cc:428
#10 cvmfs::cvmfs_lookup (req=..., parent=55380, name=0x... "alma9-alice-20260427_2") at .../cvmfs/cvmfs.cc:580
#11 loader::stub_lookup (...) at .../cvmfs/loader.cc:217

Twelve frames. Now we can see the entire story: a FUSE lookup → catalog manager → download manager → blocking PopFront() on a tube waiting for a catalog download. That's the information a developer needs to fix the bug.

Thread Enumeration: /proc/<pid>/task/

Each subdirectory in /proc/<pid>/task/ is a thread ID:

$ ls /proc/1234/task/
1234  1235  1236  1237  1238  1239  1240

We enumerate all of them and read registers for each:

int enumerate_threads(int pid, thread_info_t **threads_out) {
    char path[64];
    snprintf(path, sizeof(path), "/proc/%d/task", pid);
    DIR *dir = opendir(path);

    struct dirent *entry;
    while ((entry = readdir(dir))) {
        if (entry->d_name[0] == '.') continue;

        int tid = atoi(entry->d_name);
        threads[count].tid = tid;

        int ret = read_thread_registers(pid, tid, threads[count].regs);
        if (ret == 0) {
            count++;  // Successfully read registers
        } else if (ret == -2) {
            // Thread is running in userspace, not in a syscall
            // Still include it, but registers will be partial
            count++;
        }
    }
    // ...
}

The Main Thread First Rule

GDB treats the first NT_PRSTATUS note in the core as the "current thread" — the one shown by default when you open the core. We want this to be the main thread (TID == PID), so after collecting all threads, we swap it to index 0:

for (int i = 1; i < count; i++) {
    if (threads[i].tid == pid) {
        thread_info_t tmp = threads[0];
        threads[0] = threads[i];
        threads[i] = tmp;
        break;
    }
}

This is a small detail, but it matters for UX. When someone opens the core in GDB, they should see the main thread's state first, not a random worker thread.

Running vs. Blocked Threads

Not all threads will be in a syscall. Some might be actively running on a CPU in userspace. For these, /proc/<pid>/task/<tid>/syscall returns the string "running" instead of register values:

if (strncmp(line, "running", 7) == 0)
    return -2;  // distinct from -1 (error)

We still include these threads in the core — their registers will be zeroed, but GDB will at least show them in the thread list. This is honest: we're telling the user "this thread exists but we couldn't capture its state."

Auxiliary Vector: /proc/<pid>/auxv

The simplest data source. The auxiliary vector is a binary blob — we just read and embed it as-is:

int read_auxv(int pid, char **data_out, size_t *sz_out) {
    char path[64];
    snprintf(path, sizeof(path), "/proc/%d/auxv", pid);
    int fd = open(path, O_RDONLY);

    char *data = malloc(4096);
    size_t sz = 0;
    ssize_t n;

    while ((n = read(fd, data + sz, 4096 - sz)) > 0)
        sz += n;

    close(fd);
    *data_out = data;
    *sz_out = sz;
    return 0;
}

No parsing needed. The kernel provides it in exactly the format GDB expects.

Raw Memory: /proc/<pid>/mem

This is how we read the actual memory contents for the PT_LOAD segments. The /proc/<pid>/mem file is a virtual file that mirrors the process's entire address space. You can pread() at any virtual address:

// Read one page of the process's memory at virtual address 0x7ffc89ff0000
unsigned char page[4096];
ssize_t n = pread(mem_fd, page, 4096, 0x7ffc89ff0000);

This is remarkable. We're reading another process's memory without ptrace, without signals, without cooperation. The kernel handles it internally through its page table management.

Page-by-Page Reads

We don't try to read entire regions at once. A 8MB stack region might have guard pages in the middle — deliberately unmapped pages that catch stack overflows. Reading a guard page returns EIO:

for (size_t offset = 0; offset < region_size; offset += PAGE_SIZE) {
    unsigned char page[4096];
    ssize_t n = pread(mem_fd, page, PAGE_SIZE,
                      regions[i].start + offset);

    if (n <= 0) {
        // Guard page, unmapped page, or read error
        memset(page, 0, PAGE_SIZE);  // Zero-fill instead of aborting
        zeroed += PAGE_SIZE;
    }

    fwrite(page, PAGE_SIZE, 1, core_file);
}

Zero-filling is the key technique here. If we abort on the first failed page read, we lose everything. If we skip pages, the file offsets become wrong and the core is corrupted. By writing zeros for unreadable pages, we maintain consistent offsets while gracefully handling any memory layout surprises.

flowchart LR A["Region: 0x7ffc8a000000\n8MB, rw-p, [stack]"] --> B["Page 0: pread OK → write"] B --> C["Page 1: pread OK → write"] C --> D["Page 2: pread EIO\n(guard page)"] D --> E["Write 4096 zeros"] E --> F["Page 3: pread OK → write"] F --> G["...continue..."] style D fill:#fee,stroke:#c33 style E fill:#ffe,stroke:#cc3

Summary: The Data Pipeline

Here's the complete picture of where every piece of the core dump comes from:

Core Dump Component Data Source Key Info Extracted
ELF Header Hardcoded ET_CORE, EM_X86_64
PT_LOAD headers /proc/pid/maps Address ranges, permissions, sizes
NT_PRSTATUS (registers) /proc/pid/task/*/syscall RSP, RIP + syscall args → mapped to register slots
NT_PRSTATUS (RBP) /proc/pid/mem + heuristic Stack scanning for frame pointer chains
NT_PRPSINFO /proc/pid/cmdline Process name, command line
NT_FILE /proc/pid/maps File-backed regions → filenames
NT_AUXV /proc/pid/auxv Raw binary blob, embedded as-is
LOAD segment data /proc/pid/mem Page-by-page reads with zero-fill fallback

In Part 4, we'll put it all together — the actual build_core() function that orchestrates everything into a single, GDB-compatible ELF core file.

← PreviousPart 2: The ELF Core Dump