In Part 1, we learned that D-state processes are untouchable by debuggers because ptrace can't deliver signals to them. But we discovered a backdoor: /proc/<pid>/ still works, giving us access to memory maps, registers, and raw memory.

Now we need to take that raw data and package it into something GDB understands. That something is an ELF core dump.

If you've ever seen a core file after a segfault, you've seen one. But what's actually inside it? And which parts does GDB really need to produce a backtrace? Let's find out — byte by byte.

What Is a Core Dump, Really?

A core dump is an ELF file — the exact same binary format used for executables (/usr/bin/ls) and shared libraries (libc.so). The only difference is a single field in the header that says "I'm not a program you can run — I'm a snapshot of a program that was running."

It contains three things:

  1. CPU registers — what each thread was doing at the moment of capture
  2. Memory contents — stack, heap, loaded libraries, everything
  3. Metadata — process ID, command line, file mappings, auxiliary info

Think of it as a crime scene photograph. The process is frozen in time, and GDB is the detective examining the evidence.

The Big Picture

Here's what an ELF core dump looks like at a high level:

Let's walk through each section.

The ELF Header — "Hello, I'm a Core Dump"

Every ELF file starts with a 64-byte header. It's the file's ID card:

Elf64_Ehdr ehdr;

// Magic bytes: 0x7f 'E' 'L' 'F' — identifies this as an ELF file
memcpy(ehdr.e_ident, ELFMAG, SELFMAG);

ehdr.e_ident[EI_CLASS]   = ELFCLASS64;      // 64-bit
ehdr.e_ident[EI_DATA]    = ELFDATA2LSB;      // Little-endian
ehdr.e_ident[EI_OSABI]   = ELFOSABI_LINUX;   // Linux

ehdr.e_type    = ET_CORE;    // ← THIS is what makes it a core dump
ehdr.e_machine = EM_X86_64;  // Architecture
ehdr.e_phoff   = 64;         // Program headers start right after this header
ehdr.e_phnum   = 1 + N;      // 1 PT_NOTE + N PT_LOAD segments

The critical field is e_type = ET_CORE. When GDB opens a file and sees this, it knows: "This isn't a binary to execute — this is a memory snapshot to analyze."

The e_phnum field tells GDB how many program headers follow. Each program header describes one chunk of the file — either metadata (notes) or memory content (loadable segments).

flowchart TB A["ELF Header
(64 bytes)"] --> B["e_type = ET_CORE"] A --> C["e_machine = EM_X86_64"] A --> D["e_phoff = 64
(program headers start here)"] A --> E["e_phnum = 1 + N"] E --> F["1 × PT_NOTE
(all metadata)"] E --> G["N × PT_LOAD
(memory regions)"]

Program Headers — The Table of Contents

Right after the ELF header comes the program header table. Each entry is a 56-byte Elf64_Phdr struct that says: "At file offset X, you'll find Y bytes of data that belonged at virtual address Z."

There are two types we care about:

PT_NOTE — "Here's the metadata"

The first program header points to the note segment — a blob containing all the structured metadata about the process:

Elf64_Phdr note_phdr;
note_phdr.p_type   = PT_NOTE;
note_phdr.p_offset = note_offset;  // where in the file the notes start
note_phdr.p_filesz = all_notes_sz; // total size of all notes combined

PT_LOAD — "Here's a chunk of memory"

Every other program header is a PT_LOAD that maps a memory region:

Elf64_Phdr load_phdr;
load_phdr.p_type   = PT_LOAD;
load_phdr.p_offset = file_offset;  // where in the core file
load_phdr.p_vaddr  = 0x7f4a...;   // original virtual address
load_phdr.p_filesz = region_size;  // how many bytes
load_phdr.p_flags  = PF_R | PF_W; // permissions: readable + writable
load_phdr.p_align  = 4096;        // page-aligned

When GDB wants to examine memory at address 0x7f4a1234, it scans the PT_LOAD headers to find which one contains that address, then seeks to the corresponding file offset and reads the bytes. Simple lookup table.

The NOTE Segment — The Metadata Gold Mine

This is where the interesting stuff lives. The note segment is a sequence of ELF notes, each wrapped in a tiny header:

typedef struct {
    uint32_t n_namesz;  // Length of the name (always 5: "CORE\0")
    uint32_t n_descsz;  // Length of the descriptor (the actual data)
    uint32_t n_type;    // Note type (NT_PRSTATUS, NT_PRPSINFO, etc.)
} Elf64_Nhdr;
// Followed by: padded name + padded descriptor

Every note follows the same pattern: 12-byte header → "CORE\0" name (padded to 8 bytes) → descriptor data (padded to 4-byte boundary).

There are four note types that matter:

NT_PRSTATUS — Thread Status

One per thread. 336 bytes each. This is what GDB needs most.

NT_PRSTATUS contains the CPU register state for a single thread. Without this, GDB literally cannot produce a backtrace.

Here's the layout:

Byte Range Field What It Contains
0–15 Signal info What signal stopped/killed the thread
32–35 pr_pid Thread ID (TID)
48–111 CPU timers utime, stime, cutime, cstime
112–327 GP Registers 27 × 8-byte registers — the crown jewels
328–335 fpvalid Whether floating-point state follows

The register block at offset 112 is the entire x86_64 user_regs_struct — 27 general-purpose registers in the exact order the kernel stores them:

// The 27 x86_64 registers in user_regs_struct order:
// Index  Register   Role
//  0     R15        General purpose
//  1     R14        General purpose
//  2     R13        General purpose
//  3     R12        General purpose
//  4     RBP        Frame pointer ← critical for stack walking
//  ...
// 10     RAX        Return value / syscall number
// 12     RDX        3rd argument
// 13     RSI        2nd argument
// 14     RDI        1st argument
// 15     ORIG_RAX   Original syscall number
// 16     RIP        Instruction pointer ← where the thread is executing
// 17     CS         Code segment selector (0x33 for userspace)
// 18     EFLAGS     Processor flags
// 19     RSP        Stack pointer ← top of the thread's stack
// 20     SS         Stack segment selector (0x2b for userspace)

GDB uses RIP to know which instruction the thread was on, RSP to find the top of the stack, and RBP to walk the frame pointer chain and reconstruct the full call stack.

In our code, filling this struct looks like:

unsigned char *prstatus = calloc(1, 336);  // zero-initialized

// Set the thread ID
*(pid_t *)(prstatus + 32) = thread.tid;

// Copy all 27 registers into the register block at offset 112
memcpy(prstatus + 112, thread.regs, 27 * sizeof(unsigned long));

// Write as an ELF note
write_note(core_file, NT_PRSTATUS, prstatus, 336);

Why one per thread? Because each thread has its own register state and its own call stack. In a multi-threaded program like cvmfs2, the daemon might have 30+ threads, and only some are stuck in D-state. The per-thread NT_PRSTATUS lets GDB show you: "Thread 1 is stuck in SafeRead(), Thread 2 is stuck in Signal::Wait(), Thread 3 is idle in poll()..."

NT_PRPSINFO — Process Identity (One Per Process)

136 bytes. Written once. Tells GDB what program this core came from.

unsigned char *prpsinfo = calloc(1, 136);

prpsinfo[0] = 2;    // pr_state = TASK_UNINTERRUPTIBLE (D-state!)
prpsinfo[1] = 'D';  // pr_sname = 'D'

// Thread/process ID
*(pid_t *)(prpsinfo + 24) = pid;

// Program name (what shows up in GDB's "Core was generated by...")
strncpy((char *)(prpsinfo + 40), "cvmfs2", 16);     // pr_fname

// Full command line
strncpy((char *)(prpsinfo + 56), "/usr/bin/cvmfs2 /cvmfs/atlas.cern.ch ...", 80);

write_note(core_file, NT_PRPSINFO, prpsinfo, 136);

When you open the core in GDB, the first thing it prints is:

Core was generated by `/usr/bin/cvmfs2 /cvmfs/atlas.cern.ch fuse...'.

That comes from pr_psargs at offset 56. And the pr_state = 'D' is our little calling card — it tells anyone examining this core that the process was in uninterruptible sleep when we captured it.

NT_FILE — The Library Map

Variable size. Written once. This is how GDB finds function names.

Here's something that trips people up: the core dump doesn't contain the .text (code) section of shared libraries. It can, but for a standard-mode dump, we skip those read-only regions to save space. So how does GDB know that address 0x7f4a12345678 is inside libc.so at function read()?

NT_FILE.

This note contains a mapping of virtual address ranges to filenames:

Address Range                    File Offset    Filename
0x7f4a12340000–0x7f4a12380000    0x0            /usr/lib64/libc.so.6
0x7f4a12380000–0x7f4a12390000    0x40000        /usr/lib64/libc.so.6
0x55a8b1000000–0x55a8b1020000    0x0            /usr/bin/cvmfs2
...

When GDB sees that address 0x7f4a12345678 falls in the range for /usr/lib64/libc.so.6, it opens that file from disk, reads the symbol table and DWARF debug info, and resolves the address to a function name + line number.

This is why we always include all file-backed regions in NT_FILE, even in standard dump mode where we skip dumping their actual memory contents. The memory can be reloaded from disk; the address mapping cannot.

// NT_FILE format:
//   uint64_t count        — number of file entries
//   uint64_t page_size    — 4096
//   count × { uint64_t start, end, file_offset }
//   count × null-terminated filename strings
flowchart LR A["NT_FILE entry:
0x7f4a1234...–0x7f4a1238...
/usr/lib64/libc.so.6"] --> B["GDB opens
/usr/lib64/libc.so.6
from disk"] B --> C["Reads .symtab
and .eh_frame"] C --> D["Resolves:
0x7f4a12345678
= read+0x42"]

Key insight: NT_FILE is what makes standard dump mode work. We skip dumping 200MB of read-only .text pages from shared libraries, but GDB can still resolve every symbol because NT_FILE tells it where to find the files on disk.

NT_AUXV — The Kernel's Cheat Sheet

Variable size. Written once. Contains the auxiliary vector.

The auxiliary vector (auxv) is a block of key-value pairs that the kernel passes to every process at startup. It contains system-level information like:

Key Value GDB Uses It For
AT_ENTRY Program entry point Finding _start
AT_PHDR Address of program headers in memory Locating the dynamic linker
AT_PAGESZ System page size (4096) Memory alignment
AT_SYSINFO_EHDR vDSO address System call optimization info

We don't need to parse this — we just read the raw bytes from /proc/<pid>/auxv and embed them as-is:

char *auxv_data;
size_t auxv_size;
read_auxv(pid, &auxv_data, &auxv_size);  // reads /proc/pid/auxv

write_note(core_file, NT_AUXV, auxv_data, auxv_size);

GDB uses this primarily to find the dynamic linker (ld-linux.so), which in turn tells it where all the shared libraries are loaded. Combined with NT_FILE, this gives GDB a complete picture of the process's address space layout.

The LOAD Segments — The Actual Memory

After the note segment comes the bulk of the core file: the actual memory contents.

Each PT_LOAD program header we wrote earlier corresponds to one memory region. Now we write the raw bytes for each region, in the same order:

for (int i = 0; i < num_regions; i++) {
    if (!should_dump_region(&regions[i])) continue;

    size_t region_size = regions[i].end - regions[i].start;

    // Read the region page-by-page from /proc/pid/mem
    for (size_t offset = 0; offset < region_size; offset += PAGE_SIZE) {
        unsigned char page[4096];
        ssize_t n = pread(mem_fd, page, PAGE_SIZE,
                          regions[i].start + offset);

        if (n <= 0) {
            // Guard page or unmapped — write zeros instead of aborting
            memset(page, 0, PAGE_SIZE);
            n = PAGE_SIZE;
        }

        fwrite(page, n, 1, core_file);
    }
}

Two important details:

  1. Page-by-page reads: We don't try to read an entire 8MB stack region in one pread() call. Some pages within a region might be guard pages (deliberately unmapped to catch stack overflows). Reading them returns EIO. By reading one page at a time, we isolate failures and zero-fill just the bad pages.

  2. Zero-filling: If a page read fails, we write 4096 zero bytes instead. This keeps the file offsets consistent — every PT_LOAD header says "region X is at file offset Y with size Z," and those offsets must be exact. Skipping bytes would corrupt the entire file.

The Offset Arithmetic — Getting It Right

The trickiest part of writing a core dump isn't any individual piece — it's making all the offsets consistent. Every program header contains file offsets that must point to exactly the right byte in the output file.

Here's the calculation:

File offset 0:          ELF header (64 bytes)
File offset 64:         Program header table
                        = (1 + loadable_regions) × 56 bytes
File offset X:          NOTE segment starts
                        = 64 + phdr_table_size
File offset Y:          First LOAD segment starts
                        = X + total_notes_size
File offset Y+size[0]:  Second LOAD segment starts
...
flowchart TB A["1. Count dumpable regions → N"] --> B["2. Calculate note sizes"] B --> C["3. note_offset = 64 + (1+N) × 56"] C --> D["4. data_offset = note_offset + all_notes_size"] D --> E["5. Write ELF header"] E --> F["6. Write PT_NOTE phdr (offset = note_offset)"] F --> G["7. Write N × PT_LOAD phdrs
(each offset = running total)"] G --> H["8. Write notes at note_offset"] H --> I["9. Write memory at data_offset"] style A fill:#e8f4fd style I fill:#e8f4fd

Everything must be calculated before writing anything, because the ELF header and program headers reference offsets that come later in the file. This is why build_core() does a two-pass approach:

  1. First pass: count regions, calculate sizes, compute all offsets
  2. Second pass: write everything in order

Putting It All Together

Let's zoom out. Here's the complete pipeline from a D-state process to a GDB backtrace:

flowchart LR A["/proc/pid/maps"] -->|parse| B["Memory regions"] C["/proc/pid/task/*/syscall"] -->|parse| D["Thread registers"] E["/proc/pid/auxv"] -->|read| F["Auxiliary vector"] G["/proc/pid/mem"] -->|pread| H["Raw memory"] B --> I["build_core()"] D --> I F --> I H --> I I -->|write| J["output.core"] J -->|load| K["GDB"] K --> L["thread apply all bt"]

The core file is just a container format — a way to organize /proc data into something GDB already knows how to read. We're not inventing anything new; we're just filling in the same structs that the kernel would normally fill if it could dump core the normal way.

In Part 3, we'll dive into the trickiest part: how we actually extract the data from /proc, especially the heuristic RBP recovery that turns a one-frame backtrace into a full call stack.

← PreviousPart 1: How Debuggers Work