Cleaning up a codebase means fixing compiler warnings first.

One day, I was working through a mountain of warnings when I hit this:

    /usr/bin/ld: warning: /tmp/ccGIIjoX.o: requires executable stack
    (because the .note.GNU-stack section is executable)

It sounds harmless, but it quietly disables one line of defense against buffer overflow exploits.

Why would the compiler want an executable stack?

After some digging, I found the culprit: a clever developer had used a C feature I didn’t even know existed, a non-standard extension called nested functions.

You might know similar constructs from fancier languages, where they’re usually called “lambdas”.

Lambdas can improve code structure and make it easier to read. But in C, they can have a dangerous side effect: the compiler might be forced to make the stack executable.

And I had no idea that this was even possible in C.

The Code That Triggers It

#include <stdio.h>

int main(void) {
    int base = 10;

    int lambda(int x) {
        return base + x;
    }

    int (*fn)(int) = lambda;
    int result = fn(5);
    printf("result = %d\n", result);
    return 0;
}

This looks innocent enough. It takes the address of a nested function, which you might want to do to pass it as a function pointer for a callback argument. For this, the compiler creates a trampoline on the stack - a tiny chunk of executable code that helps the function pointer work correctly.

To run this trampoline, the stack must be executable.

Calling the nested function directly is fine; taking its address is not. It’s a very subtle difference with huge implications.

This is why C has a reputation for being hostile to secure-by-default programming. The language makes it easy to write code that looks harmless and compiles cleanly, yet quietly changes fundamental security assumptions like whether the stack is executable.

Why Is This Dangerous?

An executable stack is a gift to attackers.

It allows injected code (like shellcode from a buffer overflow) to run directly from the stack. Modern systems try to keep the stack non-executable to block exactly these kinds of attacks.

If any object file or library in your build requires an executable stack, the linker marks the whole binary as such, and the kernel will honor it. And here’s where it gets worse:

When the kernel sees the executable-stack flag, it sets a compatibility mode called READ_IMPLIES_EXEC. On some architectures and older kernels, this may weaken NX protections and expand the amount of memory that becomes executable. Data segments that were only supposed to be read-write are now executable.

Suddenly, you have executable memory at predictable addresses. For an attacker, this is like finding the front door not just unlocked, but with a welcome mat and the lights on.

The Tradeoff

So, do you compromise code structure for security?

In this case, yes. Readability matters - good code is maintainable code.

Security matters more.

If you need nested functions in C, there are alternatives. You can refactor the code to use regular functions with context structs. You can use static functions in the same compilation unit. Or you can at least isolate the nested-function code to a separate library so it doesn’t poison your entire binary.

The warning told me exactly what I needed to know: there’s a security cost to this clever trick.

Security is a feature, too.