In the world of software, the most dangerous magic trick is making a program lie to itself.
My introduction to the black art of hacking didn’t happen in a basement or on a dark-web forum; it happened in a university computer security course. We were given a simple task: take a small, boring program and force it to do something ‘impossible.’ That was the day I learned that a computer doesn’t actually ‘know’ what it’s doing - it just follows a trail of breadcrumbs. And if you can move the breadcrumbs, you own the machine.
We were given the source code of a small program and asked to exploit a weakness in it. The binary was installed with setuid permissions for a specific user and performed some trivial calculation on a string argument (counting blanks, something like that).
The vulnerability was intentional: a classic stack-based buffer overflow.
The assignment was simple to describe and brutally hard to execute: provide an input that makes the program do something it was never intended to do, and it would print a codeword. Submit the codeword, pass the assignment.
At the time (early 2000s), this wasn’t nearly as “productized” as it is today. There were no turnkey exploit kits, no Kali Linux, no Metasploit modules to paste into a terminal. There were no copy/paste exploit frameworks available today. Yes, you could find shellcode on the internet if you looked hard enough. But the point of the exercise was to understand the mechanics. We had to write a tiny assembly routine ourselves and document every single line.
It was hard. Very hard. Possibly the most difficult coding assignment I got as a student.
It was also spectacularly exciting.
And I still remember the moment it finally worked: the program did the impossible thing, printed the codeword, and I felt like I had just learned how magic tricks are done.
The Anatomy of a Buffer Overflow
If you want to understand buffer overflow vulnerabilities at a high level, you need a mental model of how a computer executes a function call.
This is where it gets a bit technical, please bear with me.
Three key facts:
- Computers use a special memory for function calls and local variables, called the stack.
- On the stack, there is a return address (the saved instruction pointer) so the computer knows where to continue after a function finishes.
When a function is called, the computer needs a way to go back to the caller afterwards. It needs to leave a breadcrumb that it can follow back. To make that happen, the return address (the crumb) gets pushed onto the stack as part of the function prologue.
The problem begins when a function also places a fixed-size buffer on the stack, for example for a local variable that the function will write to.
Suppose this is our function:
int my_function(const char* input) {
char buffer[200];
strcpy(buffer, input);
// ...
}
The odd thing about the stack is that it grows downward, toward lower memory addresses. After calling this function, the stack looks roughly like this (simplified):
[ Local Buffer ] <-- Local variable buffer is here
[ Return Address ] <-- The "Breadcrumb"
[ Function Arguments ] <-- Pointer to the input string is here
If the code copies more bytes into that buffer than it can hold, the writes continue into adjacent stack memory.
The most adjacent and most interesting targets on the stack is the return address.
That is the core of the vulnerability: the program accidentally allows an attacker to overwrite the saved instruction pointer, achieving EIP/RIP hijacking.
You don’t need to know assembly to understand the impact: once you control where execution continues after the function returns, you achieve arbitrary code execution. Anything that process can do is now on the table: reading secrets, writing files, privilege escalation (if the program has elevated permissions), remote code execution, pivoting to other systems, spawning a shell, or just crashing the process (DoS).
Once you see and understand this, the danger becomes obvious.
Modern Defenses
What makes buffer overflows so dangerous is that they look harmless, and often aren’t recognized as bugs during code reviews. They can be introduced by a single call to an unbounded string function (strcpy, strcat, gets, sprintf) or any loop that writes to a fixed-size buffer without proper bounds checking.
And because this class of bug is so common, operating systems and compilers grew a layer cake of defenses over the last two decades.
It is no longer as easy as it was in my student lab.
Address Space Layout Randomization (ASLR)
Modern kernels randomize the base addresses of key memory regions at process startup: the stack, heap, libraries, and sometimes even the executable itself (PIE - Position Independent Executable). This makes it much harder to reliably jump to a known location in memory, because that location changes every time the program is executed. Attackers can no longer hardcode addresses.
Stack Canaries (SSP - Stack Smashing Protection)
Compilers insert a stack canary (also called a guard value) between local buffers and the stored return address:
[ Local Buffer ] <-- Local variable buffer is here
[ Return Address ] <-- The "Breadcrumb"
[ Stack Canary ] <-- Randomized value
[ Function Arguments ] <-- Pointer to the input string is here
This randomized value is checked before the function returns. If a buffer overflow overwrites it, the corruption is detected and the program aborts (typically via __stack_chk_fail) instead of returning through corrupted state.
NX Bit (No-eXecute / DEP)
Modern CPUs and operating systems mark the stack (and heap) as non-executable using hardware support (NX bit on x86-64, XN on ARM). This is enforced by the MMU (Memory Management Unit). Even if an attacker gains control of the instruction pointer, simply placing shellcode on the stack and jumping to it will not work. W^X (Write XOR Execute) policies enforce that memory is either writable or executable, never both.
Stack Layout Randomization and Compiler Hardening
Compilers add additional defenses: reordering stack variables to place buffers after other locals (making it harder to overwrite critical data), adding padding, and using safer calling conventions. Combined with -fstack-protector flags and instrumentation like ASAN (AddressSanitizer) in development builds, the goal is to make memory corruption less reliable and easier to detect.
And Yet…
These protections didn’t end the game, they just changed the rules. Attackers moved from “Shellcode on the Stack” to Return-Oriented Programming (ROP) - a technique that stitches together snippets of existing, legitimate code (gadgets) to bypass NX bits. It’s the difference between bringing your own tools and building a weapon out of the furniture already in the room.
Still Too Common
The most devastating security problems are not caused by exotic crypto flaws. They stem from ordinary engineering realities, code that looks harmless but contains weaknesses.
Buffer overflows (and memory safety issues in general) are a category of bug that:
- aren’t obvious in code review,
- can survive for years in stable code,
- and have outsized impact when they occur.
What To Do in Modern Engineering Teams
The most effective way to avoid these issues is to create a culture that avoids or detects these kind of bugs, and makes them less likely to escape.
- Clean code: no compiler warnings. Modern compilers often warn about these issues. If your code usually compiles clean (without warnings), those will stick out even before it is committed to a repository.
- DevSecOps integratoin. Your CI build pipeline should run static code analysis (SAST) on every commit. And don’t ignore the output.
- Intentional language/tool choices. Consider using different languages or tools for new components, not everything needs to be written in C. Often this is more safe and cheaper.
My student assignment was exciting because it made the fundamentals tangible.
In industry, the lesson is less fun but more important: memory corruption bugs don’t fail gracefully.
Don’t let a bug control what your code does next.
Source Code Example
For those interested in source code and the technical mechanics, I’ve included a simplified demonstration program that shows how a basic buffer overflow can hijack control flow. It’s deliberately simplified to illustrate the concept without modern protections - compile it yourself and see what happens.