Log in

No account? Create an account
Pointer Encryption - Ulrich Drepper — LiveJournal [entries|archive|friends|userinfo]
Ulrich Drepper

[ website | My Website ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Pointer Encryption [Jan. 24th, 2007|02:00 pm]
Ulrich Drepper

Mark pointed out that I haven't mentioned anywhere public parts of the security features new in FC6. Well, here it is.

One of the remaining attack vectors in the runtime are function pointers in writable memory. Overwrite the value and you can redirect execution. Of course the pointer must actually be used and randomization must be overcome, but it's theoretically possible.

The remedy I've implemented in libc internally is to encrypt function pointers. I.e., they are not stored as-is but instead in a mangled form. This mangling consists in my code of XOR-ing the pointer value with a random 32/64-bit value. Each process has its own random value. The code was publicly committed back in December 2005 and is in FC6.

The only real challenge was to make this fast. Especially on platforms like x86 which have no fast PC-relative data access. To not use a fixed address the value is stored in the TCB.

What is protected? I hope meanwhile most function pointers in libc. Some are probably still missing and others cannot be handled this way since they are visible to the outside. For some broken programs (including UML) the setjmp change was the biggest. These programs tried to access the stored code address which now is not really useful anymore (program don't know how to decrypt the value). Other pointers which are encrypted are the iconv and atexit structures as well as some function pointer tables people don't really know about, they are completely internal.

Using encryption (instead of canaries) to protect structures like jmp_buf is at least as secure and in addition faster. Question is whether we can extend the use to other parts of the runtime. Runtimes for languages like C++ and Java just scream for such a protection, virtual function tables are a prime target.

(Deleted comment)
From: udrepper
2007-01-29 08:54 am (UTC)

Performance impacts

Not measurable. I doubt that you can write even a micro benchmark which shows any negative effects. Things would be different if we'd encrypt return addresses the processor puts on the stack and pointers the program maintains. Only libc-internal data structures are protected so far.
(Reply) (Parent) (Thread)
[User Picture]From: kragen
2007-01-28 03:00 am (UTC)
I can't think of any circumstances under which this breaks conservative gc, which is my usual concern with xoring pointers.

Good work, I think. I always have mixed feelings about things like this, but you're probably right that the performance tradeoff is worth it.
(Reply) (Thread)