Apple Mac OS X 10.4 temp_patch_ptrace(): Nonsense in kernel-land

Table of contents
- Published on:
- November 3, 2008
- Last modified:
- August 22, 2022
Categories
Several software vendors realized, sometime during the 1990-2000 time-frame, that exporting system call tables within kernel address space was a bad idea. This obviously doesn’t mean anything to Red Hat and other GNU/Linux vendors who are happily providing world readable System.map
files. Not like anybody needs them, though.
Then again, you have to face potential funniness of contradictory measures, like Apple’s own mistakes. This article won’t talk about yet another bug introduced by a Linux developer working at Red Hat (and later silently fixed by another employee of the very same company), but an interesting issue with Mac OS X 10.4 systems on PowerPC.
The temp_patch_ptrace() function: how to fix an issue and introduce a new one
Albeit the implementation of ptrace
on Mac OS X is severely crippled, they had time to add a nifty trick to prevent immediate debugging of certain processes. Undocumented, it was obviously used only by Apple’s own software, namely iTunes and related applications. A private flag set by a process would disallow future interaction with it via ptrace
or other mechanisms, thus causing the gdb
debugger to fail when trying to attach to the target process. A modern version of the good old trick first described publicly by Silvio Cesare in one of his anti-debugging articles.
Apple, possibly with the intention of helping anti-piracy software vendors (in their quest to preserve all that is good and just in the software industry and beyond) added a KPI (Kernel Programming Interface) that let’s a kernel extension patch the ptrace
system call. The sysent
variable (the BSD equivalent of the Linux syscall_table
, holding pointers, arguments and other data of the supported system calls) is not exported in any Mac OS X system, as a measure to prevent abuse (for example, in rootkits and other malware subverting kernel-land code).
Therefore, there’s no absolutely reliable method to patch the system call table without resorting to hacks (even though these can be extremely reliable, mostly always they are tied to specific versions and or architectures). Hence, the existence of temp_patch_ptrace
. See the implementation of the function below:
481 /*
482 * WARNING - this is a temporary workaround for binary compatibility issues
483 * with anti-piracy software that relies on patching ptrace (3928003).
484 * This KPI will be removed in the system release after Tiger.
485 */
486 uintptr_t temp_patch_ptrace(uintptr_t new_ptrace)
487 {
488 struct sysent * callp;
489 sy_call_t * old_ptrace;
490
491 if (new_ptrace == 0)
492 return(0);
493
494 enter_funnel_section(kernel_flock);
495 callp = &sysent[26];
496 old_ptrace = callp->sy_call;
497
498 /* only allow one patcher of ptrace */
499 if (old_ptrace == (sy_call_t *) ptrace) {
500 callp->sy_call = (sy_call_t *) new_ptrace;
501 }
502 else {
503 old_ptrace = NULL;
504 }
505 exit_funnel_section( );
506
507 return((uintptr_t)old_ptrace);
508 }
It’s not available on Leopard. The implications of this are fairly evident:
- A kernel extension could patch the
ptrace
system call without knowing thesysent
exact location. - A rootkit can be implemented with just a single system call patch. And
ptrace
takes a good amount of arguments, therefore providing a wide range of possibilities (as an exercise, think of a protocol based onptrace
which, upon a magicrequest
argument, performs specific actions using a data buffer pointed by theaddr
argument). - The function returns the original location of
ptrace
on kernel address space. If we wanted to locatesysent
within a specific range of addresses, knowing the location of a system call will let us calculate an offset to the start of the structure (allowing verification for known values too).
So, why would Apple stop exporting the sysent
structure and still provide a function with the purpose of patching a system call?
Why not exporting sysent
if a linear memory search is trivial to use for locating it on memory (which has been used historically by Linux rootkits)?