Apple Mac OS X 10.4 temp_patch_ptrace(): Nonsense in kernel-land

Several software vendors realized, sometime during the 1990-2000 time-frame, that exporting system call tables within kernel address space was a bad idea. This obviously doesn’t mean anything to Red Hat and other GNU/Linux vendors who are happily providing world readable System.map files. Not like anybody needs them, though.

Then again, you have to face potential funniness of contradictory measures, like Apple’s own mistakes. This article won’t talk about yet another bug introduced by a Linux developer working at Red Hat (and later silently fixed by another employee of the very same company), but an interesting issue with Mac OS X 10.4 systems on PowerPC.

The temp_patch_ptrace() function: how to fix an issue and introduce a new one

Albeit the implementation of ptrace on Mac OS X is severely crippled, they had time to add a nifty trick to prevent immediate debugging of certain processes. Undocumented, it was obviously used only by Apple’s own software, namely iTunes and related applications. A private flag set by a process would disallow future interaction with it via ptrace or other mechanisms, thus causing the gdb debugger to fail when trying to attach to the target process. A modern version of the good old trick first described publicly by Silvio Cesare in one of his anti-debugging articles.

Apple, possibly with the intention of helping anti-piracy software vendors (in their quest to preserve all that is good and just in the software industry and beyond) added a KPI (Kernel Programming Interface) that let’s a kernel extension patch the ptrace system call. The sysent variable (the BSD equivalent of the Linux syscall_table, holding pointers, arguments and other data of the supported system calls) is not exported in any Mac OS X system, as a measure to prevent abuse (for example, in rootkits and other malware subverting kernel-land code).

Therefore, there’s no absolutely reliable method to patch the system call table without resorting to hacks (even though these can be extremely reliable, mostly always they are tied to specific versions and or architectures). Hence, the existence of temp_patch_ptrace. See the implementation of the function below:

481 /* 
482  * WARNING - this is a temporary workaround for binary compatibility issues
483  * with anti-piracy software that relies on patching ptrace (3928003).
484  * This KPI will be removed in the system release after Tiger.
485  */
486 uintptr_t temp_patch_ptrace(uintptr_t new_ptrace)
487 {
488         struct sysent *         callp;
489         sy_call_t *                     old_ptrace;
490 
491         if (new_ptrace == 0)
492                 return(0);
493                 
494         enter_funnel_section(kernel_flock);
495         callp = &sysent[26];
496         old_ptrace = callp->sy_call;
497         
498         /* only allow one patcher of ptrace */
499         if (old_ptrace == (sy_call_t *) ptrace) {
500                 callp->sy_call = (sy_call_t *) new_ptrace;
501         }
502         else {
503                 old_ptrace = NULL;
504         }
505         exit_funnel_section( );
506         
507         return((uintptr_t)old_ptrace);
508 }

It’s not available on Leopard. The implications of this are fairly evident:

• A kernel extension could patch the ptrace system call without knowing the sysent exact location.
• A rootkit can be implemented with just a single system call patch. And ptrace takes a good amount of arguments, therefore providing a wide range of possibilities (as an exercise, think of a protocol based on ptrace which, upon a magic request argument, performs specific actions using a data buffer pointed by the addr argument).
• The function returns the original location of ptrace on kernel address space. If we wanted to locate sysent within a specific range of addresses, knowing the location of a system call will let us calculate an offset to the start of the structure (allowing verification for known values too).

So, why would Apple stop exporting the sysent structure and still provide a function with the purpose of patching a system call? Why not exporting sysent if a linear memory search is trivial to use for locating it on memory (which has been used historically by Linux rootkits)?