HEAPS OF TROUBLE (March 2011)
The Windows heap memory is a rich source of anti-debugging techniques. It can be altered in numerous ways to achieve interesting effects, such as the execution of arbitrary code in particular circumstances . It can also be used in indirect ways, since many APIs allocate and/or free memory as part of their standard behaviour.
A corrupted heap can result in a call to the DbgPrint() API, if the appropriate flag is set in the heap flags. This happens by default when a debugger is present. When the DbgPrint() API is called, it raises a special exception, which in turn changes the returned error code. This change can be detected by an application, and it might behave differently as a result.
The most obvious API that produces this effect is the HeapFree() function. Two other common APIs are the DeleteFiber() and the FindVolumeClose() functions. There is a special case in the ConvertFiberToThread() function that can also result in this behaviour. More recent versions of Windows have introduced some new APIs which also demonstrate this behaviour. In Windows Vista and later, there is the BasepFreeAppCompatData() function. In Windows 7, there are the BasepFreeActivationContextActivationBlock() and the SortCloseHandle() functions. The FindVolumeMountPointClose() function exists in all platforms, but the heap-related behaviour exists only on Windows 7. However, that API also calls CloseHandle(), and the CloseHandle() attack(*) is possible in all versions of Windows.
This kind of behaviour can also be used to call APIs indirectly, which can make static analysis a bit more "interesting". Need to free memory? FindVolumeClose() can do that. Open a process and read its memory? Toolhelp32ReadProcessMemory() can do that. Call a user-defined function in a sparse structure? RtlProcessFlsData() can do that. Who would have guessed? But now we know those tricks, we won't be fooled.
* by passing an invalid handle to the API, an invalid handle exception is raised, which can reveal the presence of a debugger