We start our survey of historical stack limit checking functions on Windows with the 80386 family of processors. This function has actually changed form over the years, so we’ll start with the “original flavor”.

Originally, the _chkstk function was called by putting the desired number of bytes in the eax register and calling the _chkstk function. The function touched each page of the stack, adjusted the stack pointer, and then returned with the adjusted stack pointer. This is an unusual calling convention since it is neither caller clean, nor is it callee clean. It’s callee-dirty! The function returns with more stack than it started.

_chkstk: push ecx ; remember desired allocation size

; calculate the stack pointer of the caller
mov     ecx, esp
add     ecx, 8          ; 4 bytes were auto-pushed for the return address,
                        ; we pushed 4 bytes for the ecx

touch: cmp eax, PAGE_SIZE ; less than a page to go? jb finalpage ; do the last page and finish sub ecx, PAGE_SIZE ; allocate a page from our pretend stack pointer or dword ptr [ecx], 0 ; touch the memory sub eax, PAGE_SIZE ; did a page jmp touch ; go back and do some more

finalpage: sub ecx, eax ; allocate the leftovers from our pretend stack pointer or dword ptr [ecx], 0 ; touch the memory mov eax, esp ; remember original stack pointer mov esp, ecx ; move the real stack to match our pretend stack mov ecx, [eax] ; recover original ecx mov eax, 4[eax] ; recover return address jmp eax ; “return” to caller

A function with a large stack frame would go something like

function: push ebp ; link into frame chain mov ebp, esp push ebx ; save non-volatile register push esi push edi mov ecx, 17320 ; large stack frame call _chkstk ; allocate it from our stack safely ; behaves like “sub esp, ecx”

This goes into the competition for “wackiest x86-32 calling convention.”¹

Next time, we’ll look at how stack probing happens on MIPS, which has its own quirks, but nothing as crazy as this.

Bonus chatter: The strange calling convention dates back to the 16-bit 8086. And back then, there were two versions of the chkstk function, depending on whether you were calling it far or near.

; frame size in ax

chkstk: #if NEAR pop bx ; pop 16-bit return address #else // FAR pop bx ; pop 32-bit return address pop dx #endif

inc     ax
and     al, 0xFE    ; round up to even

sub     ax, sp      ; check for stack overflow
jae     overflow    ; Y: overflow
neg     ax          ; ax = new stack pointer

cmp     ax, ss:[pStackTop]
ja      overflow    ; stack mysteriously too high

cmp     ax, ss:[pStackMin] ; new stack limit?
jbe     nochange
mov     ss:[pStackMin], ax ; update stack limit

nochange:

mov     sp, ax      ; update the stack pointer

#if NEAR jmp bx ; “return” to caller #else // FAR push dx ; restore return address push bx retf ; return to caller #endif

The post Windows stack limit checking retrospective: x86-32, also known as i386 appeared first on The Old New Thing.

From The Old New Thing via this RSS feed