How to use sysenter under Linux?

Question

I would like to know what are the different ways to perform a system call in x86 assembler under Linux. But, with no cheating, only assembler must be used (i.e. compilation with gcc must be done with -nostdlib).

I know four ways to perform a system calls, namely:

• int $0x80
• sysenter (i586)
• call *%gs:0x10 (vdso trampoline)
• syscall (amd64)

I am pretty good at using int $0x80, for example, here is a sample code of a classic 'Hello World!' in assembler using int $0x80 (compile it with gcc -nostdlib -o hello-int80 hello-int80.s):

.data
msg:
  .ascii "Hello World!\n"
  len = . - msg

.text
.globl _start

_start:
# Write the string to stdout
  movl  $len, %edx
  movl  $msg, %ecx
  movl  $1, %ebx
  movl  $4, %eax
  int   $0x80

# and exit
  movl  $0, %ebx
  movl  $1, %eax
  int   $0x80

But the sysenter is often ending with a segmentation fault error. Why ? And, how to use it right ?

Here is an example with call *%gs:0x10 (compiled with gcc -o hello-gs10 hello-gs10.s). Note that I need to go through the libc initialization before calling it properly (that is why I am using main and not anymore _start and, that is also why I removed the option -nostdlib from the compile line):

.data
msg:
  .ascii "Hello World!\n"
  len = . - msg

.text
.globl main

main:
# Write the string to stdout
  movl  $len, %edx
  movl  $msg, %ecx
  movl  $1, %ebx
  movl  $4, %eax
  call  *%gs:0x10

# and exit
  movl  $0, %ebx
  movl  $1, %eax
  call  *%gs:0x10

Also, the syscall is working pretty well also if you know the syscall codes for this architecture (thanks to lfxgroove) (compiled with: gcc -m64 -nostdlib -o hello-syscall hello-syscall.s):

.data
msg:
  .ascii "Hello World!\n"
  len = . - msg

.text
.globl _start

_start:
# Write the string to stdout
  movq  $len, %rdx
  movq  $msg, %rsi
  movq  $1, %rdi
  movq  $1, %rax
  syscall
# and exit
  movq  $0, %rdi
  movq  $60, %rax
  syscall

So, the only problem I have to trigger a system call is this sysenter way. Here is an example with sysenter ending with a segmentation fault (compiled with gcc -m32 -nostdlib -o hello-sysenter hello-sysenter.s):

.data
msg:
  .ascii "Hello World!\n"
  len = . - msg

.text
.globl _start

_start:
# Write the string to stdout
  movl  $len, %edx
  movl  $msg, %ecx
  movl  $1, %ebx
  movl  $4, %eax

  push    final
  sub $12, %esp
  mov %esp, %ebp

  sysenter
# and exit
final:  
  movl  $0, %ebx
  movl  $1, %eax

  sub $12, %esp
  mov %esp, %ebp

  sysenter

Answer 1

System calls through sysenter

sysenter is a i586 instruction, specifically tight to 32-bits applications. It has been subsumed by syscall on 64-bits plateforms.

One particularity of sysenter is that it does require, in addition to the usual register setting, a few manipulations on the stack before calling it. This is because before leaving sysenter, the process will go through the last part of the __kernel_vsyscall assembler snippet (starting from 0xf7ffd430):

Dump of assembler code for function __kernel_vsyscall:
   0xf7ffd420 <+0>:        push   %ecx
   0xf7ffd421 <+1>:        push   %edx
   0xf7ffd422 <+2>:        push   %ebp
   0xf7ffd423 <+3>:        mov    %esp,%ebp
   0xf7ffd425 <+5>:        sysenter 
   0xf7ffd427 <+7>:        nop
   0xf7ffd428 <+8>:        nop
   0xf7ffd429 <+9>:        nop
   0xf7ffd42a <+10>:       nop
   0xf7ffd42b <+11>:       nop
   0xf7ffd42c <+12>:       nop
   0xf7ffd42d <+13>:       nop
   0xf7ffd42e <+14>:       int    $0x80
=> 0xf7ffd430 <+16>:       pop    %ebp
   0xf7ffd431 <+17>:       pop    %edx
   0xf7ffd432 <+18>:       pop    %ecx
   0xf7ffd433 <+19>:       ret    
End of assembler dump.

So, the sysenter instruction expect to have the stack forged in that way:

0x______0c  saved_eip   (ret)
0x______08  saved_%ecx  (pop %ecx)
0x______04  saved_%edx  (pop %edx)
0x______00  saved_%ebp  (pop %ebp)

That's why, each time we need to call sysenter, we first have to push the values of the saved %eip, and the same with%ecx, %edx and %ebp. Which leads to:

.data
msg:
    .ascii "Hello World!\n"
    len = . - msg

.text
.globl _start
_start:
    pushl  %ebp
    movl   %esp, %ebp
# Write the string to stdout
    movl   $len, %edx
    movl   $msg, %ecx
    movl   $1, %ebx
    movl   $4, %eax
# Setting the stack for the systenter
    pushl  $sysenter_ret
    pushl  %ecx
    pushl  %edx
    pushl  %ebp
    movl   %esp,%ebp
    sysenter
# and exit
sysenter_ret:    
    movl   $0, %ebx
    movl   $1, %eax
# Setting the stack for the systenter
    pushl  $sysenter_ret # Who cares, this is an exit !
    pushl  %ecx
    pushl  %edx
    pushl  %ebp
    movl   %esp,%ebp
    sysenter