IRIX Binary Compatibility, Part 2
Pages: 1, 2

The ELF Auxiliary Table

The ELF auxiliary table is used by dynamic linkers to gather information about the program they are about to link and launch. It is a table of pairs (type, value) stored on the stack. These pairs are called auxiliary vectors. Documentation of the available vector types can be found in NetBSD's /usr/include/elf.h:

#define AT_NULL         0       /* Marks end of array */
#define AT_IGNORE       1       /* No meaning, a_un is undefined */
#define AT_EXECFD       2       /* Open file descriptor of object file */
#define AT_PHDR         3       /* &phdr[0] */
#define AT_PHENT        4       /* sizeof(phdr[0]) */
#define AT_PHNUM        5       /* # phdr entries */
#define AT_PAGESZ       6       /* PAGESIZE */
#define AT_BASE         7       /* Interpreter base addr */

The ELF interpreter will use these to discover the address of the ELF program header, which lists the executable ELF sections in the executable, for instance. This is used to discover the list of required shared libraries and the symbol table location.

We use the same stack dumping technique as described in the previous section to discover what information the IRIX kernel lists in the ELF auxiliary table. Things are just a bit different: when running a dynamic executable, the kernel launches the interpreter first, not an ELF section from the program. Therefore, we cannot just set a breakpoint at a collected address using objdump(1) on our program. Instead we need to set the breakpoint at the interpreter's entry point. On IRIX, the interpreter is libc itself, hence we can discover the entry point of a dynamic binary by using objdump(1) on libc:

$ objdump -f /lib/libc.so.1

/lib/libc.so.1:     file format elf32-bigmips
architecture: mips:6000, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x0fae0774

We just have to set up the breakpoint at 0x0fae0778, and we can see the stack as it is set up by the IRIX kernel. The ELF auxiliary table appears after the envp array. IRIX sets up the following vector types: AT_PHDR, AT_PHENT, AT_PHNUM, AT_ENTRY, AT_BASE, and AT_PAGESZ. Once we know what should be in it, then it is quite easy to add the code to copy this table to our irix_copyargs() function. This enables dynamic binaries to start, but they quickly die, complaining that a mysterious system call named syssgi() was not implemented. We will have a closer look to syssgi() in a future article.

Setting Up the CPU Registers on Startup

The CPU registers are set on startup by the function pointed by the e_setregs field of the struct emul. IRIX emulation uses the NetBSD native function, which is simply called setregs(), and this works for o32.

For n32 binaries, however, using setregs() led to an unpleasant crash before the first system call. The crash was caused by a SIGILL signal. This signal is sent by a trap raised by an illegal instruction.

As before, gdb is a good tool to help us understand what went wrong here. There are several reason why programs could issue an illegal instruction: Did we start the program at its entry point? Or did we corrupt the stack and return in a random place after a function call? Is it another problem?

Using gdb on a static n32 binary :
$ objdump -f sh
(snip)
start address 0x0e00ba44

$ gdb ./sh
(gdb) b *0x0e00ba48
Breakpoint 1 at 0xe00ba48
(gdb) run
Starting program: ./sh

Breakpoint 1, 0xe00ba48 in ?? ()
(gdb) info registers
          zero       at       v0       v1       a0       a1       a2       a3
 R0   00000000 00000000 00000000 00000000 7fffe9d8 00000000 00000000 0e090000
            t0       t1       t2       t3       t4       t5       t6       t7
 R8   00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
            s0       s1       s2       s3       s4       s5       s6       s7
 R16  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
            t8       t9       k0       k1       gp       sp       s8       ra
 R24  00000000 00000000 00000000 00000000 00000000 7fffe9d8 00000000 00000000
            sr       lo       hi      bad    cause       pc
      0000ff13 00000000 00000000 0e00ba44 00000024 0e00ba48
           fsr      fir       fp
      00000000 00000000 00000000
(gdb) x/26i 0xe00ba44
0xe00ba44:      lui     $a3,0xe09
0xe00ba48:      lw      $a0,0($sp)
0xe00ba4c:      addiu   $a3,$a3,200
0xe00ba50:      lw      $a3,0($a3)
0xe00ba54:      addiu   $a1,$sp,4
0xe00ba58:      li      $at,-16
0xe00ba5c:      lui     $gp,0xe09
0xe00ba60:      and     $sp,$sp,$at
0xe00ba64:      addiu   $a2,$a1,4
0xe00ba68:      sll     $v0,$a0,0x2
0xe00ba6c:      addiu   $gp,$gp,27744
0xe00ba70:      addiu   $sp,$sp,-16
0xe00ba74:      bnez    $a3,0xe00ba88
0xe00ba78:      addu    $a2,$a2,$v0
0xe00ba7c:      lui     $at,0xe09
0xe00ba80:      addiu   $at,$at,200
0xe00ba84:      sw      $a2,0($at)
0xe00ba88:      lui     $at,0xe09
0xe00ba8c:      addiu   $at,$at,15424
0xe00ba90:      sw      $a0,0($at)
0xe00ba94:      lui     $at,0xe09
0xe00ba98:      addiu   $at,$at,15456
0xe00ba9c:      sw      $a1,0($at)
0xe00baa0:      sd      $zero,8($sp)
0xe00baa4:      jal     0xe0715dc

(gdb) c
Continuing.

Program received signal SIGILL, Illegal instruction.
warning: Hit heuristic-fence-post without finding
warning: enclosing function for address 0xe00baa0
0xe00baa0 in ?? ()

The problem was caused by the sd instruction, which stands for "store double word". The credits for debugging this go to Wayne Knowles: the sd instruction is only allowed when the processor is running in 64-bit mode. Execution of sd in 32-bit mode causes a reserved instruction exception. The kernel turns this exception into a SIGILL signal.

The solution is to set up the processor in 64-bit mode for execution of n32 binaries. This is done by setting a flag in the SR register. This flag is called MIPS3_SR_UX in NetBSD's sys/arch/mips/include/psl.h. The fix to this problem is therefore to write a setregs_n32() function to set up the registers for IRIX n32 binaries. This function just sets the MIPS3_SR_UX flag and then calls the regular setregs().

In the exec switch from sys/exec_conf.c, IRIX has two entries: one for o32 binaries, which uses an o32 probe function called irix_elf32_probe_o32(); and the other the emul_irix_o32 struct emul (defined in sys/compat/irix/irix_exec.c). This struct emul contains a pointer to setregs(). The other entry is for n32 binaries; it uses irix_elf32_probe_n32() and the emul_irix_n32 struct emul. emul_irix_n32 contains a pointer to setregs_n32() as the function to set up CPU registers.

With this setregs_n32() function, n32 binaries are able to start up and do a few system calls. They crash on the first system call manipulating 64-bit data, which is the case for mmap(), lseek(), or stat(). Simple static n32 are hence actually able to work because they do not need mmap() to link. It is possible to run a static n32 /bin/sh and use it to launch shell commands, but it quickly dies (as soon as it hits a system call that uses 64-bit data, in fact).

To reliably run n32 binaries, we need 64-bit support in the kernel. This will be discussed in more details in a later article.

Emmanuel Dreyfus is a system and network administrator in Paris, France, and is currently a developer for NetBSD.

---

Return to the BSD DevCenter.