16

A key tool in reverse engineering is a good disassembler, so to ensure that a disassembler is performing properly, are there any good test suites available for use to test the correctness of a disassembler? Are these architecture specific, or can they be configured to work across multiple object architectures? A good test should include checking the more obscure architecture instructions and malformed portable execution files.

Here is one specifically for i86 that I have seen. Are there any that are modular across architectures?

JMcAfreak
  • 862
  • 7
  • 25
WilliamKF
  • 927
  • 2
  • 11
  • 12
  • 1
    When you say "correctness of a disassembler", do you mean correctness of an instruction decoder, or, do you mean correct control flow graph recovery and other kinds of high-level semantic information? – Andrew Mar 20 '13 at 16:17
  • Well, which is the more interesting question? I'm guessing the later, correct flow graph recovery... – WilliamKF Mar 20 '13 at 16:25

5 Answers5

14

There is a paper called "N-version Disassembly: Differential Testing of x86 Disassemblers" (PDF) by Roberto Paleari, Lorenzo Martignoni, Giampaolo Fresi Roglia, and Danilo Bruschi which compares several x86 disassemblers in a formal fashion.

To quote the papers abstract:

The output of a disassembler is used for many different purposes (e.g., debugging and reverse engineering). Therefore, disassemblers represent the first link of a long chain of stages on which any high-level analysis of machine code depends upon. In this paper we demonstrate that many disassemblers fail to decode certain instructions and thus that the first link of the chain is very weak. We present a methodology, called N-version disassembly, to verify the correctness of disassemblers, based on differential analysis

Not sure if this is slightly off topic to your question but may be of interest to you.

Ruslan
  • 293
  • 3
  • 10
QAZ
  • 2,571
  • 22
  • 22
  • Interesting paper! Not a wholly surprising conclusion. Godefroid and Taly have an MSR TR (http://research.microsoft.com/apps/pubs/default.aspx?id=156020) where they automate instruction synthesis and discovered errors in their manually specified semantics. – Andrew Mar 20 '13 at 16:39
8

In a lot of papers I've read, decompilation tool authors use the SPEC benchmarks to measure the effectiveness of their decompiler. This produces kind of a holistic view of how well the system works, from instruction decoding to control flow recovery. Those benchmarks aren't free or open source though.

The GCC and clang compilers also ship with benchmarks for testing. Those might be worth investigating.

Also, Regehr's csmith project can generate arbitrary C programs for compiler testing/fuzzing. This could be useful for testing decompilers and binary analysis systems?

Andrew
  • 1,522
  • 12
  • 15
6

The gas testcase suite that you link to is not only for i386. The parent directory contains test cases for x86-64, arm, alpha, and many other architectures.

Janus Troelsen
  • 105
  • 3
Ed McMan
  • 944
  • 8
  • 9
5

I made a file containing all x86 opcodes (at the time) and many encoding forms, that helped me to actively find many bugs in many disassemblers (IDA, Hiew, BeaEngine, OllyDbg, XED, hte...), particularly lesser known forms and undocumented ones.

(slightly off-topic maybe - not a ready-made test suite)

Ange
  • 6,694
  • 3
  • 28
  • 62
2

The radare2 project uses an extensive test-suite for each of its disassembler engine, along with more specific tests, like formats, its own analysis capabilities, …

jvoisin
  • 2,516
  • 16
  • 23