Failure Observation Engine (FOE) 2.1 README



Change Log

See NEWS.txt

Quick Start

Because fuzzing can fill temporary directories, put the target application  in an unusable state, or trigger other operating-system-level bugs, we recommend that FOE be used in a virtual machine.

Run FOE-2.1-setup.exe in a virtual machine to install FOE 2.1.

The installer should detect and attempt to download prerequisites and configure your environment appropriately.

Running FOE

  1. Click the FOE2 item in the Windows Start menu.
  2. Run
  3. Run tools\ to check fuzzing progress when you wish.

How it works

When a campaign starts, FOE will gather available seed files and create scorable sets:

  1. The seed files themselves
  2. The fuzz percent ranges for each seed file

Each interval of a campaign will choose a seed file, and then for that file, it will choose an percent range to mangle the file. After mangling the file, FOE will launch the target application, using the configured command line to have it parse the fuzzed file. If the "winrun" runner is compatible with the current platform, this is accomplished by preloading a crash-intercepting hook into the target application's process space. This allows crash detection without relying on a debugger. The "nullrun" runner simply runs each invocation  through the debugger (cdb).

When a crash is detected, it is then verified using a combination of cdb and the Microsoft !exploitable debugger extension. If the crash is determined to be unique (by the chain of !exploitable crash hashes), then some additional analysis steps are taken:

  1. A !exploitable report is created for each continuable exception.
  2. If configured to do so, FOE will create a minimized test case.
  3. The seed file and percent range that were used to fuzz are scored

Seed files that produce more crashes are given a preference over less-productive files, and for each seed file, the mangling percent ranges that are more productive are also given preference. These scoring features together minimize the amount of knowledge required to perform an effective fuzzing campaign.

Analyzing results

  +- <configname>.yaml
  +- version.txt
  +- <SEVERITY>/
     +- <hash_1>/
        +- minimizer_log.txt
        +- sf_<seedfile_hash>.<ext>
        +- sf_<seedfile_hash>-<iteration>-<EFA>.<ext>
        +- sf_<seedfile_hash>-<iteration>-<EFA>-<SEVERITY>.<ext>.msec
        +- sf_<seedfile_hash>-<iteration>-<EFA>-minimized.<ext>
        +- sf_<seedfile_hash>-<iteration>.<ext>.e<n>.msec
     +- <hash_2>/
     +- ...
     +- <hash_n>/

This is a copy of the config file used for this run. It is stored for historical purposes ("Which options did I use for that run?").

This file stores the version of FOE that was used for fuzzing.

This is the "Exploitability Classification" assigned to the crash by !exploitable. Values can be EXPLOITABLE, PROBABLY_EXPLOITABLE, UNKNOWN, or PROBABLY_NOT_EXPLOITABLE. For crashes that include multiple exceptions, the highest exploitability of any of the exceptions is used for this directory. Be aware that !exploitable has limitations and only provides a rough (possibly false-positive) assesment of a crash.
More information on !exploitable can be found here:

This is the hash in Major.Minor form provided by !exploitable.

This is the log file that was produced during crash minimization.

This is the original file (pre-fuzz). This is provided as a convenient "diff" source.

This is the fuzzed file that caused the crash. <EFA> is the exception faulting address, as reported by !exploitable.

This is the cdb text output from the crash, which includes output from the !exploitable tool.

This is the minimized version of the crashing test case. It is the "least different" version of the original fuzzed file that caused a specific crash (hash).

This is the cdb output for an exception that is continued <n> number of times. One file is provided for each continued exception until an uncontinuable exception is encountered, or the handled exception limit has been reached, or the target application proceeds without encountering another exception.

Fuzzing on your own

Once you are comfortable with FOE's default ImageMagick fuzz run, you can try fuzzing an application of your choice. The first step is to place seed
files into the FOE seedfiles directory. These are the files that will be mangled and opened by the target application. Next modify the foe.yaml file to suit your needs.  The foe.yaml file is documented to describe what each of the features mean. The important parts to modify are:

campaign: id:
    This field is used in determining the fuzzing campaign, and subsequently, where the results should be stored. This should probably be the target application name and version.

campaign: use_buttonclicker:
    When fuzzing a GUI application, the FOE button clicker can increase throughput and code coverage. Note that the button clicker is not configurable, but rather it has a built-in heuristic for determining which buttons to click.

target: program:
    This is the full path to the target application that you wish to fuzz.

target: cmdline_template:
    This specifies the commandline syntax for invoking the target application.

runner: runtimeout:
    This value specifies how long FOE should wait before terminating the application and moving on to the next iteration. Note that this setting only applies to the "winrun" runner (32-bit Windows XP and Server 2003 systems).

debugger: runtimeout:
    This value specifies how long FOE should allow the target application to run when it is invoked from the debugger. On platforms that use the "null" runner (64-bit Windows or Windows Vista or newer), this is the only timeout value that is used.

FOE periodically saves state of a fuzzing campaign, so it will by default continue a cached campaign if foe.yaml has not been modified.
To clear the FOE cached state, run:
For additional options, run:
tools\ --help

Digging deeper into results

When FOE has produced results, you may wish to perform some additional steps.

Finding interesting crashes:

With some target applications, FOE may produce too many uniquely-crashing test cases to investigate manually in a reasonable amount of time. We have provided a script called to pick out crashes that are most likely to be exploitable and list those cases in a ranked order (most exploitable first).

To run this script, run:
For command-line usage, run:
tools\ --help

Reproducing crashes:

The tools\ script can be used to reproduce a crash by running it in the same manner that FOE did.
For command-line usage, run:
tools\ --help

Comparing zip-based files:

The tools\ script can be used to compare zip-based files.
For command-line usage, run:
tools\ --help

Minimization to string:

Say you have a crashing test case, but you really need to get it to a proof-of-concept exploit. The problem is when you load the crash into your debugger you can't easily tell which registers, stack values, or memory locations are under your control. But what if you could change the crashing test case so that it had only the bytes required to cause that crash, and the rest were all masked out with a fixed value, say "x" (0x78)? Then you'd know that if you saw EIP=0x78787878, you may already be a winner. The minimize-to-string option does just that. To get command-line usage of the minimizer, run:
tools\ --help

To minimize a crashing testcase to the Metasploit string pattern, run:
tools\ --stringmode <crashing_testcase>

When minimizing to the Metasploit pattern, FOE will use the resulting byte map to create an additional minimized file that uses a string of 'x' characters. Note that this file is not guaranteed to produce the same crash as the original string minimization.

Metasploit pattern enumeration:

Especially with larger files, you may notice that the Metasploit pattern repeats several times over the length of a Metasploit-minimized crasher. Given any particular dword, it may not be obvious which instance is the one that you are dealing with. This is where the tools\ script comes in handy. For example, let's say that you have a crasher.doc were EIP = "Aa0A" If you run: tools\ Aa0A crasher.doc you will end up with a file called crasher-enum.doc.  With this file, every instance of the byte pattern "Aa0A" will be replaced with a unique, incrementing replacement. For example, "0a0A", "1a0A", "2a0A", etc. Now when you open crasher-enum.doc, you could for example get EIP = "5a0A". If you search for that pattern in the file, there should be only once instance of it. Note that you can use a search pattern of any length and you can also search for hex values. For example: "\x01\x02\x03\x04"

Included Fuzzing Strategies

bytemut: replace bytes with random values
swap: swap adjacent bytes
wave: cycle through every possible single-byte value, sequentially
drop: removes one byte from the file for each position in the file
insert: inserts a random byte for each position in the file
truncate: truncates bytes from the end of the file
crmut: replace carriage return bytes with random values
crlfmut: replace carriage return and linefeed bytes with random values
nullmut: replace null bytes with random values
verify: do not mutate file. Used for verifying crashing testcases
range_list: byte ranges to be fuzzed. One range per line, hex or decimal

Verifying crashing results

FOE can be used to verify crashing test cases. This can be useful for when a new version of an application is released or if you are the developer and you want to see how many uniquely-crashing test cases disappear when you fix a bug. To perform a verfification campaign:

  1. Run tools\ to collect all of the crashing cases from a campaign. By default it will copy all of the uniquely-crashing test cases to the "seedfiles" directory, which should be empty.
  2. Modify configs\foe.yaml to use the "verify" fuzzer and also specify a new campaign ID.

When you run FOE, it will run each case with the target application, and cases that still crash will be placed in the results directory for the new campaign.

Manually Installing FOE

If you have installed FOE using the installer, you can skip this section. To install FOE manually, you will need the following prerequisites: