Article: The Art of Software Security Testing
Shawn
which security methods should be used to investigate file formats.
Fuzzing File Formats
Applications such as Web browsers, image viewers, and media players
regularly process files provided by untrusted remote users. The
formats and encoding of these files, especially those used for
compressed images, video, and audio, are quite complex and thus are
difficult to parse securely. It is therefore essential that the
applications' processing of these files be properly scrutinised and
tested.
As an example of a common file format vulnerability, consider the
following code fragment. It is an example of a style of code commonly
seen parsing binary file formats. The file format may consist of a
file header and a number of sections, each with section headers. Each
section header contains a section size field that describes how many
bytes of data are contained within that section. If the file format
parsing code uses these values unchecked in a memory allocation
request size or as an offset into the file, a denial-of-service or
memory trespass vulnerability may be likely.
Full story here:
http://www.computerweekly.com/Articles/2007/06/12/224710/the-art-of-software-security-testing.htm
Shawn