Question about file extraction provided by SecurityOnion

[C.L. Martinez]

Hi all,

Is it possible to extract .zip, .docx, .xlsx files apart of exe files
defined in the extraction file??

I can see another mime type formats apart exe in the extraction
definition, like:

global ext_map: table[string] of string = {
["application/x-dosexec"] = "exe",
["text/plain"] = "txt",
["image/jpeg"] = "jpg",
["image/png"] = "png",
["text/html"] = "html",
} &default ="";

.. but only exe files are extracted, and only for http service...

And another question. Is it possible to extract files (only zips or pdf)
from smtp events??

Thanks.

[Doug Burks]

Hi C.L.,

Replies inline.

On Fri, Sep 4, 2015 at 4:03 PM, C.L. Martinez <carlo...@gmail.com> wrote:
> Hi all,
>
> Is it possible to extract .zip, .docx, .xlsx files apart of exe files
> defined in the extraction file??
>
> I can see another mime type formats apart exe in the extraction definition,
> like:
>
> global ext_map: table[string] of string = {
> ["application/x-dosexec"] = "exe",
> ["text/plain"] = "txt",
> ["image/jpeg"] = "jpg",
> ["image/png"] = "png",
> ["text/html"] = "html",
> } &default ="";

That's just a default mapping table that maps mime types to file extensions...

> .. but only exe files are extracted, and only for http service...

...the actual extraction happens later in that script. In the
file_sniff event handler, the following lines tell Bro to ONLY extract
EXE files:

if ( ! meta?$mime_type || meta$mime_type != "application/x-dosexec" )
return;

You could change this to extract other file types.

> And another question. Is it possible to extract files (only zips or pdf)
> from smtp events??

Yes, you should be able to add code to check f$source to see if it is
SMTP and check the meta$mime_type for zips or pdf.


--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Seth Hall]

> On Sep 4, 2015, at 9:33 PM, Doug Burks <doug....@gmail.com> wrote:
>
> Yes, you should be able to add code to check f$source to see if it is
> SMTP and check the meta$mime_type for zips or pdf.

For fun, here are all of the mime type that Bro 2.4 identifies by default (ugh, now that I look at this, I see that I probably have more work to do on these)...

application/dicom
application/epub+zip
application/f4m
application/font-woff
application/java-archive
application/javascript
application/mac-binhex40
application/marc
application/msword
application/ocsp-request
application/ocsp-response
application/ogg
application/pdf
application/pgp
application/pgp-keys
application/pgp-signature
application/pkcs7-signature
application/postscript
application/skp
application/sla
application/soap+xml
application/vnd.cups-raster
application/vnd.fdf
application/vnd.fdo.journal
application/vnd.font-fontforge-sfd
application/vnd.google-earth.kml+xml
application/vnd.google-earth.kmz
application/vnd.lotus-wordpro
application/vnd.ms-cab-compressed
application/vnd.ms-fontobject
application/vnd.ms-opentype
application/vnd.ms-tnef
application/vnd.oasis.opendocument.chart
application/vnd.oasis.opendocument.chart-template
application/vnd.oasis.opendocument.database
application/vnd.oasis.opendocument.formula
application/vnd.oasis.opendocument.formula-template
application/vnd.oasis.opendocument.graphics
application/vnd.oasis.opendocument.graphics-template
application/vnd.oasis.opendocument.image
application/vnd.oasis.opendocument.image-template
application/vnd.oasis.opendocument.presentation
application/vnd.oasis.opendocument.presentation-template
application/vnd.oasis.opendocument.spreadsheet
application/vnd.oasis.opendocument.spreadsheet-template
application/vnd.oasis.opendocument.text
application/vnd.oasis.opendocument.text-master
application/vnd.oasis.opendocument.text-template
application/vnd.oasis.opendocument.text-web
application/vnd.openxmlformats-officedocument
application/vnd.openxmlformats-officedocument.presentationml.presentation
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/vnd.rn-realmedia
application/vnd.symbian.install
application/vnd.tcpdump.pcap
application/warc
application/x-7z-compressed
application/x-arc
application/x-archive
application/x-arj
application/x-bittorrent
application/x-bzip2
application/x-compress
application/x-coredump
application/x-cpio
application/x-dbm
application/x-debian-package
application/x-dmg
application/x-dosexec
application/x-dvi
application/x-eet
application/x-elc
application/x-epoc-agenda
application/x-epoc-app
application/x-epoc-data
application/x-epoc-jotter
application/x-epoc-opl
application/x-epoc-opo
application/x-epoc-sheet
application/x-epoc-word
application/x-executable
application/x-font-sfn
application/x-font-ttf
application/x-freemind
application/x-freeplane
application/x-gdbm
application/x-gnucash
application/x-gnumeric
application/x-gnupg-keyring
application/x-gzip
application/x-hdf
application/x-ia-arc
application/x-ichitaro4
application/x-ichitaro5
application/x-ichitaro6
application/x-java-applet
application/x-java-jce-keystore
application/x-java-jnlp-file
application/x-java-keystore
application/x-java-pack200
application/x-kdelnk
application/x-lha
application/x-lrzip
application/x-lz4
application/x-lzh
application/x-lzip
application/x-lzma
application/x-mach-o-executable
application/x-mif
application/x-ms-reader
application/x-ms-shortcut
application/x-msaccess
application/x-object
application/x-pem
application/x-pgp-keyring
application/x-quark-xpress-3
application/x-quicktime-player
application/x-rar
application/x-rpm
application/x-sc
application/x-scribus
application/x-setupscript
application/x-shar
application/x-sharedlib
application/x-shockwave-flash
application/x-stuffit
application/x-svr4-package
application/x-tar
application/x-tokyocabinet-btree
application/x-tokyocabinet-fixed
application/x-tokyocabinet-hash
application/x-tokyocabinet-table
application/x-xar
application/x-xz
application/x-zoo
application/xml
application/xml-rpc
application/xml-sitemap
application/zip
audio/basic
audio/m4a
audio/midi
audio/mp4
audio/mpeg
audio/vnd.dolby.dd-raw
audio/x-adpcm
audio/x-aiff
audio/x-ape
audio/x-dec-basic
audio/x-flac
audio/x-hx-aac-adif
audio/x-hx-aac-adts
audio/x-mod
audio/x-mp4a-latm
audio/x-musepack
audio/x-pn-realaudio
audio/x-w64
audio/x-wav
chemical/x-pdb
image/gif
image/jp2
image/jpeg
image/jpm
image/jpx
image/png
image/svg+xml
image/tiff
image/vnd.adobe.photoshop
image/vnd.djvu
image/vnd.dwg
image/x-award-bioslogo
image/x-award-bmp
image/x-canon-cr2
image/x-canon-crw
image/x-coreldraw
image/x-cpi
image/x-cursor
image/x-dpx
image/x-epoc-mbm
image/x-epoc-sketch
image/x-exr
image/x-icon
image/x-lss16
image/x-ms-bmp
image/x-niff
image/x-olympus-orf
image/x-paintnet
image/x-polar-monitor-bitmap
image/x-portable-bitmap
image/x-portable-greymap
image/x-portable-pixmap
image/x-quicktime
image/x-x3f
image/x-xcf
image/x-xcursor
magnus-internal/cold-fusion
message/news
message/rfc822
model/vrml
model/x3d
text/PGP
text/atom
text/calendar
text/html
text/inf
text/json
text/plain
text/rss
text/rtf
text/x-awk
text/x-cross-domain-policy
text/x-diff
text/x-lua
text/x-m4
text/x-msdos-batch
text/x-perl
text/x-php
text/x-po
text/x-python
text/x-ruby
text/x-shellscript
text/x-tcl
text/x-vcard
text/x-xmcd
video/3gpp
video/h264
video/mj2
video/mp2p
video/mp2t
video/mp4
video/mpeg
video/mpv
video/quicktime
video/webm
video/x-flc
video/x-fli
video/x-flv
video/x-jng
video/x-matroska
video/x-mng
video/x-ms-asf
video/x-msvideo
video/x-sgi-movie
x-epoc/x-sisx-app

Have fun!

.Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

[C. L. Martinez]

Sorry for this later reply. Replies inline.

On Sat, Sep 5, 2015 at 1:33 AM, Doug Burks <doug....@gmail.com> wrote:
> Hi C.L.,
>
> Replies inline.
>
> On Fri, Sep 4, 2015 at 4:03 PM, C.L. Martinez <carlo...@gmail.com> wrote:
>> Hi all,
>>
>> Is it possible to extract .zip, .docx, .xlsx files apart of exe files
>> defined in the extraction file??
>>
>> I can see another mime type formats apart exe in the extraction definition,
>> like:
>>
>> global ext_map: table[string] of string = {
>> ["application/x-dosexec"] = "exe",
>> ["text/plain"] = "txt",
>> ["image/jpeg"] = "jpg",
>> ["image/png"] = "png",
>> ["text/html"] = "html",
>> } &default ="";
>
> That's just a default mapping table that maps mime types to file extensions...
>
>> .. but only exe files are extracted, and only for http service...
>
> ...the actual extraction happens later in that script. In the
> file_sniff event handler, the following lines tell Bro to ONLY extract
> EXE files:
>
> if ( ! meta?$mime_type || meta$mime_type != "application/x-dosexec" )
> return;
>
> You could change this to extract other file types.

Uhmm . I have added the following mime types:

if ( ! meta?$mime_type || meta$mime_type != "application/x-dosexec" ||

meta$mime_type != "application/msword" || meta$mime_type !=
"application/x-gzip" || meta$mime
_type != "application/x-7z-compressed" || meta$mime_type !=
"application/zip" || meta$mime_type != "application/pdf" ||
meta$mime_type != "application/vnd.openxmlformat
s-officedocument.wordprocessingml.document" || meta$mime_type !=
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" ||
meta$mime_type != "application/v
nd.openxmlformats-officedocument.presentationml.presentation" ||
meta$mime_type != "application/x-executable" )

... but nothing is extracted ... What am I doing wrong??

>> And another question. Is it possible to extract files (only zips or pdf)
>> from smtp events??
>
> Yes, you should be able to add code to check f$source to see if it is
> SMTP and check the meta$mime_type for zips or pdf.
>

Uhmm .. but reading Bro's docs, using only f$source like in the script
appears, it wil extract all attachments, right??

[riema...@gmail.com]

I did the same thing. I think the if statement logic is wrong. Try this:

if ( ! meta?$mime_type || meta$mime_type != "application/x-dosexec" &&
meta$mime_type != "application/msword" && meta$mime_type !=
"application/x-gzip" && ... )

-de

[C.L. Martinez]

Thanks riemann913. It works.

Now, I would like to extract files from http and smtp traffic only...
Maybe I need to put this:

if ( f$source != "SMTP" && f$source != "HTTP" )
return;

before "if ( ! meta?$mime_type || meta$mime_type != ...."??

Is it correct??

[Doug Burks]

Just like the mime_type block checks to see if the mime_type exists
(meta?$mime_type) before checking for any specific value, you'll
probably want to do the same for source.

[Tajammul Rafique]

Hi all

I tried using this script, but the noting is extracted, the folder "extracted' is is empty. The goal is the capture any attachment uploaded or downloaded

global ext_map: table[string] of string = {

["application/x-dosexec"] = "exe",

["application/msword"] = "doc",

["application/x-dmg"] = "dmg",

["application/x-gzip"] = "gz",

["application/x-rar"] = "rar",

["application/x-tar"] = "tar",

["application/x-xar"] = "pkg",

["application/x-rpm"] = "rpm",

["application/x-stuffit"] = "sif",

["application/x-archive"] = "",

["application/x-arc"] = "arc",

["application/x-eet"] = "eet",

["application/x-zoo"] = "zoo",

["application/x-lz4"] = "lz4",

["application/x-lrzip"] = "lrz",

["application/x-lzh"] = "lzh",

["application/warc"] = "warc",

["application/x-7z-compressed"] ="7z",

["application/x-xz"] = "xz",

["application/x-lha"] = "lha",

["application/x-arj"] = "arj",

["application/x-cpio"] = "cpio",

["application/x-compress"] = "",

["application/x-lzma"] = "",

["application/zip"] = "zip",

["application/vnd.ms-cab-compressed"] = "cab",

["application/pdf"] = "pdf",

["application/vnd.openxmlformats-officedocument.wordprocessingml.document"] = "docx",

["application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"] = "xlsx",

["application/vnd.openxmlformats-officedocument.presentationml.presentation"] ="pptx",

["application/font-woff"] = "woff",

["application/x-font-ttf"] = "ttf",

["application/vnd.ms-fontobject"] = "eot",

["application/x-font-sfn"] = "",

["application/vnd.ms-opentype"] = "otf",

["application/x-mif"] = "mif",

["application/vnd.font-fontforge-sfd"] = "sfd",

["audio/mpeg"] = "mp3",

["audo/m4a"] = "mp4",

["image/tiff"] = "tiff",

["image/gif"] = "gif",

["image/jpeg"] = "jpg",

["image/x-ms-bmp"] = "bmp",

["image/x-icon"] = "ico",

["image/x-cursor"] = "cur",

["image/vnd.adobe.photoshop"] = "pnd",

["image/png"] = "png",

["text/html"] = "html",

["text/plain"] = "txt",

["text/json"] = "json",

["text/rtf"] = "rtf",

["application/xml"] = "xml",

["text/rss"] = "rss",

["application/java-archive"] = "jar",

["application/x-java-applet"] = "jar",

["application/x-shockwave-flash"] = "swf",

["application/pkcs7-signature"] = "p7",

["application/x-pem"] = "pem",

["application/x-java-jnlp-file"] = "jnlp",

["application/vnd.tcpdump.pcap"] = "pcap",

["text/x-shellscript"] = "sh",

["text/x-perl"] = "pl",

["text/x-ruby"] = "rb",

["text/x-python"] = "py",

["text/x-awk"] = "awk",

["text/x-lua"] ="lua",

["application/javascript"] = "js",

["text/x-php"] = "php",

["application/x-executable"] = "",

["application/x-coredump"] = "core",

["video/x-flv"] = "flv",

["video/x-fli"] = "fli",

["video/x-flc"] = "flc",

["video/mj2"] = "mj2",

["video/x-mng"] = "mng",

["video/x-jng"] = "jng",

["video/mpeg"] = "mpg",

["video/mpv"] = "mpv",

["video/h264"] = "264",

["video/webm"] = "webm",

["video/matroska"] = "mkv",

["vidoe/x-sgi-movie"] = "sgi",

["video/quicktime"] = "qt",

["video/mp4"] = "mp4",

["video/3gpp"] = "3gp",

} &default ="";

module FileExtraction;

export {

        ## Path to store files

        const path: string = "" &redef;

        ## Hook to include files in extraction

        global extract: hook(f: fa_file, meta: fa_metadata);

        ## Hook to exclude files from extraction

        global ignore: hook(f: fa_file, meta: fa_metadata);

}

event file_sniff(f: fa_file, meta: fa_metadata)

        {

        if ( meta?$mime_type && !hook FileExtraction::extract(f, meta) )

                {

                if ( !hook FileExtraction::ignore(f, meta) )

                        return;

                if ( meta$mime_type in mime_to_ext )

                        local fext = mime_to_ext[meta$mime_type];

                else

                        fext = split_string(meta$mime_type, /\//)[1];

                if ( f?$info && f$info?$filename )

                        local fname = fmt("/nsm/bro/extracted/%s%s-%s", path, f$source,

f$info$filename);

                else

                        fname = fmt("/nsm/bro/extracted/%s%s-%s.%s", path, f$source, f$id,

fext);

                Files::add_analyzer(f, Files::ANALYZER_EXTRACT,

                        [$extract_filename=fname]);

                }

        }