API or report interface to export scans and/or all vulnerabilities open or closed?

[Bob Rich]

Hi Folks,

Two quick questions:

First, is there a way to export/extract/report on all vulnerabilities for an application, open or closed?  Basically I'm trying to find a way to use ThreadFix to work in the context of multiple scans and/or scanners for a given assessment.  Currently of course it will close vulnerabilities that don't appear in subsequent scans, but this doesn't map to our workflow.

Alternatively, the ability to export the mapped raw scan records would work as well.

Second, are there any plans to add request/response information for a given vulnerability or at least parameter value information for a given vulnerability?  It's very difficult to manage currently with certain MVC applications that send the same parameters to the same controller, differentiated only by parameter values.

Thanks!

Bobs

[Bob Rich]

Hi Folks,

I went ahead and did a little tinkering/bludgeoning in the code today.   Again my use case for Threadfix is almost strictly aggregator/normalizer rather than vulnerability management/sdlc/lifecycle work.  These changes are with that in mind.

First set of changes were to eliminate the scan sequencing and auto-closing of vulnerabilities that are no longer in the results for a given channel.  This was a hack job and simply comments out the relevant portions of code.  I left no provisions to make it optional or modulate the behavior...it's built in to the code and going back requires another build.  This required the following changes:

/threadfix-main/src/main/java/com/denimgroup/threadfix/service/channel/AbstractChannelImporter.java

  int result = scan.getImportTime().compareTo(testDate);

  if (result == 0)

  return ScanImportStatus.DUPLICATE_ERROR;

//else if (result > 0)

// return ScanImportStatus.OLD_SCAN_ERROR;

  }

  }

  }

/threadfix-main/src/main/java/com/denimgroup/threadfix/data/entities/Vulnerability.java

@Transient

public void closeVulnerability(Scan scan, Calendar closeTime) {

  // active = false;

  // if (closeTime == null) {

  // this.closeTime = Calendar.getInstance();

  // } else {

  // this.closeTime = closeTime;

  // }

  //

  //

  // // This constructor maps the objects for us

  // if (scan != null) {

  // new ScanCloseVulnerabilityMap(this, scan);

  // }

}

The effect of these two changes is that you can a) import scans regardless of when they were done relative to others within a given application, and b) vulnerabilities do not close automatically (or at all).  This is good for cases where you have to run multiple scans from one product to get full coverage of the application.

The second set of changes begin to pull some of the attack/variant information into Threadfix.  I basically extended the Finding entity to include the 'attack' or 'injection' string as well as a copy of the full HTTP request and response if available.  This is highly scanner dependent, and i made changes to the NTObjectives, Netsparker, Burp and Arachni plugins to accomodate it.  I then essentially wedged it into the UI and partially into the Vulnerability report so i could get the information back out.

The full diff, *including* the changes listed above, is here:

https://gist.github.com/bobrich/f4fa5562a52feae3f4f5

A couple of screenshots from the vulnerability and finding details are here: 

http://imgur.com/a/H3Pu5

There are clearly going to be bugs in this and there was absolutely no 'design' involved.  I extended Finding rather than SurfaceArea because it seemed to be more likely that it would merge correctly that way, but a new entity would probably be the best option given that some scanners supply multiple copies of the pertinent information, including good/bad comparisons and screenshots.

Not sure if this is of any use to anyone.  If you have any questions just drop me a note.

Thanks

Bob

[Dan Cornell]

Bob:

That's good stuff! I think we currently have one organization with a custom build of ThreadFix that also disables the auto-close functionality. If that is of use to you as well we can look at how we might integrate that into the mainline codebase.

Also we've had some requests to import more of the scanner result data beyond the (CWE, relative URL, [injection point]) that we need to do virtual patching, merging, etc and that is something we want to do but have been working through some other features requests in the meantime. I looked at the screenshot you provided and on first glance it looks pretty cool - we'll be reaching out for more info.

Thanks so much for reaching out and posting this!

Dan

[Bob Rich]

Thanks Dan,

I'm sure you guys will come up with something much better than this.  I just wanted to toss it out there in the event anyone had a short-term need that it would help fulfill.  Of course if you do have any questions I'd be happy to help however i can.

Please extend my thanks to the team for keeping the code base so easy to bootstrap in a new development environment.  It was a treat to be able to get it up and running so quickly.

Take care...

Bob

[Bob Rich]

OK, this will probably be the last bit of noise from me for a while.  I just wanted to chuck up a patch against 2.1M1 this time (rather than master) that includes some additional changes to import scanner detail and recommendations.

Patch is here:  https://gist.github.com/bobrich/3001f30f1c1eb045c7ce     I don't know what i'm doing with git, so it's probably going to take a little work to merge.  This is against what I believe is the most recent pull of 2.1M1 this evening...but again I don't know what I'm doing.

I nuked changes to the vulnerability detail page because i don't have any chops with all of that snazzy new HTML stuff.  So i just added some tables to the finding detail page and that's where all the data lies as far as the UI is concerned:

http://i.imgur.com/a3NiyjO.png

Main changes in this patch are:

- eliminates check for earlier scan

- eliminates auto-close of vulnerabilities that are not in later scans

- refresh Arachni importer to work with XML files from recent releases

- update Arachni, Burp, Netsparker and NTO Spider importers to pull in finding detail, recommendation detail, attack request, attack response and attack string

- importers also attempt to synthesize the raw XML fragment from the import file.  this is also attached to the finding

- modification to findingDetail.jsp to show all of the above

This hasn't been tested substantially, bugs are guaranteed.  

Thanks!

[Dan Cornell]

- refresh Arachni importer to work with XML files from recent releases

The Arachni fixes should now be reflected in the mainline development branch after this commit:

https://github.com/denimgroup/threadfix/commit/18ebc9fdf8d729790abd866f76d717e623a97b6c

Thanks,

Dan