Re: Boofuzz uses up all RAM and CPU after a long run

[Joshua Pereyda]

It sounds like the fuzzing process is repeatedly starting the target. Boofuzz will call procmon.start_target when:

1. It detects a process failure (in this case it will not call stop_target)
   
2. The restart interval is exceeded (see Session.__init__) (in this case it will call stop_target)
   

So multiple targets being started could indicate:

1. Failures are being detected even though the target is not crashing. Therefore the target is being "re"started repeatedly.
2. The restart interval is being hit, but the stop_target method is not working.

To find out if 2 is the case, I would look at the process monitor output (feel free to post it here). To find out if 1 is the case, I would check out the boofuzz output to see if many failures are being reported (also feel free to post here).

One note: It can be helpful to run your fuzzer and target on different machines. VMs are nice for this, and you can even download free Windows VMs from Microsoft: https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ (this won't actually solve your root problem though; just some general input)

On Friday, April 14, 2017 at 2:07:23 AM UTC-7, martin...@gmail.com wrote:

I found a workaround by killing all target processes that were running for more than X seconds, thus always leaving the current fuzzed target process running:

get-process target_name | ? { ([DateTime]::Now - $_.StartTime).TotalSeconds -gt X } | stop-process 

Still I would like to know if this is by design or if the problem is in my target not exiting properly.

On Friday, 14 April 2017 10:55:11 UTC+2, mar wrote:

Hi, 

I leave Boofuzz over night and in the morning I find the system crashed with a lot of out of memory errors.

When I look in Task Manager I see that it spawns a new process of the target for each new test case and after a long run it creates so many processes of the same target that they fill up the RAM and the CPU is constantly at 100%. I am fuzzing on Windows 10 machine with latest Boofuzz. 

A workaround that I tried is to schedule a task that kills all target process every hour, but then Boofuz exits with: 

"Error!!!! Cannot connect to target; target presumed down. Note: Normally a failure should be detected, and the target reset. This error may mean you have no restart method configured, or your error detection is not working."

This error occurs even if I leave one target process running (run as Administrator), while I kill the rest (kill all target process not run by Administrator). Seems that it exits if the currently fuzzed target process exits. 

Is this expected behaviour? How do I fix it or go around it? 

[Joshua Pereyda]

Nice, way to get to the bottom of it! In that case, you'll want to set check_data_received_each_request to False when you create your Session object. I think I added it when I was in your same situation.

Apparently the visibility on that option is not high enough. :/ Maybe we should change the default. Let me know if it works.

On Saturday, April 15, 2017 at 1:34:32 AM UTC-7, martin...@gmail.com wrote:

Thanks for the reply!

To me it seems to me that it goes like this:

I send few valid packets in order to go into a certain depth level and the target replies to all of those. 

Then a fuzzed packet is sent to which the target doesn't reply, because the packet is malformed. So the fuzzer just prints "Nothing received from target" and "Restarting target process". 

I think it assumes that the target has exited since it is not replying, so it start a new instance. 

I also see a ton of "Nothing received from target." in the web interface (one for each malformed packet that didn't triggered response from the target).

Some fuzzed packets do trigger response, when they pass through the target validation, but most don't.

Please, also note that I have increased the crash_threshold variable to a very large number to make Boofuzz fuzz even when the target is not responding. I did this because initially Boofuzz stopped fuzzing the current request when the target didn't replied after a few trials. However my target is expected to not answer when the packets are malformed above a certain degree, so I wanted it to continue fuzzing the rest of the request.

Here is some output:

Process monitor

[10:02.18] starting target process

[10:02.18] done. target up and running, giving it 5 seconds to settle in.

2017-04-15 10:02:18 Server started

[10:02.26] updating target process name to 'server.exe'

[10:02.26] updating stop commands to: ['wmic process where (name="server.exe") delete']

[10:02.26] updating start commands to: ['C:\\server\\server.exe']

[10:02.26] pre_send(15440)

2017-04-15 10:02:26 [127.0.0.1] Client added

2017-04-15 10:02:26 [127.0.0.1] The client requests PDU size of 480 bytes

2017-04-15 10:02:26 [127.0.0.1] Read request, ID:0x0011 INDEX:0x0000 --> OK

2017-04-15 10:02:26 [127.0.0.1] Malformed PDU of 28 bytes rejected

2017-04-15 10:02:26 [127.0.0.1] Client disconnected by peer

[10:02.26] starting target process

[10:02.26] done. target up and running, giving it 5 seconds to settle in.

2017-04-15 10:02:26 Server started

(the whole thing repeats again)

 Boofuzz

2017-04-15 10:07:37,097]   Test Step: Prep Node 'start_upload'

[2017-04-15 10:07:37,099]     Transmitting 35 bytes: 03 00 00 23 02 f0 80 32 01 00 00 06 00 00 12 00 00 1d 00 00 00 00 00 00 00 09 5f 30 42 30 30 30 30 30 41 b'\x03\x00\x00#\x02\xf0\x802\x01\x00\x00\x06\x00\x00\x12\x00\x00\x1d\x00\x00\x00\x00\x00\x00\x00\t_0B00000A'

[2017-04-15 10:07:37,100]     Info: 35 bytes sent

[2017-04-15 10:07:37,100]     Info: Receiving...

[2017-04-15 10:07:37,100]     Received: 03 00 00 13 02 f0 80 32 02 00 00 06 00 00 00 00 00 d2 41 b'\x03\x00\x00\x13\x02\xf0\x802\x02\x00\x00\x06\x00\x00\x00\x00\x00\xd2A'

[2017-04-15 10:07:37,101]     Check: Verify some data was received from the target.

[2017-04-15 10:07:37,101]       Check OK: Some data received from target.

[2017-04-15 10:07:37,101]   Test Step: Fuzzing Node 'start_download'

[2017-04-15 10:07:37,102]     Transmitting 35 bytes: 03 00 00 23 02 f0 80 32 01 00 00 07 00 00 12 10 05 1d 00 00 00 00 00 00 00 09 5f 30 42 30 30 30 30 30 41 b'\x03\x00\x00#\x02\xf0\x802\x01\x00\x00\x07\x00\x00\x12\x10\x05\x1d\x00\x00\x00\x00\x00\x00\x00\t_0B00000A'

[2017-04-15 10:07:37,102]     Info: 35 bytes sent

[2017-04-15 10:07:37,102]     Info: Receiving...

[2017-04-15 10:07:37,104]     Received:  b''

[2017-04-15 10:07:37,104]     Check: Verify some data was received from the target.

[2017-04-15 10:07:37,109]       Check Failed: Nothing received from target.

[2017-04-15 10:07:37,109]   Test Step: Calling post_send function:

[2017-04-15 10:07:37,109]     Info: No post_send callback registered.

[2017-04-15 10:07:37,110]   Test Step: Sleep between tests.

[2017-04-15 10:07:37,110]     Info: sleeping for 0.030000 seconds

[2017-04-15 10:07:37,140]   Test Step: Contact process monitor

[2017-04-15 10:07:37,140]     Check: procmon.post_send()

[2017-04-15 10:07:37,157]       Check OK: No crash detected.

[2017-04-15 10:07:37,158]   Test Step: Failure summary

[2017-04-15 10:07:37,158]     Info: Nothing received from target.

[2017-04-15 10:07:37,159]   Test Step: restarting target

[2017-04-15 10:07:37,159]     Info: restarting target process

Seems that the general assumption with Boofuzz is that the target will respond to each request, because if it doesn't respond then a new target is started (the old assumed crashed due to the lack of response, even though it may still be running) and the fuzzing process is terminated after a few trials (if not tweaked as I did).

[Joshua Pereyda]

Interesting... I've found bugs just as a result of slamming a target with connections and requests. The sleep_time parameter can be used to add delay between test cases.