Logging to a saz file in real time

[Steve Shier]

I want Fiddler 4 to log my captured traffic in real time or at worst periodically (perhaps every 5 minutes) so I can view the captured traffic in the morning from an overnight Fiddler session without losing an session data.  My filter is set to Keep:All sessions". 

I don't want to set Capture traffic before I leave work at the end of the day and then save all sessions manually in the morning because I am concerned I could lose session data overnight. The reason I need this is that we have a search engine which crawls thousands of pages overnight and I want to debug a problem we are having with it.  I cannot afford to lose data that might occur because Fiddler starts overwriting session data because its allocated memory is filled in the middle of the night.  I would rather have it continuously log to disk so I can be confident I didn't lose any data.

How do I continuously log to a .saz file on disk and where do I set the path to the .saz file?

TIA,

Steve Shier

[EricLaw]

>Fiddler starts overwriting session data because its allocated memory is filled in the middle of the night. 

I'm not really sure I understand your concern here. Fiddler doesn't have any feature to "overwrite" session data; its "allocated memory" is only limited by the size of your swap file, which automatically grows on most modern versions of Windows until disk space is exhausted (at which point saving to a SAZ wouldn't likely work anyway).

http://pastebin.com/KVnx8e8d shows the general set of APIs you'd use in your FiddlerScript to automatically dump Sessions to a SAZ file.

Regardless of your data storage strategy, to minimize the amount of data stored in ram or on disk, you should consider using Filters or FiddlerScript to drop traffic which isn't related to whatever problem you're attempting to track.

[Steve Shier]

Thank you Eric.  My concern was based on my lack of knowledge of how Fiddler uses RAM.  I thought it might have a limited circular buffer that starts to overwrite before all data is logged. I see that there is a limit though larger than I expected. Yesterday, I ran Fiddler while my search engine did the crawl and it used 5GB+ of RAM. I then tried to do a save to a .saz file. I had to abort the save after a while-it had saved 3GB+ after about 10 minutes.    When I look at the session, it just contains urls that I need. The crawler is already filtering out images and other urls I don't need.  If there is a way to filter the data per url rather than which urls are saved to the session, I could do that to reduce memory use.  I need every url, in the log to trap my problem. However,  I only need the cookies and Http response codes to for each url.  I don't really need the other data retrieved with each url.

The solution I was looking for could be based on the link http://pastebin.com/KVnx8e8d you posted.  I would like to clear all the session data after it is saved to keep memory use down but still be able to log additional session data that comes in after the save. Does the FiddlerObject.UI.actSaveSessionsToZip(sFile); statement clear out the session data in memory and if not, how is that done?  Also, how do I trigger a save at certain session data size.  E.g., after 10MB of session data I want to write a .saz file.  When the next 10MB of data comes in, I want to log the next .saz file with a different name until I am done.  The object of this would be to have small manageable .saz files when done (a 3GB file takes a long time to load into Fiddler later to examine what happened) and also to keep memory use modest since I want to leave our search server in a state where it is not strapped for memory.

Fiddler is a great tool. I was wondering why there is not an automatic save to file or log4net (with separate config file) in Fiddler.  It seems to me this would be a very useful feature. log4net would be particularly useful because you can use a rolling appender and have the logging roll over to a new file after the first log file gets to a specified size.  It is also possible to filter using log4net.  I am guessing that it might be possible to have a Fiddler script that uses log4net although I am not sure how to do that.

Thanks,

Steve

[EricLaw]

Thanks for explaining, Steve!

Building a plugin for Fiddler that uses log4net to write traffic to a log file would be a straightforward exercise; you'd only need to write about a dozen lines of Fiddler-related code and everything else would be what the logging framework requires. (I haven't used that framework myself, which is probably the primary reason I haven't written such a thing myself).

The call FiddlerObject.UI.actRemoveAllSessions(); removes all of the sessions from Fiddler's Web Sessions list, which results in the references getting garbage collected by the system.

If the only thing you need is [ID, URL, OutboundCookie, ResponseCode] then keeping around all of the HTTP data (in the SAZ file) is definitely overkill. Instead, you'd be better off just keeping the data you need. (If you use FiddlerCore instead of Fiddler, you could do the entire thing headless).

[Steve Shier]

Thanks a bunch Eric!  I can use actRemoveAllSessions.  I will filter down to just the data I need to reduce session size.

I will also consider using FiddlerCore since I already have log4net code I can use.

Steve

[EricLaw]

Neat, log4net is pretty cool. Here's a Fiddler extension which uses it.

using System;

using log4net;
using Fiddler;
 
[assembly: log4net.Config.XmlConfigurator(Watch=true)]
[assembly: Fiddler.RequiredVersion("2.4.5.0")]
namespace RTLogger
{
    public class RTLogger: IAutoTamper
    {
        private static readonly ILog log = LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);
        public void AutoTamperRequestAfter(Session oSession) { }
        public void AutoTamperRequestBefore(Session oSession) { }
        public void AutoTamperResponseAfter(Session oSession) { }
        public void AutoTamperResponseBefore(Session oSession)
        {

            // oSession["ui-hide"] = "Hide from Session list to allow GC";
            log.InfoFormat("#{0}\t{1}\t{2}",
                oSession.id,
                oSession.fullUrl,
                oSession.responseCode);
        }
 
        public void OnBeforeReturningError(Session oSession)
        {
            log.InfoFormat("*{0}\t{1}\t{2}",
                oSession.id,
                oSession.fullUrl,
                oSession.responseCode);
        }
 
        public void OnBeforeUnload()
        {
            log.Info("RTLogger Unloading");
        }
 
        public void OnLoad()
        {
            log.Info("RTLogger Loading");
            if (log.IsInfoEnabled) FiddlerApplication.AlertUser("RTLogger", "Logging is enabled");
        }
    }
}

 

And you just update fiddler.exe.config with:

<configuration>

<configSections>

<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,Log4net"/>

</configSections>

<log4net>

<root>

<level value="ALL"/>

<appender-ref ref="LogFileAppender" />

</root>

<appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender" >

<file value="c:\temp\log.txt"/>

<appendToFile value="true" />

<rollingStyle value="Size" />

<maxSizeRollBackups value="5" />

<maximumFileSize value="10MB" />

<staticLogFileName value="true" />

<layout type="log4net.Layout.PatternLayout">

<conversionPattern value="%d %m%n" />

</layout>

</appender>

</log4net>

<runtime>

<generatePublisherEvidence enabled="false"/>

</runtime>

</configuration>

[Steve Shier]

Yes log4net is a really nice tool.  Thanks so much Eric for reducing my workload.  very helpful!

Steve