Capturing HTTP traffic using FiddlerCore in C# or VB.NET

[clemen...@hotmail.fr]

Hello,
I would like to capture the traffic URL on a local computer with my program in VisualBasic.NET. I don't know if it's possible using VB.NET, but I am sure that it's possible with C# because this person did it : https://www.youtube.com/watch?v=-ZCsuJtJc_o
The problem is that when I want to make the same program, the code who this person has written didn't work.
I don't find help in documentation on Fiddler website.
Thanks for help !

[EricLaw]

Yes, you can use FiddlerCore in VB.NET if you'd like (https://groups.google.com/forum/#!searchin/httpfiddler/vb.net$20fiddlercore/httpfiddler/dV-HTjXVEHQ/iqzYePHg7FoJ) although it's typically easier to build complex programs (like those that use FiddlerCore) in C#.

[clemen...@hotmail.fr]

Hello,
happy to know that the creator of Fiddler can help me :D ! I am searching for 2 weeks a fiddlercore like ! Now I have successful listed the URL's in VB.NET ! That I would like to do, it's create a program that analyse the HTML code of each page web and extract the text to compare if a word is in the web page texte. The problem is it not work very well, a lot of URL's appears so the WebClient can't intercept all that url, for example if the user is in a web page, the real URL appears, and after, a URL of png, js, and lots of add-ons ...
Do you think it's possible to intercept only the real url ? For exemple http://www.google.fr, http://www.facebook.com ...
Thanks a lot Eric !
Clément.

[clemen...@hotmail.fr]

I have forgottent to show you the code :

Option Explicit On
Imports Fiddler
Imports System.Net
Imports System.IO

Public Class Form1

    Dim url As String = Nothing
    Public Sub New()

        ' This call is required by the designer.
        InitializeComponent()

        ' Add any initialization after the InitializeComponent() call.

        AddHandler FiddlerApplication.BeforeResponse, AddressOf FiddlerBeforeResponseHandler
        AddHandler FiddlerApplication.BeforeRequest, AddressOf FiddlerBeforeRequestHandler

        AddHandler Application.ApplicationExit, AddressOf ShutdownFiddlerApp

        Dim oFlags As FiddlerCoreStartupFlags = FiddlerCoreStartupFlags.Default
        FiddlerApplication.Startup(0, oFlags)
        MsgBox("Started proxy on port " & FiddlerApplication.oProxy.ListenPort)
    End Sub

    Private Sub ShutdownFiddlerApp()
        FiddlerApplication.Shutdown()
        MsgBox("Unloaded proxy")
        Threading.Thread.Sleep(1000)

    End Sub

    Private Sub FiddlerBeforeRequestHandler(ByVal tSession As Session)

        RichTextBox1.BeginInvoke(New AsyncMethodCaller(AddressOf AddText), tSession.fullUrl)
        url = tSession.ToString

    End Sub

    Private Sub FiddlerBeforeResponseHandler(ByVal tSession As Session)


    End Sub

    Public Sub Verification()
        Try
            Dim webClient As New System.Net.WebClient
            Dim result As String = webClient.DownloadString(url)
            Dim client As WebClient = New WebClient()
            Dim data As Stream = client.OpenRead(url)
            Dim reader As StreamReader = New StreamReader(data)
            Dim Page As String = reader.ReadToEnd
            ListeMot.AddRange(IO.File.ReadAllLines("C:\Users\Clément\Documents\ListeDeMots.txt"))
            Dim found As Boolean = False
            For Each s As String In ListeMot
                If Page.ToLower.Contains(" " & s.ToLower & " ") Then
                    MsgBox("Un mot interdit a été détecté :" & s)
                End If
            Next
        Catch
        End Try

    End Sub
    Dim ListeMot As New List(Of String)

    Private Sub AddText(sText As String)

        RichTextBox1.AppendText(sText & vbCrLf)
        Dim Thread As New Threading.Thread(AddressOf Verification)
        Thread.Start()

    End Sub

    Protected Overrides Sub Finalize()
        MyBase.Finalize()
    End Sub

End Class

Public Delegate Sub AsyncMethodCaller(sText As String)

[EricLaw]

I'm not sure I understand your question.

 

Are you asking: "Hey, FiddlerCore sees all of the content downloaded in a webpage, including the html, CSS, script, images, fonts, audio, video, flash objects, etc. How can I restrict my processing to just the HTML?"

 

If so, the simplest way to do this is to look at the Content-Type of the response and only process the HTML. For instance, you'd do something like this:

 

    if oSession.oResponse.headers.ExistsAndContains("Content-Type", "html") then

    ' process the html

    End If

 

On the other hand, maybe you're asking: "Hey, the WebClient object in .NET only downloads the URL I tell it to and doesn't download all of the CSS, script, images, fonts, audio, video, flash objects, etc. How can I download those?" If that's your question, then the answer is that this is somewhat complicated. You can either change your code to use the .NET WebBrowser control instead of WebClient, or attempt to perform HTML parsing and JavaScript execution yourself manually (very complicated).

 

-Eric

 

 

 

[clemen...@hotmail.fr]

Hello Eric,
thanks for your help, sorry for my English I am French ...
You have understand my question, I would like that the richtextbox show only the real URL's for example https://www.facebook.com/, http://www.youtube.com/ ... And not the content of these page for exemple the script, and other ...
I have tryed your code but when I want to access a web page, the loading is very very long ... Maybe I don( well integrated the code :

    Private Sub FiddlerBeforeRequestHandler(ByVal tSession As Session)

        If tSession.oResponse.headers.ExistsAndContains("Content-Type", "html") Then

            RichTextBox1.BeginInvoke(New AsyncMethodCaller(AddressOf AddText), tSession.fullUrl)
            url = tSession.ToString

        End If
    End Sub

( I have changed oSession by tSession because it don't work)
Thanks !

[EricLaw]

You cannot access oResponse inside a handler for BeforeRequest.

 

...but you don't need any of this code if you're really using WebClient to perform the downloads; WebClient only downloads the URLs you tell it to. It does not parse HTML or script and trigger additional downloads like the WebBrowser control does.

[clemen...@hotmail.fr]

Yes I know the Webclient don't download html, but that I want, it's the rich textbox don't "scan" the files who are in the webpage, so the webclient will be less work and will be faster. do you understand ?
I post the code to help you to understand the problem.
Thanks Eric.

Option Explicit On
Imports Fiddler
Imports System.Net
Imports System.IO

Public Class Form1

    Dim url As String = Nothing
    Public Sub New()

        ' This call is required by the designer.
        InitializeComponent()

        ' Add any initialization after the InitializeComponent() call.

        AddHandler FiddlerApplication.BeforeResponse, AddressOf FiddlerBeforeResponseHandler
        AddHandler FiddlerApplication.BeforeRequest, AddressOf FiddlerBeforeRequestHandler

        AddHandler Application.ApplicationExit, AddressOf ShutdownFiddlerApp

        Dim oFlags As FiddlerCoreStartupFlags = FiddlerCoreStartupFlags.Default
        FiddlerApplication.Startup(0, oFlags)
        MsgBox("Started proxy on port " & FiddlerApplication.oProxy.ListenPort)
    End Sub

    Private Sub ShutdownFiddlerApp()
        FiddlerApplication.Shutdown()
        MsgBox("Unloaded proxy")
        Threading.Thread.Sleep(1000)
    End Sub

    Private Sub FiddlerBeforeRequestHandler(ByVal tSession As Session)

        RichTextBox1.BeginInvoke(New AsyncMethodCaller(AddressOf AddText), tSession.fullUrl)
        url = tSession.ToString

    End Sub

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load

[EricLaw]

I don't understand what your question is.

 

FiddlerCore captures all of the requests you make.

WebClient issues only the requests you specify.

 

So what's the problem?

[clemen...@hotmail.fr]

"FiddlerCore captures all of the requests you make. WebClient issues only the requests you specify." Yes I know, and this is my problem.
I would like Fiddler captures only this URL for exemple.
It's possible ?
Thank you in advance Eric.
Clément.

[EricLaw]

Okay, so I understand:

 

You have a list of URLs, each of which you request using the WebClient object.

You want to download all of these URLs.

You only want FiddlerCore to act on some of these URLs.

 

So, to do that, you'd simply examine the .fullUrl property of the oSession object, and if it's one you care about, you process it, otherwise you ignore it.

[clemen...@hotmail.fr]

Hello Eric,

You have a list of URLs, each of which you request using the WebClient object. : Yes but I think, some URL can't be checked by WebClient because they appaers very fast.

You want to download all of these URLs. : No, if I can download the "primary" URL, just the page and not the complement object, java script, images ... It's better !

You only want FiddlerCore to act on some of these URLs. : Yes, I just want that FiddlerCore download or show me the "primary" URL, the page for exemple : https://www.facebook.com/, https://www.google.fr/ ...
You speak me about oSession object, use tSession it's the same no ? Because I don't have oSession in my VB.NET project, just in C#.

I have tried this solution but it's not the better :

    Private Sub FiddlerBeforeRequestHandler(ByVal tSession As Session)

        If tSession.fullUrl.Contains("html") Then

            RichTextBox1.BeginInvoke(New AsyncMethodCaller(AddressOf AddText), tSession.fullUrl)
            url = tSession.ToString

        End If
    End Sub

Again, thank you for your help. I hope we will be able to solve my problem.
Clément.

[clemen...@hotmail.fr]

Hello Eric !
I have found the solution :D ! I filter the URL who appears in the RichTextbox, so fewer links appear and my webclient can work peacefully ! This is the code !

Option Explicit On
Imports Fiddler
Imports System.Net
Imports System.IO

Public Class Form1

    Dim url As String = Nothing

    Public Sub New()
        ' This call is required by the designer.
        InitializeComponent()
        ' Add any initialization after the InitializeComponent() call.
        AddHandler FiddlerApplication.BeforeResponse, AddressOf FiddlerBeforeResponseHandler
        AddHandler FiddlerApplication.BeforeRequest, AddressOf FiddlerBeforeRequestHandler
        AddHandler Application.ApplicationExit, AddressOf ShutdownFiddlerApp
        Dim oFlags As FiddlerCoreStartupFlags = FiddlerCoreStartupFlags.Default
        FiddlerApplication.Startup(0, oFlags)
        MsgBox("Started proxy on port " & FiddlerApplication.oProxy.ListenPort)
    End Sub

    Private Sub ShutdownFiddlerApp()
        FiddlerApplication.Shutdown()
        MsgBox("Unloaded proxy")
        Threading.Thread.Sleep(1000)
    End Sub

    Private Sub FiddlerBeforeRequestHandler(ByVal tSession As Session)
        RichTextBox1.BeginInvoke(New AsyncMethodCaller(AddressOf AddText), tSession.fullUrl)

    End Sub

    Private Sub FiddlerBeforeResponseHandler(ByVal tSession As Session)
    End Sub

    Public Sub Verification()
        Try

            ListeMot.AddRange(IO.File.ReadAllLines("C:\Users\Clément\Documents\ListeDeMots.txt"))
            Dim found As Boolean = False
            For Each s As String In ListeMot

                If url.ToLower.Contains("-" & s.ToLower & "-") Or url.ToLower.Contains("." & s.ToLower & ".") Or url.ToLower.Contains("." & s.ToLower & "-") Or url.ToLower.Contains("-" & s.ToLower & ".") Then
                    MsgBox("Un mot interdit dans l'URL a été détecté :" & s)

                End If
            Next
        Catch
        End Try
        Try
            Dim webClient As New System.Net.WebClient
            Dim result As String = webClient.DownloadString(url)
            Dim client As WebClient = New WebClient()
            Dim data As Stream = client.OpenRead(url)
            Dim reader As StreamReader = New StreamReader(data)
            Dim Page As String = reader.ReadToEnd
            ListeMot.AddRange(IO.File.ReadAllLines("C:\Users\Clément\Documents\ListeDeMots.txt"))
            Dim found As Boolean = False
            For Each s As String In ListeMot
                If Page.ToLower.Contains(" " & s.ToLower & " ") Then
                    MsgBox("Un mot interdit a été détecté :" & s)
                End If
            Next
        Catch
        End Try
    End Sub

    Dim ListeMot As New List(Of String)

    Private Sub AddText(sText As String)

        Dim url2 As Uri
        url2 = New Uri(sText)
        If url2.AbsolutePath.EndsWith(".js") = False And url2.AbsolutePath.EndsWith(".jpg") = False And url2.AbsolutePath.EndsWith(".gif") = False And url2.AbsolutePath.EndsWith(".png") = False And url2.AbsolutePath.EndsWith(".css") = False And url2.AbsolutePath.EndsWith(".ico") = False And url2.AbsolutePath.EndsWith(":443") = False Then
            RichTextBox1.AppendText(sText & vbCrLf)
            url = sText
            Dim Thread As New Threading.Thread(AddressOf Verification)
            Thread.Start()
        End If

    End Sub

    Protected Overrides Sub Finalize()
        MyBase.Finalize()
    End Sub

End Class

Public Delegate Sub AsyncMethodCaller(sText As String)

Thanks Eric ! Bye !