curl works, Mojo::UserAgent doesn't

62 views
Skip to first unread message

Michael Lackhoff

unread,
May 13, 2020, 5:56:26 PM5/13/20
to Mojolicious
I had quite a bit of success recently using Mojo::UserAgent so I tried to replace a curl command to do a backup of my Fritz.box router with M::U.

Here is the curl command:
curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
        --form ConfigExport= http://fritz.box/cgi-bin/firmwarecfg

It should be equivalent to this M::U request:

my $tx = $ua->build_tx(
    POST => 'http://fritz.box/cgi-bin/firmwarecfg' =>
        {
            'Accept'       => '*/*',
            'Content-Type' => 'multipart/form-data',
        } => form => {
            sid                  => $SID,
            ImportExportPassword => $BAKPWD,
            ConfigExport         => '',
    }
);

# for debugging:
print $tx->req->to_string;

$tx = $ua->start($tx);
$tx->res->save_to($OUT);

As far as I can tell both the headers and the POST body is very much the same (except the boundary value to separate the form fields) but to my surprise the curl command works ($OUT is the backup file) but with the M::U version $OUT consists of some HTML output indicating an error.

If I could see a difference I could try to better adjust my script but as I said, they look very much the same (I compared it with the -v and --trace-ascii output of curl), so I run out of ideas what could trigger the differnt response of my Fritz.box.
Any ideas? At the moment I just solve it by using the curl command with "system" but I would prefer a Perl-only solution and what is even more important to me: I want to understand what is going on here.

-Michael

Stefan Adams

unread,
May 13, 2020, 11:08:17 PM5/13/20
to mojolicious
I'm certain you'll be able to accomplish this with M::UA. Can you share what the HTML error produced is?

Can you share the output that you are comparing that you are noticing is so similar between curl and M::UA? You're using -v for curl. Also set the MOJO_CLIENT_DEBUG env variable to 1.

See how it works comparing curl to:

   $ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f ImportExportPassword=$BAKPWD -f ConfigExport= http://fritz.box/cgi-bin/firmwarecfg

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/14df452c-e7a4-4b7c-90a2-9e1f5e15becc%40googlegroups.com.

Scott Wiersdorf

unread,
May 14, 2020, 12:35:07 AM5/14/20
to Mojolicious
It's possible the `-k` option to curl is the difference. The equivalent in Mojo::UserAgent is `$ua->insecure(1)` (which you can chain off of).

Scott

Michael Lackhoff

unread,
May 14, 2020, 7:37:45 AM5/14/20
to Mojolicious
Am 14.05.2020 um 05:08 schrieb Stefan Adams:

> I'm certain you'll be able to accomplish this with M::UA. Can you share

Just to reduce complexity I changed everything to plain HTTP, so no SSL,
keys or certificates.


> what the HTML error produced is?

It is not really a normal error page, just a page from the UI. The most
relevant lines are these:
<p class="ErrorMsg">Invalid variable name.</p>
<p>Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.</p>
[German for: repeat the update(sic! I wanted to backup the box, not
update) or restart your FRITZ!Box]


> Can you share the output that you are comparing that you are noticing is so
> similar between curl and M::UA? You're using -v for curl. Also set the
> MOJO_CLIENT_DEBUG env variable to 1.

Here is what curl gives (with --trace-ascii):
=> Send header, 200 bytes (0xc8)
0000: POST /cgi-bin/firmwarecfg HTTP/1.1
0024: Host: fritz.box
0035: User-Agent: curl/7.58.0
004e: Accept: */*
005b: Content-Length: 370
0070: Content-Type: multipart/form-data; boundary=--------------------
00b0: ----565a580ebeb0c567
00c6:
=> Send data, 370 bytes (0x172)
0000: --------------------------565a580ebeb0c567
002c: Content-Disposition: form-data; name="sid"
0058:
005a: 4e240384c30c909d
006c: --------------------------565a580ebeb0c567
0098: Content-Disposition: form-data; name="ImportExportPassword"
00d5:
00d7: backup
00df: --------------------------565a580ebeb0c567
010b: Content-Disposition: form-data; name="ConfigExport"
0140:
0142:
0144: --------------------------565a580ebeb0c567--
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK



> See how it works comparing curl to:
> 
>    $ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f
> ImportExportPassword=$BAKPWD -f ConfigExport= http

> ://fritz.box/cgi-bin/firmwarecfg <http://fritz.box/cgi-bin/firmwarecfg>

The empty ConfigExport parameter gets lost with this command and the
Content-Type is different:

-- Blocking request (http://fritz.box/cgi-bin/firmwarecfg)
-- Connect 7c84d2d7396d9d2ff9571982f73240cb (http://fritz.box:80)
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
Content-Length: 48\x0d
User-Agent: Mojolicious (Perl)\x0d
Accept-Encoding: gzip\x0d
Content-Type: application/x-www-form-urlencoded\x0d
Host: fritz.box\x0d
\x0d
ImportExportPassword=backup&sid=4e240384c30c909d
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
HTTP/1.1 200 OK\x0d



With DEBUG on and my code (s. my original message and below) I get this:

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
Accept-Encoding: gzip, deflate\x0d
Content-Type: multipart/form-data; boundary=tyUPX\x0d
Accept: */*\x0d
Host: fritz.box\x0d
Content-Length: 230\x0d
User-Agent: Mojolicious (Perl)\x0d
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="ConfigExport"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="ImportExportPassword"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
backup
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="sid"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
33cb272bcd34940b
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX--\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
HTTP/1.1 200 OK\x0d

which for me looks very much the same as the curl version but as I said,
curl gives the backup file whereas M::U just gives the HTML page back,
indicating that something went wrong.
I can only spot very minor differences that is why I am running out of
ideas:
- the boundary used by M::U is quite short
- the form parameters are not in the same order
- Accept-Encoding header (is also sent from Firefox where it works)
- the different SID is normal (my script always makes a fresch login)

Looks very strange to me
-Michael


> On Wed, May 13, 2020, 4:56 PM 'Michael Lackhoff' via Mojolicious <
> mojol...@googlegroups.com> wrote:
> 

>> --
>> You received this message because you are subscribed to the Google Groups
>> "Mojolicious" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to mojolicious...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/mojolicious/14df452c-e7a4-4b7c-90a2-9e1f5e15becc%40googlegroups.com

Felipe Gasper
unread,
May 14, 2020, 7:47:41 AM5/14/20
to mojol...@googlegroups.com
FWIW, you can use libcurl with Mojo via Net::Curl::Promiser::Mojo. That may be your easiest solution?

-FG

On May 14, 2020, at 07:38, 'Michael Lackhoff' via Mojolicious <mojol...@googlegroups.com> wrote:


Michael Lackhoff
unread,
May 14, 2020, 8:06:31 AM5/14/20
to Mojolicious
Well, the easiest solution would be to just use curl.
But M::U is a powerful tool I find myself using it more and more and I would like to better understand such tools.
It looks great but if it has some severe shortcomings I want to know before doing lots of mission critical stuff with it.

But thanks for your suggestion. It is always good to know further options.
-Michael



> 









-- 

You received this message because you are subscribed to the Google Groups "Mojolicious" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mojol...@googlegroups.com.
Tobias Oetiker
unread,
May 14, 2020, 8:17:33 AM5/14/20
to mojol...@googlegroups.com
Michael,

I find the descriptions in https://mojolicious.org/perldoc/Mojo/UserAgent/Transactor#tx pretty helpful

cheers
tobi

To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/03a573c0-5df4-448c-9256-bb851abf7285%40googlegroups.com.

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch to...@oetiker.ch +41 62 775 9902
Felipe Gasper
unread,
May 14, 2020, 8:23:41 AM5/14/20
to mojol...@googlegroups.com
Net::Curl::Promiser wouldn’t keep you from using any of Mojo’s other capabilities. It’s just an alternative way of doing asynchronous HTTP requests; i.e., it’s a substitute specifically for Mojo::UserAgent. It can also facilitate use of FTP, IMAP, etc. by virtue of libcurl’s support for those protocols.



I wouldn’t necessarily describe the interface itself as “easy” since it’s basically a direct port of libcurl’s own interface. The idea is more that, if you know curl is doing what you want, then just use curl rather than M::UA, and keep on trucking with all else that Mojo has to offer.



-FG

>> Content-Disposition: form-data; name="ImportExportPassword"\

>> x0d

>> \x0d

>> 

>> -- Client >>> Server (

>> http://fritz.box/cgi-bin/firmwarecfg

>> )

>> backup

>> -- Client >>> Server (

>> http://fritz.box/cgi-bin/firmwarecfg

>> )

>> \x0d

>> --tyUPX\x0d

>> 

>> -- Client >>> Server (

>> http://fritz.box/cgi-bin/firmwarecfg

>> )

> To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/03a573c0-5df4-448c-9256-bb851abf7285%40googlegroups.com.



Michael Lackhoff
unread,
May 14, 2020, 8:36:26 AM5/14/20
to Mojolicious


Am Donnerstag, 14. Mai 2020 14:23:41 UTC+2 schrieb Felipe Gasper:
 
The idea is more that, if you know curl is doing what you want, then just use curl rather than M::UA, and keep on trucking with all else that Mojo has to offer.




Of course you are right. Your suggestion would be the most sensible way of dealing with it.
I just happen to have a little daemon sitting on my shoulder, saying "you have to understand what is going on here!" "Don't give up until your tool that is made for this kind of task does what you want"
It isn't easy for me to silence this little daemon ;-) and so I still hope there are others more knowledgable than me in this group who have the same kind of ambition...

-Michael
Michael Lackhoff
unread,
May 14, 2020, 8:52:48 AM5/14/20
to Mojolicious

I think I am getting closer.
When I change the sequence of the form parameters in curl, it fails in the same way. So I guess the sequence is important and my question now is:
  Is there a way to pass the form parameters with fixed sequence in M::U (the form-hash of course has a random order)? Perhaps as an array?

-Michael

Michael Lackhoff
unread,
May 14, 2020, 9:47:40 AM5/14/20
to Mojolicious
Ok, got it :-

1. get the content-parts:
$tx->req->content->parts;
2. by checking each parts body_contains method identify them
3. build a new parts array with correct sequence by copying the objects into a new array.
4. change the whole thing:
$tx->req->content->parts($sorted_parts);

This request eventually saves the backup as I want it :-)

You might ask if it is really worth it but I learned quite a lot debugging this problem
And there is a wish: please allow adding form parameters as an array. After all they end up in an array eventually (parts is an array_ref). It is a pitty that the sequence is just determined by a sort on the names i.e. arbitrarily.
In most cases it doesn't matter but if it does this saves a lot of fuzz.

-Michael
Stefan Adams
unread,
May 15, 2020, 12:10:02 AM5/15/20
to mojolicious
On Thu, May 14, 2020 at 7:52 AM 'Michael Lackhoff' via Mojolicious <mojol...@googlegroups.com> wrote:
When I change the sequence of the form parameters in curl, it fails in the same way. So I guess the sequence is important and my question now is:
  Is there a way to pass the form parameters with fixed sequence in M::U (the form-hash of course has a random order)? Perhaps as an array?

Good question.  Probably what you want then is to override the built-in "form" generator.  You can either override (by simply redefining it), or you can create your own generator and use it.

$ua->transactor->add_generator(fritz => sub {
  my ($t, $tx, $args) = @_;
  my ($sid, $pwd, $config) = @$args;
  my $sorted_parts = [...];
  $tx->req->content->parts($sorted_parts);
});
$ua->post('http://fritz.box/cgi-bin/firmwarecfg' => fritz => [$SID, $BAKPWD, undef]);

You could use a hash instead of an array for your fritz args if you wanted to, and still pull out the pieces you want to assemble the array as desired.  It's very flexible.

But since you're currently using the "form" generator and learning a good deal about it and have determined a solution using the form generator, you might want to copy the form generator into your own fritz generator and then do the differences you need to (as you wished).  Or you might be able to simplify it and just do the content->parts as you alluded to.  TMTOWTDI!  :D


Dan Book
unread,
May 15, 2020, 4:24:12 AM5/15/20
to mojol...@googlegroups.com
The built in "multipart" generator is pretty similar to this. You need to create the parts and set the content type yourself still but it does give you a little flexibility with hashrefs. See the end of the docs for 

https://metacpan.org/pod/Mojo::UserAgent::Transactor#tx

-Dan



-- 

You received this message because you are subscribed to the Google Groups "Mojolicious" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/CACyQ%2BFRSawHXv92ScBL8nc2U_dWVFWwD5KnLqwjCvtKdX8eaPg%40mail.gmail.com.




lack...@googlemail.com
unread,
May 15, 2020, 11:10:02 AM5/15/20
to mojol...@googlegroups.com
Am 14.05.2020 um 05:08 schrieb Stefan Adams:

> I'm certain you'll be able to accomplish this with M::UA. Can you share



Just to reduce complexity I changed everything to plain HTTP, so no SSL,

keys or certificates.



> what the HTML error produced is?



It is not really a normal error page, just a page from the UI. The most

relevant lines are these:

<p class="ErrorMsg">Invalid variable name.</p>

<p>Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.</p>

[German for: repeat the update(sic! I wanted to backup the box, not

update) or restart your FRITZ!Box]



> Can you share the output that you are comparing that you are noticing is so

> similar between curl and M::UA? You're using -v for curl. Also set the

> MOJO_CLIENT_DEBUG env variable to 1.



> See how it works comparing curl to:

> 

>    $ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f

> ImportExportPassword=$BAKPWD -f ConfigExport= http

> ://fritz.box/cgi-bin/firmwarecfg <http://fritz.box/cgi-bin/firmwarecfg>



The empty ConfigExport parameter gets lost with this command and the

Content-Type is different:



-- Blocking request (http://fritz.box/cgi-bin/firmwarecfg)

-- Connect 7c84d2d7396d9d2ff9571982f73240cb (http://fritz.box:80)

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

POST /cgi-bin/firmwarecfg HTTP/1.1\x0d

Content-Length: 48\x0d

User-Agent: Mojolicious (Perl)\x0d

Accept-Encoding: gzip\x0d

Content-Type: application/x-www-form-urlencoded\x0d

Host: fritz.box\x0d

\x0d

ImportExportPassword=backup&sid=4e240384c30c909d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)



-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)

HTTP/1.1 200 OK\x0d







With DEBUG on and my code (s. my original message and below) I get this:



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

POST /cgi-bin/firmwarecfg HTTP/1.1\x0d

Accept-Encoding: gzip, deflate\x0d

Content-Type: multipart/form-data; boundary=tyUPX\x0d

Accept: */*\x0d

Host: fritz.box\x0d

Content-Length: 230\x0d

User-Agent: Mojolicious (Perl)\x0d

\x0d

--tyUPX\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

Content-Disposition: form-data; name="ConfigExport"\x0d

\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

\x0d

--tyUPX\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

Content-Disposition: form-data; name="ImportExportPassword"\x0d

\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

backup

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

\x0d

--tyUPX\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

Content-Disposition: form-data; name="sid"\x0d

\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

33cb272bcd34940b

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

\x0d

--tyUPX--\x0d



-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)



-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)

HTTP/1.1 200 OK\x0d



which for me looks very much the same as the curl version but as I said,

curl gives the backup file whereas M::U just gives the HTML page back,

indicating that something went wrong.

I can only spot very minor differences that is why I am running out of

ideas:

- the boundary used by M::U is quite short

- the form parameters are not in the same order

- Accept-Encoding header (is also sent from Firefox where it works)

- the different SID is normal (my script always makes a fresch login)



Looks very strange to me

-Michael



>> <https://groups.google.com/d/msgid/mojolicious/14df452c-e7a4-4b7c-90a2-9e1f5e15becc%40googlegroups.com?utm_medium=email&utm_source=footer>

>> .

>>

> 



Reply all
Reply to author
Forward
0 new messages