Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
url-retrieve-synchronously results differ from curl
19 views
Skip to first unread message
Artur Malabarba
Jan 19, 2015, 9:28:53 PM1/19/15
to help-gnu-emacs, Sean Allred
There are further details on this phenomenon at this link
http://emacs.stackexchange.com/q/7545/50
In short, we're trying to do a post request with
`url-retrieve-synchronously', but it's yielding a different result than the
same request using curl. The url request *works*, in that it performs the
post and returns sane results. However, these results are not the expected
ones, and performing the same request using curl does return the expected
results.
It's possible this is just a bug in the API we're posting to, but it's more
likely we just haven't built this request correctly in
url-retrieve-synchronously. This would explain why it's being handled
differently from the request we make with curl. We'd appreciate the input
from anyone who knows more about the `url' package, before we run off
blaming the API. :-)
Less relevant details can be found at the linked question, but below is the
basic url call we're using. The address and data-args variables are defined
elsewhere, but I'm just looking to know whether we're doing something wrong
with the url call.
(let ((url-automatic-caching t)
(url-inhibit-uncompression t)
(url-request-data data-args)
(url-request-method "post")
(url-request-extra-headers
'(("Content-Type" . "application/x-www-form-urlencoded"))))
(with-current-buffer
(url-retrieve-synchronously address)
(goto-char (point-min))
(search-forward "\n\n")
(delete-region (point-min) (point))
(buffer-string)))
For comparison, here's the curl command we use (which returns different
results)
(format "curl --silent -X POST --data %S %s"
data-args
address)
Thanks a lot
Artur Malabarba
http://emacs.stackexchange.com/q/7545/50
In short, we're trying to do a post request with
`url-retrieve-synchronously', but it's yielding a different result than the
same request using curl. The url request *works*, in that it performs the
post and returns sane results. However, these results are not the expected
ones, and performing the same request using curl does return the expected
results.
It's possible this is just a bug in the API we're posting to, but it's more
likely we just haven't built this request correctly in
url-retrieve-synchronously. This would explain why it's being handled
differently from the request we make with curl. We'd appreciate the input
from anyone who knows more about the `url' package, before we run off
blaming the API. :-)
Less relevant details can be found at the linked question, but below is the
basic url call we're using. The address and data-args variables are defined
elsewhere, but I'm just looking to know whether we're doing something wrong
with the url call.
(let ((url-automatic-caching t)
(url-inhibit-uncompression t)
(url-request-data data-args)
(url-request-method "post")
(url-request-extra-headers
'(("Content-Type" . "application/x-www-form-urlencoded"))))
(with-current-buffer
(url-retrieve-synchronously address)
(goto-char (point-min))
(search-forward "\n\n")
(delete-region (point-min) (point))
(buffer-string)))
For comparison, here's the curl command we use (which returns different
results)
(format "curl --silent -X POST --data %S %s"
data-args
address)
Thanks a lot
Artur Malabarba
Nicolas Richard
Jan 20, 2015, 2:37:41 AM1/20/15
to Artur Malabarba, Sean Allred, help-gnu-emacs
Artur Malabarba <bruce.c...@gmail.com> writes:
> It's possible this is just a bug in the API we're posting to, but it's
> more likely we just haven't built this request correctly in
> url-retrieve-synchronously. This would explain why it's being handled
> differently from the request we make with curl.
(I'm going a bit off topic here, sorry about that.)
> It's possible this is just a bug in the API we're posting to, but it's
> more likely we just haven't built this request correctly in
> url-retrieve-synchronously. This would explain why it's being handled
> differently from the request we make with curl.
Did you inspect the actual request being sent via e.g. wireshark (this
is GPL software) ? Perhaps you can spot a difference and get at least a
starting point.
If you never used wireshark, here's how I would do it :
1. type something in the filter box (near the top) like:
(ip.src==208.118.235.148 or ip.dst == 208.118.235.148) and http
That would only show HTTP traffic from/to gnu.org.
2. select a capture interface (e.g. eth0) and hit start
3. send the requests
4. hit "Stop" and inspect what you've got.
HTH,
--
Nicolas
Artur Malabarba
Jan 20, 2015, 10:35:43 AM1/20/15
to Nicolas Richard, Sean Allred, help-gnu-emacs
I can't say I fully understand the results.
A single call to `url-retrieve-synchronously' yields 11 entries
meeting the ip.dst filter (most TCP and a few TLS). I tried looking
through these entires to figure out what data/headers were being sent,
but had no success.
Nicolas Richard
Jan 21, 2015, 2:13:15 AM1/21/15
to Artur Malabarba, Sean Allred, help-gnu-emacs
Artur Malabarba <bruce.c...@gmail.com> writes:
> A single call to `url-retrieve-synchronously' yields 11 entries
> meeting the ip.dst filter (most TCP and a few TLS). I tried looking
> through these entires to figure out what data/headers were being sent,
> but had no success.
Does restricting to 'http' help ? This is the kind of thing I have :
> A single call to `url-retrieve-synchronously' yields 11 entries
> meeting the ip.dst filter (most TCP and a few TLS). I tried looking
> through these entires to figure out what data/headers were being sent,
> but had no success.
http://i.imgur.com/cOuz8QR.png
--
Nicolas
Thien-Thi Nguyen
Jan 21, 2015, 2:57:42 AM1/21/15
to help-gnu-emacs
() Artur Malabarba <bruce.c...@gmail.com>
() Tue, 20 Jan 2015 00:28:46 -0200
(url-request-method "post")
IIRC, HTTP specifies the spelling as "POST" (all upcase).
--
Thien-Thi Nguyen
GPG key: 4C807502
(if you're human and you know it)
read my lisp: (responsep (questions 'technical)
(not (via 'mailing-list)))
=> nil
() Tue, 20 Jan 2015 00:28:46 -0200
(url-request-method "post")
IIRC, HTTP specifies the spelling as "POST" (all upcase).
--
Thien-Thi Nguyen
GPG key: 4C807502
(if you're human and you know it)
read my lisp: (responsep (questions 'technical)
(not (via 'mailing-list)))
=> nil
Artur Malabarba
Jan 21, 2015, 8:15:00 AM1/21/15
to Nicolas Richard, Sean Allred, help-gnu-emacs
Retricting to http yields no results (the list becomes empty), so
maybe I'm doing something wrong.
The api call is indeed https, with the domain `api.stackexchange.com',
but none of the entries in the list use the http protocol. I thought I
might be using a wrong destination IP in the filter (which I
discovered by pinging the domain above), but even if I remove the
(ip.dst == 198.252.206.140) part I still get an empty list for the
`http' filter (also for the `http2').
I also tried changing the interface to `any' (in case I was using the
wrong one), but the result is the same. I get some http entries listed
if I do some browsing, but I get nothing by contacting the API with
url-retrieve-synchronously.
maybe I'm doing something wrong.
The api call is indeed https, with the domain `api.stackexchange.com',
but none of the entries in the list use the http protocol. I thought I
might be using a wrong destination IP in the filter (which I
discovered by pinging the domain above), but even if I remove the
(ip.dst == 198.252.206.140) part I still get an empty list for the
`http' filter (also for the `http2').
I also tried changing the interface to `any' (in case I was using the
wrong one), but the result is the same. I get some http entries listed
if I do some browsing, but I get nothing by contacting the API with
url-retrieve-synchronously.
Nicolas Richard
Jan 21, 2015, 8:42:30 AM1/21/15
to bruce.c...@gmail.com, Sean Allred, help-gnu-emacs
Le 21/01/2015 14:14, Artur Malabarba a écrit :
> The api call is indeed https, with the domain `api.stackexchange.com',
> but none of the entries in the list use the http protocol. I thought I
> might be using a wrong destination IP in the filter (which I
> discovered by pinging the domain above), but even if I remove the
> (ip.dst == 198.252.206.140) part I still get an empty list for the
> `http' filter (also for the `http2').
I had overlooked the fact you were using https, sorry about this. I
> The api call is indeed https, with the domain `api.stackexchange.com',
> but none of the entries in the list use the http protocol. I thought I
> might be using a wrong destination IP in the filter (which I
> discovered by pinging the domain above), but even if I remove the
> (ip.dst == 198.252.206.140) part I still get an empty list for the
> `http' filter (also for the `http2').
confirm that when I try with an https url, the output seems unusable.
Nicolas.
Artur Malabarba
Jan 21, 2015, 9:15:01 AM1/21/15
to Thien-Thi Nguyen, help-gnu-emacs
>
> (url-request-method "post")
>
> IIRC, HTTP specifies the spelling as "POST" (all upcase).
>
Oh god. This actually works! It fixes the bug.
> (url-request-method "post")
>
> IIRC, HTTP specifies the spelling as "POST" (all upcase).
>
Why the hell would it work 50% with lower case method?! I mean, the posting
succeeds anyway, only the return value is kind of off.
I'll do some more testing later, but this does seem to be the culprit.
Thanks a lot, Thien.
Sean Allred
Jan 21, 2015, 10:13:29 AM1/21/15
to Nicolas Richard, help-gnu-emacs, Artur Malabarba
Artur discovered something this morning that I wanted to share here as he works on a proper bug report. (We’re both especially busy today, unfortunately.) Perhaps someone can shed light on this behavior.
Binding `url-request-method` to `”POST”` rather than `”post”` seems to fix the issues en masse. This has not undergone proper testing, but it definitely seems to work [1]. I apologize that I don’t have an example that works out-of-the-box – I really have a very limited experience with web stuff – but using “POST” rather than “post” makes a significant difference.
The question we both have at this point is *why* – *why* does “post” ‘partly’ work? *Why* does “POST” work fully? And frankly, in my personal opinion, *why* aren’t these methods taken in as symbols in the first place?
Either Artur or I will be filing a proper bug report for url.el on this issue as time allows (assuming of course that someone doesn’t beat both of us to it).
(Unless this thread can count as a bug-report?)
All the best,
Sean Allred
[1]: https://gist.github.com/90be70f12cd097be1247 <https://gist.github.com/90be70f12cd097be1247>, reproduced below
(defconst tmp:access-token
;; Needed to post answers. Considered a secret. If you would like
;; a key, use `sx-authenticate' from sx.el (available from MELPA)
;; and look in ~/.emacs.d/.sx/auth.el
"YOUR ACCESS TOKEN HERE")
(defconst tmp:key
;; not considered a secret
"0TE6s1tveCpP9K5r5JNDNQ((")
(defun tmp:api-bug (use-curl access-token key)
"Post a test answer to the formatting sandbox."
(let ((random-body-1 (md5 (current-time-string)))
(random-body-2 (md5 (md5 (current-time-string))))
(method "https://api.stackexchange.com/2.2/questions/3122/answers/add")
(args
(mapconcat
#'identity
`(,(format "access_token=%s"
(replace-regexp-in-string
"%" "%%" (url-hexify-string access-token)))
,(format "key=%s"
(replace-regexp-in-string
"%" "%%" (url-hexify-string key)))
"site=meta"
"pagesize=100"
"filter=%%21GoYr1we0U5inG5G7wBg4JBGpbgX%%29C7LDqpy-%%2AbfwPOujOr4SR4W%%29bLNSyYUpQDdTwTj.XChTFB0gfLaAJq0hv"
"body=this-is-an-answer-test-for-sx.el--%s")
"&")))
(if use-curl
(shell-command-to-string
(format
"curl --silent -X POST --data %S %s | gunzip"
(format args random-body-2)
method))
;; emacs-devel: note vvvv
(url-request-method "POST")
(require 'json)
(json-read-from-string (tmp:api-bug t tmp:access-token tmp:key))
((total . 1) ; this data structure used curl
(page_size . 100)
(page . 1)
(quota_remaining . 9979)
(quota_max . 10000)
(has_more . :json-false)
(items .
[((link . "http://meta.stackexchange.com/questions/3122//247295#247295")
(body_markdown . "this-is-an-answer-test-for-sx.el--e3eeb6228ed9c2c58e5385b73493f0f0")
(share_link . "http://meta.stackexchange.com/a/247295/188148")
(answer_id . 247295)
(creation_date . 1421684370)
(last_activity_date . 1421684370)
(score . 0)
(upvoted . :json-false)
(downvoted . :json-false)
(owner
(display_name . "Sean Allred")
(reputation . 160)))]))
(json-read-from-string (tmp:api-bug nil tmp:access-token tmp:key))
((total . 1) ; this data structure used "POST"
(page_size . 100)
(page . 1)
(quota_remaining . 9999)
(quota_max . 10000)
(has_more . :json-false)
(items .
[((link . "http://meta.stackexchange.com/questions/3122//247396#247396")
(body_markdown . "this-is-an-answer-test-for-sx.el--5b447b87e7078ed0fd34a3169ee84319")
(share_link . "http://meta.stackexchange.com/a/247396/188148")
(answer_id . 247396)
(creation_date . 1421847252)
(last_activity_date . 1421847252)
(score . 0)
(upvoted . :json-false)
(downvoted . :json-false)
(owner
(display_name . "Sean Allred")
(reputation . 160)))]))
((total . 1) ; this data structure used "post"
(page_size . 100)
(page . 1)
(quota_remaining . 9977)
(quota_max . 10000)
(has_more . :json-false)
(items .
[((link . "http://meta.stackexchange.com/questions/3122//247297#247297")
(share_link . "http://meta.stackexchange.com/a/247297/188148")
(answer_id . 247297)
(creation_date . 1421684555)
(last_activity_date . 1421684555)
(score . 0)
(owner
(display_name . "Sean Allred")
(reputation . 160)))]))
Binding `url-request-method` to `”POST”` rather than `”post”` seems to fix the issues en masse. This has not undergone proper testing, but it definitely seems to work [1]. I apologize that I don’t have an example that works out-of-the-box – I really have a very limited experience with web stuff – but using “POST” rather than “post” makes a significant difference.
The question we both have at this point is *why* – *why* does “post” ‘partly’ work? *Why* does “POST” work fully? And frankly, in my personal opinion, *why* aren’t these methods taken in as symbols in the first place?
Either Artur or I will be filing a proper bug report for url.el on this issue as time allows (assuming of course that someone doesn’t beat both of us to it).
(Unless this thread can count as a bug-report?)
All the best,
Sean Allred
[1]: https://gist.github.com/90be70f12cd097be1247 <https://gist.github.com/90be70f12cd097be1247>, reproduced below
(defconst tmp:access-token
;; Needed to post answers. Considered a secret. If you would like
;; a key, use `sx-authenticate' from sx.el (available from MELPA)
;; and look in ~/.emacs.d/.sx/auth.el
"YOUR ACCESS TOKEN HERE")
(defconst tmp:key
;; not considered a secret
"0TE6s1tveCpP9K5r5JNDNQ((")
(defun tmp:api-bug (use-curl access-token key)
"Post a test answer to the formatting sandbox."
(let ((random-body-1 (md5 (current-time-string)))
(random-body-2 (md5 (md5 (current-time-string))))
(method "https://api.stackexchange.com/2.2/questions/3122/answers/add")
(args
(mapconcat
#'identity
`(,(format "access_token=%s"
(replace-regexp-in-string
"%" "%%" (url-hexify-string access-token)))
,(format "key=%s"
(replace-regexp-in-string
"%" "%%" (url-hexify-string key)))
"site=meta"
"pagesize=100"
"filter=%%21GoYr1we0U5inG5G7wBg4JBGpbgX%%29C7LDqpy-%%2AbfwPOujOr4SR4W%%29bLNSyYUpQDdTwTj.XChTFB0gfLaAJq0hv"
"body=this-is-an-answer-test-for-sx.el--%s")
"&")))
(if use-curl
(shell-command-to-string
(format
"curl --silent -X POST --data %S %s | gunzip"
(format args random-body-2)
method))
(let ((url-automatic-caching t)
(url-inhibit-uncompression t)
(url-request-data (format args random-body-1))
(url-inhibit-uncompression t)
;; emacs-devel: note vvvv
(url-request-method "POST")
(url-request-extra-headers
'(("Content-Type" . "application/x-www-form-urlencoded"))))
(with-current-buffer
(url-retrieve-synchronously method)
'(("Content-Type" . "application/x-www-form-urlencoded"))))
(with-current-buffer
(goto-char (point-min))
(search-forward "\n\n")
(delete-region (point-min) (point))
(buffer-string))))))
(search-forward "\n\n")
(delete-region (point-min) (point))
(require 'json)
(json-read-from-string (tmp:api-bug t tmp:access-token tmp:key))
((total . 1) ; this data structure used curl
(page_size . 100)
(page . 1)
(quota_remaining . 9979)
(quota_max . 10000)
(has_more . :json-false)
(items .
[((link . "http://meta.stackexchange.com/questions/3122//247295#247295")
(body_markdown . "this-is-an-answer-test-for-sx.el--e3eeb6228ed9c2c58e5385b73493f0f0")
(share_link . "http://meta.stackexchange.com/a/247295/188148")
(answer_id . 247295)
(creation_date . 1421684370)
(last_activity_date . 1421684370)
(score . 0)
(upvoted . :json-false)
(downvoted . :json-false)
(owner
(display_name . "Sean Allred")
(reputation . 160)))]))
(json-read-from-string (tmp:api-bug nil tmp:access-token tmp:key))
((total . 1) ; this data structure used "POST"
(page_size . 100)
(page . 1)
(quota_remaining . 9999)
(quota_max . 10000)
(has_more . :json-false)
(items .
[((link . "http://meta.stackexchange.com/questions/3122//247396#247396")
(body_markdown . "this-is-an-answer-test-for-sx.el--5b447b87e7078ed0fd34a3169ee84319")
(share_link . "http://meta.stackexchange.com/a/247396/188148")
(answer_id . 247396)
(creation_date . 1421847252)
(last_activity_date . 1421847252)
(score . 0)
(upvoted . :json-false)
(downvoted . :json-false)
(owner
(display_name . "Sean Allred")
(reputation . 160)))]))
((total . 1) ; this data structure used "post"
(page_size . 100)
(page . 1)
(quota_remaining . 9977)
(quota_max . 10000)
(has_more . :json-false)
(items .
[((link . "http://meta.stackexchange.com/questions/3122//247297#247297")
(share_link . "http://meta.stackexchange.com/a/247297/188148")
(answer_id . 247297)
(creation_date . 1421684555)
(last_activity_date . 1421684555)
(score . 0)
(owner
(display_name . "Sean Allred")
(reputation . 160)))]))
Thien-Thi Nguyen
Jan 21, 2015, 11:30:53 AM1/21/15
to help-gnu-emacs
() Sean Allred <co...@seanallred.com>
() Wed, 21 Jan 2015 09:27:16 -0500
The question we both have at this point is *why* – *why* does
“post” ‘partly’ work? *Why* does “POST” work fully?
That can only be answered precisely by the programmers of the
server software. They were probably smitten by Postel, blech,
and spewful of one-armed-if expressions and other such half-
sense. [Insert more curmudgeonly harrumphing here. :-D]
() Wed, 21 Jan 2015 09:27:16 -0500
The question we both have at this point is *why* – *why* does
“post” ‘partly’ work? *Why* does “POST” work fully?
server software. They were probably smitten by Postel, blech,
and spewful of one-armed-if expressions and other such half-
sense. [Insert more curmudgeonly harrumphing here. :-D]
Artur Malabarba
Jan 21, 2015, 8:15:02 PM1/21/15
to Sean Allred, Nicolas Richard, help-gnu-emacs
> Binding url-request-method to ”POST” rather than ”post” seems to fix
> the issues en masse. This has not undergone proper testing, but it
> definitely seems to work [1].
I'd just like to confirm (now that I've done a bit more testing) that
> the issues en masse. This has not undergone proper testing, but it
> definitely seems to work [1].
this does indeed fix the issue. So thanks again to Thien-Thi for
suggesting that.
I'll see if I can find out where this happens in url.el. If not, I'll
just file a bug report. IMHO, the package should either support both
versions indiscriminately, or warn the user if they use the wrong
version. Even better (as Sean suggests) would be to support and
encourage the use of symbols.
Artur Malabarba
Jan 21, 2015, 9:41:20 PM1/21/15
to Sean Allred, help-gnu-emacs
Upon further (much much further) investigation, it seems that nothing
in the url package could cause this specific behavior. Still, I see
many places where the package will silently bug if the user downcases
the "get" method (and a couple other methods). Given that
`url-request-method' makes no mention of that, I think the following
would be worthy patches (and I'm willing to write them if people
agree):
1. Document in `url-request-method' that method names are uppercase
strings (the current version doesn't even say they are strings).
2. Warn the user (with a message?) in `url-retrieve-internal' if the
above variable is dynamically bound to a lowercase string.
in the url package could cause this specific behavior. Still, I see
many places where the package will silently bug if the user downcases
the "get" method (and a couple other methods). Given that
`url-request-method' makes no mention of that, I think the following
would be worthy patches (and I'm willing to write them if people
agree):
1. Document in `url-request-method' that method names are uppercase
strings (the current version doesn't even say they are strings).
2. Warn the user (with a message?) in `url-retrieve-internal' if the
above variable is dynamically bound to a lowercase string.
Thien-Thi Nguyen
Jan 22, 2015, 3:56:57 AM1/22/15
to help-gnu-emacs
() Artur Malabarba <bruce.c...@gmail.com>
() Wed, 21 Jan 2015 23:14:56 -0200
I'll see if I can find out where this happens in url.el.
If not, I'll just file a bug report. IMHO, the package
should either support both versions indiscriminately, or
warn the user if they use the wrong version.
This is not a legitimate complaint. HTTP specifies the
all-upcase spelling explicitly, so url.el DTRT already.
(Nod to pedants: Actually, RFC1945 sez "The method is
case-sensitive." and lists "POST" in the RHS in the
‘Method’ non-terminal production...)
If anything, a bug-report is indicated for the server side:
unrecognized method names should ellicit status code 501
(error; "not implemented").
Even better (as Sean suggests) would be to support and
encourage the use of symbols.
This is a separate issue, but legitimate. Personally, i'd love
to see more symbols and fewer strings-as-symbols, all around.
(Insert rant on languages lacking a "symbol" data type, here.)
() Wed, 21 Jan 2015 23:14:56 -0200
I'll see if I can find out where this happens in url.el.
If not, I'll just file a bug report. IMHO, the package
should either support both versions indiscriminately, or
warn the user if they use the wrong version.
all-upcase spelling explicitly, so url.el DTRT already.
(Nod to pedants: Actually, RFC1945 sez "The method is
case-sensitive." and lists "POST" in the RHS in the
‘Method’ non-terminal production...)
If anything, a bug-report is indicated for the server side:
unrecognized method names should ellicit status code 501
(error; "not implemented").
Even better (as Sean suggests) would be to support and
encourage the use of symbols.
to see more symbols and fewer strings-as-symbols, all around.
(Insert rant on languages lacking a "symbol" data type, here.)
0 new messages