SSL / HTTPS example request

[konfu]

Hi,

can someone please give an example of how to establish a https client
connection with cpp-netlib?
I can't find anything about it in the documentation or the web.

Assuming that Boost.Asio is used internally, another question would
be, if it's possible to provide the io_service to be used by the
client object, but that's not so important.

Nontheless answering both questions would be a great addition to the
docs ;-)

Thanks

[Dean Michael Berris]

On Fri, Sep 9, 2011 at 7:28 AM, konfu <kon...@web.de> wrote:
> Hi,
>
> can someone please give an example of how to establish a https client
> connection with cpp-netlib?
> I can't find anything about it in the documentation or the web.
>

using namespace boost::network::http;
client c;
request req("https://somedomain.com/");
response res = c.get(req);

There's no difference between making an HTTPS or an HTTP request as
far as the interface is concerned. You don't have to worry about that
detail, the library does it for you.

> Assuming that Boost.Asio is used internally, another question would
> be, if it's possible to provide the io_service to be used by the
> client object, but that's not so important.
>

With 0.9.x, yes:

using namespace boost::network::http;
client c(_io_service=my_io_service_instance);

You can find this here:
http://cpp-netlib.github.com/0.9.1/reference_http_client.html#constructors

> Nontheless answering both questions would be a great addition to the
> docs ;-)
>

I wasn't sure that making the URI include whether it's HTTP/HTTPS
wasn't obvious enough.

HTH

--
Dean Michael Berris
Technical Solutions Engineer
Google

[Terry]

you need
#define BOOST_NETWORK_ENABLE_HTTPS
before
#include <boost/network/protocol/http/client.hpp>
and have openssl in your compiler's include paths

[Divye Kapoor]

Can I make a case for the default behaviour to be inclusion of OpenSSL rather than exclusion? It is reasonably expected that clients of an http library will usually require support of both protocols - http and https. Those who don't have OpenSSL will immediately get compiler errors and those who are looking to optimize or omit SSL support can be directed to the pre processing definition to remove SSL support if it suits the application domain. Doing so will ensure that the common case of a user just wanting a quick and dirty C++ way to be http(s) enabled will have as few surprises as possible. Http and https will "just work" as intended out of the box. If there is support for this, I am willing to write a patch for this. (I'm assuming that it will be trivial but I haven't seen the codebase in quite sometime).

Please let me know your thoughts.

-- Divye

> --
> You received this message because you are subscribed to the Google Groups "The C++ Network Library" group.
> To post to this group, send email to cpp-n...@googlegroups.com.
> To unsubscribe from this group, send email to cpp-netlib+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cpp-netlib?hl=en.
>

[Nelson, Erik - 2]

Divye Kapoor wrote on Wednesday, September 14, 2011 1:12 AM

>Can I make a case for the default behaviour to be inclusion of OpenSSL
>rather than exclusion? It is reasonably expected that clients of an http
>library will usually require support of both protocols - http and https.
>Those who don't have OpenSSL will immediately get compiler errors and
>those who are looking to optimize or omit SSL support can be directed
>to the pre processing definition to remove SSL support if it suits the
>application domain. Doing so will ensure that the common case of a user
>just wanting a quick and dirty C++ way to be http(s) enabled will have
>as few surprises as possible. Http and https will "just work" as intended
>out of the box. If there is support for this, I am willing to write a
>patch for this. (I'm assuming that it will be trivial but I haven't
>seen the codebase in quite sometime).
>Please let me know your thoughts.

I strongly disagree. OpenSSL is a huge dependency. OpenSSL itself depends on Perl. What's the benefit in making cpp-netlib dependent on so many third-party packages?

IMO, the simple one-line http GET is what should work out of the box. It requires no dependencies, and that's where just about everybody will start. If that doesn't compile, many people will say "It's too complicated" and walk away.

FWIW, this isn't the first time we've had this idea floated on the list.

Erik

----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited.
Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law.
The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses.

References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link:
http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.

[Dean Michael Berris]

Sorry guys for the slow turn-around on this, I've been attending a
conference and it's only now that I get time to actually sit down and
address some pending issues with cpp-netlib.

Please see in-lined below.

On Wed, Sep 14, 2011 at 8:16 AM, Nelson, Erik - 2
<erik.l...@bankofamerica.com> wrote:
> Divye Kapoor wrote on Wednesday, September 14, 2011 1:12 AM
>
>>Can I make a case for the default behaviour to be inclusion of OpenSSL
>>rather than exclusion? It is reasonably expected that clients of an http
>>library will usually require support of both protocols - http and https.
>>Those who don't have OpenSSL will immediately get compiler errors and
>>those who are looking to optimize or omit SSL support can be directed
>>to the pre processing definition to remove SSL support if it suits the
>>application domain. Doing so will ensure that the common case of a user
>>just wanting a quick and dirty C++ way to be http(s) enabled will have
>>as few surprises as possible. Http and https will "just work" as intended
>>out of the box. If there is support for this, I am willing to write a
>>patch for this. (I'm assuming that it will be trivial but I haven't
>>seen the codebase in quite sometime).
>>Please let me know your thoughts.
>

There's two pieces to the puzzle here.

When cpp-netlib used to be header-only exclusively, making it depend
on OpenSSL by default is bad. The surprise of not being able to at
least compile when OpenSSL is not present is a killer. Lowering the
cost of using the library by not implicitly depending on OpenSSL is
good thing.

Now though that we're moving away from the header-only model, I would
think that at build time of the external libraries, we should be able
to determine whether OpenSSL is present or not. Making cpp-netlib
explicitly depend on OpenSSL on deployment time is another matter (in
case people want to package cpp-netlib for Windows/Linux/Mac). I would
think though that making OpenSSL a *requirement* is going to be
tricky.

For now I would say we should stick to the status quo until we can
find out an acceptable solution while in the transition from
header-only to non-header-only.

> I strongly disagree.  OpenSSL is a huge dependency.  OpenSSL itself depends on Perl.  What's the benefit in making cpp-netlib dependent on so many third-party packages?
>
> IMO, the simple one-line http GET is what should work out of the box.  It requires no dependencies, and that's where just about everybody will start.  If that doesn't compile, many people will say "It's too complicated" and walk away.
>

I agree here. Unfortunately though the reality is becoming clear that
as more and more of the web move to an SSL-protected HTTP
implementation, that would mean the likelihood that they will need
HTTPS support out of the box is something that needs to be recognized.
How we're going to solve the problem of usability and making
deployment easier may be a matter outside of just way the code is
written but also how the libraries are packaged.

> FWIW, this isn't the first time we've had this idea floated on the list.
>

Indeed. That being said though that means we may need to address it
one way or another. At this time I'm not ready to give a definitive
answer yet, but I'd like to try out a few things first and see how far
we can get with different potential approaches (even just
theoretically).

Cheers

[Jorge García Rodríguez]

Hi,

I'm not having any problem in Windows 7 with HTTP requests. 

However, for HTTPS I'm getting always the exception  (curl client or any other works OK):

                http::client::request request("https://api.test.com/rest/someservice");

request << header("Accept", "application/json");

request << header("Connection", "Keep-Alive");

http::client client;

                http::client::response response = client.post(request, "some JSON here", "application/json");

cout << body(response) << endl;

And I get:

     An existing connection was forcibly closed by the remote host

cpp-netlib is compiled with the boost network https enable macro and OpenSSL libs linked properly 

Could you please give me a hand?

Thanks!

[Dean Michael Berris]

This is a known issue, and I have this fixed partially in my branch.
I'll prepare a pull request and I forget if I've already done this for
cpp-netlib/cpp-netlib master. If you try with the latest version in
the master branch and let us know if that fixes the issue it would be
greatly appreciated.

Cheers

[Jorge García Rodríguez]

Hello,

     I got the master branch (cpp-netlib-cpp-netlib-0.9.3-45-geef48fa) and is still happening. Depending on the URL, I get the same exception again (An existing connection was forcibly closed by the remote host) or "short read". 

Thanks for your fast response.

Regards,

Jorge.-

El miércoles 4 de abril de 2012 03:27:35 UTC+2, Dean Michael Berris escribió:

On Tue, Apr 3, 2012 at 6:57 PM, Jorge García Rodríguez

[Dean Michael Berris]

On Wed, Apr 4, 2012 at 8:17 PM, Jorge García Rodríguez
<jorgeg...@gmail.com> wrote:
> Hello,
>
>      I got the master branch (cpp-netlib-cpp-netlib-0.9.3-45-geef48fa) and
> is still happening. Depending on the URL, I get the same exception again (An
> existing connection was forcibly closed by the remote host) or "short
> read".
>

Ouch. Looks like it's not fixed yet in cpp-netlib/cpp-netlib's master
branch. I'll try and fix this back-porting what I did in
deanberris/cpp-netlib's master just so that we can remove this wart
from 0.9.3 -- and hopefully push 0.9.4 out soonish.

> Thanks for your fast response.
>

You're welcome. Thanks for your report. :)

[Dean Michael Berris]

Hi Jorge,

I've just pushed a fix to the master branch for this failure. This
should fix all short-read related failures with HTTPS.

Can you verify that this has been fixed?

[Jorge García Rodríguez]

Hi,

    I've had the opportunity to test it today and I don't get "short_read" anymore, but now requests don't finish:

    For example: simple_wget.exe https://www.google.com used to end in "short read" and now it doesn't end anymore :(. 

   C:\cpp-netlib-build\example\Debug>simple_wget.exe https://www.google.com

   Saving to: index.html ---> and it will stay here forever

   At least something has changed, but there's still a problem. I'm sorry!

   Thanks,

Jorge.-

El jueves, 12 de abril de 2012 03:22:29 UTC+2, Dean Michael Berris escribió:

> On Wed, Apr 4, 2012 at 8:17 PM, Jorge García Rodríguez

[Dean Michael Berris]

On Fri, Apr 13, 2012 at 8:22 PM, Jorge García Rodríguez
<jorgeg...@gmail.com> wrote:
> Hi,
>
>     I've had the opportunity to test it today and I don't get "short_read"
> anymore, but now requests don't finish:
>
>     For example: simple_wget.exe https://www.google.com used to end in
> "short read" and now it doesn't end anymore :(.
>
>    C:\cpp-netlib-build\example\Debug>simple_wget.exe https://www.google.com
>    Saving to: index.html ---> and it will stay here forever
>
>    At least something has changed, but there's still a problem. I'm sorry!
>

This is, unfortunately, well-defined behavior. Here are the reasons:

- The simple_wget example assumes that it's using HTTP/1.0 which has
since 0.9.3 been technically demoted (the default is now HTTP/1.1).
- This won't block "forever" as the Google server will terminate the
connection at some point, but thanks to HTTP/1.1 persistent
connections it doesn't close the connection actively, or within a
certain amount of time.
- If you add `<< header("Connection", "close")` to the request object
before calling `client.get(request)` then the Google HTTP server
*should* terminate the connection from their end once the request has
been responded to.

The bug then resides in the example, and not in the implementation.

Just curiously have you tried running the test suite to confirm
whether you're still seeing the issue on your end?

[Dean Michael Berris]

+cpp-netlib

I've fixed this last remaining test failure. I'll now be tagging 0.9.4
and releasing it in the next few minutes.

Glyn, can you help with the announcement for general availability once
this is done?

Thanks in advance.

Cheers

On Tue, Apr 17, 2012 at 5:44 PM, Jorge García Rodríguez
<jorgeg...@gmail.com> wrote:
> The complete tests suite ends properly and without errors, but the test
> http-client_get:
>
> C:\cpp-netlib-build\tests\Release>cpp-netlib-http-client_get_test.exe
> Running 24 test cases...
>
> It doesn't end (at least in 3 hours that I let it running).
>
> I guess it is affected by the same behavior that you mention. This:
>
>         http::client::request request("https://www.google.es");
>         request << header("Connection", "close");
> http::client client;
>
>         http::client::response response = client.get(request);
>
> cout << body(response) << endl;
>
> ends is "short read".
>
> I'm using cpp-netlib-cpp-netlib-0.9.3-48-g792f0d0 with boost 1.47 with
> OpenSSL 1.0.1, every thing compiled with Visual Studio 2010 (x86) in a
> 64bits Windows 7 machine.
>
> Greetings and thanks!
>
> Jorge.-
>
> El lunes, 16 de abril de 2012 04:04:07 UTC+2, Dean Michael Berris escribió:

[Jorge García Rodríguez]

All tests are working now for me. However, I'm still getting shortread with the example below.

El viernes, 20 de abril de 2012 19:41:34 UTC+2, Dean Michael Berris escribió:

+cpp-netlib

I've fixed this last remaining test failure. I'll now be tagging 0.9.4
and releasing it in the next few minutes.

Glyn, can you help with the announcement for general availability once
this is done?

Thanks in advance.

Cheers

On Tue, Apr 17, 2012 at 5:44 PM, Jorge García Rodríguez

[Dean Michael Berris]

On Thu, Apr 26, 2012 at 2:47 AM, Jorge García Rodríguez
<jorgeg...@gmail.com> wrote:
> All tests are working now for me. However, I'm still getting shortread with
> the example below.
>

Thanks Jorge. Have you re-built the example and the libraries it links
against? Also, please make sure you're adding the "Connection: close"
header when making the requests if you want to get the server to
disconnect.

[Daniel Larimer]

I would like to see a https server example as well.

[Dean Michael Berris]

On Wed, May 16, 2012 at 3:57 PM, Daniel Larimer <dlar...@gmail.com> wrote:
> I would like to see a https server example as well.

Thanks. Unfortunately this is not supported by the library yet. Will
keep everyone posted when I get around to doing development on the
library again.

[Ross Rogers]

Mr. Berris,

Any support for https server yet?  Work in progress?  Directly on the trunk perhaps?

 - Ross

[Dean Michael Berris]

Hi Ross, unfortunately I've not prioritized this for the server
implementation. If you or someone else are willing to take this on, I
would be very happy to review a pull request to make it happen (either
in master, or on 0.11-devel).

> --
> You received this message because you are subscribed to the Google Groups
> "The C++ Network Library" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to cpp-netlib+...@googlegroups.com.

> To post to this group, send email to cpp-n...@googlegroups.com.

> Visit this group at http://groups.google.com/group/cpp-netlib.
> For more options, visit https://groups.google.com/groups/opt_out.