3p-curl autobuild on linux

[Nicky Perian]

I have been trying to trackdown what I think is dependency problem building 3p-curl.

Dependencies seem to be in order.

openssl is dependent on zlib and openssl build fine.

curl is dependent on openssl

The link for curl fails with:

[ 16%] Linking C executable curl
/home/nicky/.linuxbrew/bin/ld: /home/nicky/kokua/libs/3p-curl/stage/packages/lib/release/libcrypto.a(dso_dlfcn.o): in function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x11): undefined reference to `dlopen'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x24): undefined reference to `dlsym'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x2f): undefined reference to `dlclose'
/home/nicky/.linuxbrew/bin/ld: /home/nicky/kokua/libs/3p-curl/stage/packages/lib/release/libcrypto.a(dso_dlfcn.o): in function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x344): undefined reference to `dlsym'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x402): undefined reference to `dlerror'
/home/nicky/.linuxbrew/bin/ld: /home/nicky/kokua/libs/3p-curl/stage/packages/lib/release/libcrypto.a(dso_dlfcn.o): in function `dlfcn_bind_var':
dso_dlfcn.c:(.text+0x474): undefined reference to `dlsym'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x532): undefined reference to `dlerror'
/home/nicky/.linuxbrew/bin/ld: /home/nicky/kokua/libs/3p-curl/stage/packages/lib/release/libcrypto.a(dso_dlfcn.o): in function `dlfcn_load':
dso_dlfcn.c:(.text+0x599): undefined reference to `dlopen'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x5fd): undefined reference to `dlclose'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x635): undefined reference to `dlerror'
/home/nicky/.linuxbrew/bin/ld: /home/nicky/kokua/libs/3p-curl/stage/packages/lib/release/libcrypto.a(dso_dlfcn.o): in function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x6c1): undefined reference to `dladdr'
/home/nicky/.linuxbrew/bin/ld: dso_dlfcn.c:(.text+0x721): undefined reference to `dlerror'
/home/nicky/.linuxbrew/bin/ld: /home/nicky/kokua/libs/3p-curl/stage/packages/lib/release/libcrypto.a(dso_dlfcn.o): in function `dlfcn_unload':
dso_dlfcn.c:(.text+0x782): undefined reference to `dlclose'
collect2: error: ld returned 1 exit status

I have searched the net and found others having the similar issues that are recommending  the placement of linker switch "-ldl".

 If anyone else has encountered and solved; Please let me know.

Thanks in advance,

Nicky 

[slacker]

https://bitbucket.org/Drakeo/p64_3p-curl/commits/

may want  to look what I did all in gcc 5.5.0

shuld be fine in your gcc 5.4.0

do not thing this is a zlib issue.

but that said this is a 7 year old open ssl issue

[Henri Beauchamp]

On Tue, 27 Apr 2021 15:18:28 -0500, Nicky Perian wrote:

> I have been trying to trackdown what I think is dependency problem building
> 3p-curl.
> Dependencies seem to be in order.

> .../...

> The link for curl fails with:

> .../...

> dso_dlfcn.c:(.text+0x11): undefined reference to `dlopen'

Are you mixing static and shared libraries (e.g. compiling a shared
libcurl against a static libopenssl) ?... That could explain it, if
inappropriate options are passed to cmake.

You might want to take a look at the build scripts I use in:
http://sldev.free.fr/libraries/sources/cvlv-curl-7.47-20201001.tar.bz2
or:
http://sldev.free.fr/libraries/sources/cvlv-curl-7.54.1-20201210.tar.bz2

Henri.

[Nicky Perian]

Henri,

Thank You.

curl-7.47 builds perfectly from your script.

curl-7.54-1 has the same issue that I see when using autobuild.

I'll rollback to 7.47 unless there is a a known solution you have encountered with 7-54-1.

 'Nicky

[Henri Beauchamp]

On Wed, 28 Apr 2021 10:01:48 -0700 (PDT), Nicky Perian wrote:

> curl-7.47 builds perfectly from your script.
> curl-7.54-1 has the same issue that I see when using autobuild.
>
> I'll rollback to 7.47 unless there is a a known solution you have
> encountered with 7-54-1.

Strange, v7.54.1 builds just fine for me...

As I see it, it could be an issue with how you built nghttp2, since
it's the only difference in linked libraries between the two versions;
do check that nghttp2 is built statically (see my build script in:
http://sldev.free.fr/libraries/sources/cvlv-nghttp2-20200409.tar.bz2).

Henri.

---------------------------------------------
(*) but I'm still using v7.47 for official viewer builds because
of HTTP v1.1 (i.e. pipelining) bugs in newer versions, which can be
a bother in OpenSIM grids since very few of them use HTTP v2...

[Nicky Perian]

Henri,

It looks like the issue is libcrypto.a. Can I use your build script for openssl? I may be getting a different version as a result of system installation from other packages.

Thanks,

Nicky

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.
To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20210428193122.d89c7ece3a09c2f5cda71079%40free.fr.

[Henri Beauchamp]

On Thu, 29 Apr 2021 11:25:41 -0500, Nicky Perian wrote:

> Henri,
> It looks like the issue is libcrypto.a. Can I use your build script for
> openssl?

Sure, it's all Open Source. :-)

Henri.

[Alex]

No issues for me with a recent curl and openssl on linux (with
firestorm):

libcurl Version: libcurl/7.74.0 OpenSSL/1.1.1i zlib/1.2.8 nghttp2/1.42.0

Why not update to a current version of curl?

[Henri Beauchamp]

On Thu, 29 Apr 2021 10:36:45 +1000, 'Alex' via opensource-dev wrote:

> No issues for me with a recent curl and openssl on linux (with
> firestorm):
>
> libcurl Version: libcurl/7.74.0 OpenSSL/1.1.1i zlib/1.2.8 nghttp2/1.42.0
>
> Why not update to a current version of curl?

All versions after the LL-patched v7.47 have an HTTP pipelining bug
(pipeline stalls, resulting in "rainbow" textures), including v7.54.1
in use by LL for their own viewer...
Probably not much of a problem any more with SL, since I would assume
all their CDN servers can do HTTP 2 now... But OpenSim grids are
another story.

I also tried a newer version in the past (v7.60-ish), but it lamentably
failed. There has very recently been an attempt by LL at using v7.76.1
( https://bitbucket.org/lindenlab/viewer/commits/45b5ac72b2384ea370e4138e08c4c8344d94b4e3 )
but it got promptly reverted (Revert "SL-15141 Update CURL to 7.76.1"
due to OPS-4251, which we, poor TPV developers, cannot know what it is
all about, because we do not have access to any of those issues on the
JIRA), in:
https://bitbucket.org/lindenlab/viewer/commits/2de92430efa91fec0fddcec9be0755a69ca4f7b5

Henri.

[Alex]

I only had to make one change to make recent curl versions work (in
addition to the switch statement issue thats caused by changes to the
libcurl header file); that is, drop the 'CURLOPT_ENCODING' param to curl
init. It doesnt work because the LL CDN doesnt support compressed
content - why the change is suddenly needed in later versions of curl, i
dont know, but you will see the same error that the viewer encounters
with the newer curl if you call curl from the command line and add the
--compressed param and give it a URL for a texture for example.


Unless you do that, you will log into a grey world that looks pretty
horrible :)

A.

[Alex]

Strange, I havent seen any sort of issue like that with recent versions
of curl. Seems to be rock solid for me. Never seen a corrupted texture
etc.

On 2021-05-01 07:19, Henri Beauchamp wrote:

[Henri Beauchamp]

On Sat, 01 May 2021 14:08:20 +1000, 'Alex' via opensource-dev wrote:

> I only had to make one change to make recent curl versions work (in
> addition to the switch statement issue thats caused by changes to the
> libcurl header file); that is, drop the 'CURLOPT_ENCODING' param to curl
> init. It doesnt work because the LL CDN doesnt support compressed
> content - why the change is suddenly needed in later versions of curl, i
> dont know, but you will see the same error that the viewer encounters
> with the newer curl if you call curl from the command line and add the
> --compressed param and give it a URL for a texture for example.
>
>
> Unless you do that, you will log into a grey world that looks pretty
> horrible :)

Ah, yes, I seem to remember about warnings with bad llsd+gz encoding
(or such) in the log when I tested curl v7.60-something...
Did not bother to investigate back then, since I had more urgent matters
to deal with (and all modern curl versions got HTTP 1.1 screwed up
anyway: curl devels gave up on fixing it years ago, sadly).

I'll give it another try, thanks ! :-)

On Sat, 01 May 2021 14:14:15 +1000, 'Alex' via opensource-dev wrote:

> Strange, I havent seen any sort of issue like that with recent versions
> of curl. Seems to be rock solid for me. Never seen a corrupted texture
> etc.

Well, you do need a fast viewer (configured with aggressive download
parameters) and high bandwidth network (1Gpbs downlink on optic fiber,
here), else the pipeline trashing might not get triggered...

Henri.

[Monty Brandenberg]

On 4/30/2021 5:19 PM, Henri Beauchamp wrote:

> ... but it got promptly reverted (Revert "SL-15141 Update CURL to 7.76.1"

> due to OPS-4251, which we, poor TPV developers, cannot know what it is
> all about, because we do not have access to any of those issues on the
> JIRA),

Well, OPS is necessarily internal but this one I think I can safely say
is about http/2 infrastructure for the CDN. Probably deferred for
Uplift but ready to be picked up when circumstances allow.

m

[Henri Beauchamp]

On Mon, 3 May 2021 08:49:40 -0400, Monty Brandenberg wrote:

> Well, OPS is necessarily internal but this one I think I can safely say
> is about http/2 infrastructure for the CDN. Probably deferred for
> Uplift but ready to be picked up when circumstances allow.

Thank you ! :-)

And this is interesting but might deserve more explanations since curl
v7.54.1 (in use by LL's viewer and most TPVs) already had HTTP/2 support.
So, is it that the CDN infrastructure is not quite ready for HTTP/2, or
is it something changed in curl v7.76 that causes issues with it ?

I had a try (under Linux) at curl v7.76.1, nghttp2 v1.43 and OpenSSL
v1.1.1k, and they seem to work fine now, in SL, but if you tell me you'd
prefer not to see TPVs using curl v7.76.1 for now, it is time to speak
up (I was planing enabling it for next Saturday's release of my viewer,
but it's just a matter of toggling a define to keep using the good old
v7.47 instead)...

Regards,

Henri.

[Alex]

I dont see why using a newer curl would be detrimental to the CDN. I
dont think the viewer even uses HTTP/2 in its curl init anyway. To use
HTTP/2 in curl you need to set CURLOPT_HTTP_VERSION to
CURL_HTTP_VERSION_2 other wise its plain old HTTP 1.1 by default.

To be honest I cant see HTTP/2 helping much with improving the
performance of asset downloads in the viewer.

Dear LL - Please enable gzip compression support on your CDN.

[Henri Beauchamp]

On Tue, 04 May 2021 20:44:18 +1000, 'Alex' via opensource-dev wrote:

> I dont see why using a newer curl would be detrimental to the CDN. I
> dont think the viewer even uses HTTP/2 in its curl init anyway. To use
> HTTP/2 in curl you need to set CURLOPT_HTTP_VERSION to
> CURL_HTTP_VERSION_2 other wise its plain old HTTP 1.1 by default.

There's test code in LL's viewer in indra/llcorehttp/_httpoprequest.cpp:

// Also try requesting HTTP/2.
/******************************/
// but for test purposes, only if overriding VIEWERASSET
if (getenv("VIEWERASSET"))
/******************************/
check_curl_easy_setopt(mCurlHandle, CURLOPT_HTTP_VERSION,
CURL_HTTP_VERSION_2_0);

I.e. you need to set the VIEWERASSET environment variable to make it
(try and) work...

I just made a test here: curl v7.76.1 does not seem to protest when
HTTP/2 is enabled, and things seem to work fine (but it would need
a much longer and thorough test to be affirmative); however, I cannot
confirm whether HTTP/2 is actually used or not...

> To be honest I cant see HTTP/2 helping much with improving the
> performance of asset downloads in the viewer.

It should perform as least as good as HTTP/1.1, and given the curl
bugs for the latter, it could result in less failed downloads in the
end...

> Dear LL - Please enable gzip compression support on your CDN.

Not sure this is actionable by LL... Plus, most (and the largest)
assets downloaded via the CDNs are textures (already compressed
in the best possible way since in JPEG2000 format) and meshes (that
are (un)compressed in gzip format on the viewer side)... So the
(de)compression at the HTTP server level (and then at libcurl level
viewer-side) might actually cost more in processing power than it
would save in download time and size (the rest of the assets are
pretty small in size anyway).

Henri.

[Monty Brandenberg]

On 5/4/2021 6:44 AM, 'Alex' via opensource-dev wrote:

> To be honest I cant see HTTP/2 helping much with improving the
> performance of asset downloads in the viewer.

Well, HTTP/2 will be a significant change in the environment and
things will change. Performance could be negatively impacted due
to the implied encryption (https:). However, that same encryption
would also have prevented cheap, buggy http caches (e.g. mobile
vendors) from making a mess of the HTTP/1.1 pipelining flows
(an experiment we haven't had a chance to try). Future loss of
correct caching between endpoints might be another negative (for
those who insisted on it and set up their own caching schemes).

I haven't done any testing (been working server-side for awhile)
but I don't expect HTTP/2 multiplexing to match the potential
of HTTP/1.1 pipelining for throughput. But with asset source
ping times lowered due to CDN use, the gap should be lessened.
And I expect vendors to better support multiplexing as it is
core to HTTP/2 and not an 'exotic' mode.

m

[Monty Brandenberg]

On 5/4/2021 4:09 AM, Henri Beauchamp wrote:

> And this is interesting but might deserve more explanations since curl
> v7.54.1 (in use by LL's viewer and most TPVs) already had HTTP/2 support.
> So, is it that the CDN infrastructure is not quite ready for HTTP/2, or
> is it something changed in curl v7.76 that causes issues with it ?

Some infrastructure work needs to happen at a minimum. I'm
pretty certain no one has looked at 7.76 at Linden yet but I do
follow the project. Pipelining should be disabled as shipped
by curl.se and the DNS resolution needs to be checked as defaults
may have been changed. And disable IPv6, that will just waste
time right now. That's about all I know right now.

m

[Alex]

I think it is actionable from cloudfront - and no gzip compression
support does cause some performance reduction. The CDN isnt just used
for serving jpeg and png images, the TPV widgets provided by LL load
other types of content off the CDN. Running
https://phoenixviewer.com/app/loginV3/ through various tests (GT Metrix,
webpagetest.org) scores an 'F' and its not content from the FS servers
that is the issue, its LL's CDN. I have a Jira open for this but I doubt
its high on LL's priority list:

https://jira.secondlife.com/browse/BUG-228229?filter=19350 (see my later
comments and screenshots attached)

Maybe Monty can comment on this.

[Monty Brandenberg]

On 5/4/2021 11:48 PM, 'Alex' via opensource-dev wrote:

> https://jira.secondlife.com/browse/BUG-228229?filter=19350 (see my later
> comments and screenshots attached)
>
> Maybe Monty can comment on this.

'F' doesn't mean 'Fabulous'?

I'm torn between optimizing content processing and transport
for performance and filling every web app with MBs of useless
javascript so the universe dies from heat death. But, yeah,
some aspects of this should be a normal part of 'the service.'
I just don't have a when/who for you.

m

[Alex]

Well, at least you didnt do a classic Oz on me and say "soon(tm)"

laughs :)

[Henri Beauchamp]

On Wed, 05 May 2021 13:48:00 +1000, 'Alex' via opensource-dev wrote:

> I think it is actionable from cloudfront

Sure, but is Cloudfront actionable by LL for this setting ?... :-P

> and no gzip compression support does cause some performance reduction.

It does, when you try to compress something that is already compressed
much better than gzip could achieve: try and compress a large set of
JPEG2000 images (rezzing with 512m DD in main land can cause 30000+
images to get downloaded) and you will see the time it takes and the
final total size of the compressed assets (larger than original).
Same when you try and compress an already gzipped contents (mesh
assets, for example).
It is pure waste of processing power both on server and client side !

> The CDN isnt just used for serving jpeg and png images,

There is no PNG *asset* served by the CDN. All images in SL are served
in J2C format. But even PNG is compressed and cannot be compressed
better by gzip.

> the TPV widgets provided by LL load other types of content off the
> CDN.

What contents ?... A few LL-site web pages ?... Who cares ?
They do not intervene in rezzing speed (and frame rates during rezzing:
if you must decompress assets that got uselessly compressed, you waste
power that could be used to render things or fetch other assets faster),
which is all I care about, personally. :-P

> Running https://phoenixviewer.com/app/loginV3/ through various tests
> (GT Metrix, webpagetest.org) scores an 'F' and its not content from
> the FS servers that is the issue, its LL's CDN. I have a Jira open for
> this but I doubt its high on LL's priority list:

If it were just about me, Javascript would be banned from browsers ! :-D

But frankly, the slow rendering of web pages is largely the fault of CEF
itself (even if gzipped pages transfers would help a little).
This (totally bloated and bug-ridden) thing is *much* slower than the
Webkit engine it is based upon; see this message of mine in this list
archive:
https://list-archives.secondlife.com/opensource-dev/2015-July/010106.html
The speed comparison is in the last paragraphs (like 3 to 5 times
slower to render a web page in my benchmarks, and this was years before
CDN migration of LL's web sites).

Henri.

[Alex]

Wow, a lot of negativity and criticism henry.

I have reasons for my concerns. Do you know who I am?

https://www.firestormviewer.org/developers/ (scroll down to
infrastructure).

[Alex]

Sorry Henri, but I disagree with a number of things you have said. I
have concerns of my own of course, and for good reason... If you want to
know who I really am:

https://www.firestormviewer.org/developers/ (scroll down to

infrastructure since it shows my role)

In case theres is any doubt as to who I am, Jess will confirm. I can
also be reached on mygoditsf...@firestormviewer.org or
mygoditsf...@phoenixviewer.com

On 2021-05-05 18:31, Henri Beauchamp wrote:

[Henri Beauchamp]

On Thu, 06 May 2021 22:49:20 +1000, 'Alex' via opensource-dev wrote:

> Wow, a lot of negativity and criticism henry.

Just being factual.

> I have reasons for my concerns. Do you know who I am?

No. And it does not matter.

Arguments are not a problem with the person defending them, as far as
I am concerned, but with their validity or lack thereof.

Pure logic and verified facts are the *only* things I go by. Sadly,
many people tend to take things personally when they should not. :-/

Henri.

[slacker]

Sorry for the late reply that is a GLIBC issue

everyone of those.

sorry did not have any of them issues.

Are your uning multiple glibc  with multiple  gcc such as

the ubuntu and debian love to do.

I am glade you got it linked.

this was a GLIBC_2.25  for you because your building in the 2.24 or 2.23 with  gcc 5.4.0

make sure your alternates are correct

On Tuesday, April 27, 2021 at 3:18:40 PM UTC-5 nicky...@gmail.com wrote:

[choraz...@gmail.com]

I got involved in tracking this one down – in fact it was a build configuration issue for openssl

 

Discussion: https://github.com/curl/curl/issues/2199

 

The fix: https://github.com/curl/curl/pull/2659/files

 

./configure had to be manually re-run afterwards

--

Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/827d3c0f-2d2b-489d-9c47-a42b226ff7f3n%40lists.secondlife.com.