Openssl compilation problems [revisited]
Thomas Braun
I wanted to compile a new openssl version in msysgit. My hope is to get
it a bit faster, and of course the security updates also don't hurt.
After I didn't had any luck with version 1.0.0* I tried 0.9.8r, here the
jump from the current version 0.9.8k to r should not be that big.
After fixing one of the patches (0004-openssl-build-DLL-directly.patch)
still no luck.
The problem is a loop in openssl*/util/mkdef.pl line 880 which never
finishes.
As it turns out the problem is that the line endings of the complete
openssl* directory which was installed with release.sh was changed from
LF to CRLF presumably by fast-import.pl.
It now turns out that the EOL style of the archiv openssl-0.9.8r.tar.gz
was changed from LF to CRLF on unpacking by fast-import.pl and that
confuses the above mkdef.pl script.
Therefore even compiling the current version 0.9.8k with a fresh msysgit
installation (up to date) fails.
Suggestions on where to start fixing this problem?
PS: Yes I've used msys.bat to enter the msysgit environment
Erik Faye-Lund
<thomas...@virtuell-zuhause.de> wrote:
> Hi there,
>
> I wanted to compile a new openssl version in msysgit. My hope is to get it a
> bit faster
Perhaps something like this could help? (a patch to remove -march=i486
from the mingw-target...)
---8<---
diff --git a/Configure b/Configure
index c6dbfae..52a2801 100755
--- a/Configure
+++ b/Configure
@@ -480,7 +480,7 @@ my %table=(
"BC-32","bcc32::::WIN32::BN_LLONG DES_PTR RC4_INDEX
EXPORT_VAR_AS_FN:${no_asm}:win32",
# MinGW
-"mingw", "gcc:-mno-cygwin -DL_ENDIAN -fomit-frame-pointer -O3
-march=i486 -Wall -D_WIN32_WINNT=0x333:::MINGW32:-lwsock32
-lgdi32:BN_LLONG ${x86_gcc_des} ${x86_gcc_opts}
EXPORT_VAR_AS_FN:${x86_coff_asm}:win32:cygwin-shared:-D_WINDLL
-DOPENSSL_USE_APPLINK:-mno-cygwin -shared:.dll.a",
+"mingw", "gcc:-mno-cygwin -DL_ENDIAN -fomit-frame-pointer -O3 -Wall
-D_WIN32_WINNT=0x333:::MINGW32:-lwsock32 -lgdi32:BN_LLONG
${x86_gcc_des} ${x86_gcc_opts}
EXPORT_VAR_AS_FN:${x86_coff_asm}:win32:cygwin-shared:-D_WINDLL
-DOPENSSL_USE_APPLINK:-mno-cygwin -shared:.dll.a",
# UWIN
"UWIN", "cc:-DTERMIOS -DL_ENDIAN -O -Wall:::UWIN::BN_LLONG
${x86_gcc_des} ${x86_gcc_opts}:${no_asm}:win32",
---8<---
> and of course the security updates also don't hurt.
>
Yeah, this would be really nice. There's been found a serious issue in
OpenSSL since 0.9.8k was released.
> After I didn't had any luck with version 1.0.0* I tried 0.9.8r, here the
> jump from the current version 0.9.8k to r should not be that big.
> After fixing one of the patches (0004-openssl-build-DLL-directly.patch)
> still no luck.
> The problem is a loop in openssl*/util/mkdef.pl line 880 which never
> finishes.
>
I gave this a stab some weeks ago, and stumbled into this problem as well.
> As it turns out the problem is that the line endings of the complete
> openssl* directory which was installed with release.sh was changed from LF
> to CRLF presumably by fast-import.pl.
>
> It now turns out that the EOL style of the archiv openssl-0.9.8r.tar.gz was
> changed from LF to CRLF on unpacking by fast-import.pl and that confuses the
> above mkdef.pl script.
>
> Therefore even compiling the current version 0.9.8k with a fresh msysgit
> installation (up to date) fails.
>
> Suggestions on where to start fixing this problem?
>
Thanks for digging. Did you try to disable core.autocrlf before
running the script?
Erik Faye-Lund
As an alternative route, perhaps we could use a tag from
git://pubgit.metux.de/mirror/openssl.git (or
git://repo.or.cz/mirror-openssl.git) instead?
Johannes Schindelin
Good find! I think you need to integrate the call
git config core.autocrlf false
into the release.sh script.
Ciao,
Johannes
Thomas Braun
Am 07.04.2011 13:37, schrieb Johannes Schindelin:
> Good find! I think you need to integrate the call
>
> git config core.autocrlf false
>
> into the release.sh script.
>
Thanks for your answers Erik and Johannes,
indeed setting autocrlf to false fixed the problem.
I can now compile 0.9.8k again and also the newer version 0.9.8r :)
Good idea to track http://repo.or.cz/w/mirror-openssl.git instead of
downloading the tarball, but maybe we keep this task for later.
I took the liberty to push the changes to the mob branch, commits
a5f56cf..82b515b.
The important commits are
#########
commit e9a851b6a60ad583228b8454bad34f1169fb2ffb
Author: Thomas Braun <thomas...@virtuell-zuhause.de>
Date: Thu Apr 7 14:23:06 2011 +0200
update to openssl version 0.9.8r from 0.9.8k
commit 8b155bc37e114566d33db00fb87a028c7505d687
Author: Thomas Braun <thomas...@virtuell-zuhause.de>
Date: Thu Apr 7 14:21:53 2011 +0200
Set autocrlf to false to preserve unix eol-style
This makes it possible again to compile version 0.9.8k of openssl.
Using windows eol-style results in a endless loop in
openssl*/util/mkdef.pl line 880.
#######
bye,
Thomas
Thomas Braun
> Thanks, great work! These two patches look exactly like what I
> expected it to do, except for a sign-off. Do you mind if I add one?
Of course not :) I just forgot to do it.
> Did upgrading resolve your performance issues? If not, did you try
> with the patch I suggested?
No, using the new openssl version didn't help anything. Neither did
tweaking the gcc options as you suggested.
I'm still stuck with 1MiB/s on Windows XP compared to 10MiB/s on my
debian . The command was git clone (same box, same remote repo on
GBit-Ethernet, connection via ssh).
>
> You see, if I'm going to add that patch, I'd like to do so before
> merging this topic :)
>
Yes you are right.
So basically we now have a new openssl version, but from a performance
POV nothing changed.
Looking at the system monitor on windows also doesn't show high CPU load
or memory consumption during cloning. The system is just idling during
cloning.
regards,
Thomas
Thomas Braun
> I'm still stuck with 1MiB/s on Windows XP compared to 10MiB/s on my
> debian . The command was git clone (same box, same remote repo on
> GBit-Ethernet, connection via ssh).
Actually openssl isn't used in this type of setup if I understand it
properly.
Erik Faye-Lund
<thomas...@virtuell-zuhause.de> wrote:
> Am Donnerstag, den 07.04.2011, 15:30 +0200 schrieb Erik Faye-Lund:
>> Thanks, great work! These two patches look exactly like what I
>> expected it to do, except for a sign-off. Do you mind if I add one?
>
> Of course not :) I just forgot to do it.
>
>> Did upgrading resolve your performance issues? If not, did you try
>> with the patch I suggested?
>
> No, using the new openssl version didn't help anything. Neither did
> tweaking the gcc options as you suggested.
>
> I'm still stuck with 1MiB/s on Windows XP compared to 10MiB/s on my
> debian . The command was git clone (same box, same remote repo on
> GBit-Ethernet, connection via ssh).
>
>>
>> You see, if I'm going to add that patch, I'd like to do so before
>> merging this topic :)
>>
>
> Yes you are right.
> So basically we now have a new openssl version, but from a performance
> POV nothing changed.
>
OK, I've run release.sh and merged the result.
I did remove the -march=i486 flag. It's not a big deal, but it should
be a sliiight improvement at least ;)
So yes, at least we have a newer and more secure openssl. Thanks for
putting in the work!
> Looking at the system monitor on windows also doesn't show high CPU load
> or memory consumption during cloning. The system is just idling during
> cloning.
Hmh, strange. Perhaps you could try to run it in gdb and try to break
the process to see if there's an obvious stall?
Thomas Braun
> So yes, at least we have a newer and more secure openssl. Thanks for
> putting in the work!
Well only for https stuff I guess. At least
###
$ ssh -v
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
###
suggests that openssl is statically linked into ssh.
I'm currrently trying to compile a newer openssh version also, rumor in
the internet suggests that this might also help in improving speed.
Btw, is there a standard replacement in msysgit for the missing
sys/socket.h include file?
I ask because ./configure of openssh[1] complains that socklen_t is not
defined.
>> Looking at the system monitor on windows also doesn't show high CPU load
>> or memory consumption during cloning. The system is just idling during
>> cloning.
>
> Hmh, strange. Perhaps you could try to run it in gdb and try to break
> the process to see if there's an obvious stall?
>
Good idea. I'll give it a shot.
[1]: Patched version 5.4p1-1 from msys project
Thomas Braun
I'm back on the openssl topic.
We now have openssl version 0.9.8r in /mingw/bin, but we still have
0.9.8e in /bin.
The one in /bin is compiled in msys and therefore links implicitly to
msys-1.0.dll.
So if I want to upgrade the one in /bin I just copy the libraries and
executables from msys and push the changes to mob?
Thanks for your patience :),
Thomas
PS: I've managed compiling openssl 1.0.0d in /src/openssl. It uses the
standard build system (no more mingw32.bat) as suggested on the openssl
mailing list [1].
Only one patch is still necessary (the drop march-486 one).
The result looks a bit different, although compiled as shared lib, as
the main dlls are named ssleay32.dll and libeay32.dll. Nevertheless
openssl.exe works and "make tests" passes all tests.
[1]: http://www.mail-archive.com/opens...@openssl.org/msg27545.html
Erik Faye-Lund
<thomas...@virtuell-zuhause.de> wrote:
> Am 08.04.2011 14:54, schrieb Erik Faye-Lund:
>>
>> So yes, at least we have a newer and more secure openssl.
>
> I'm back on the openssl topic.
> We now have openssl version 0.9.8r in /mingw/bin, but we still have 0.9.8e
> in /bin.
>
/mingw/bin is what we distribute as /bin for Git for Windows (and link
to when we create native windows binaries), /bin is our msys
environment.
In other words, this is exactly as expected.
> The one in /bin is compiled in msys and therefore links implicitly to
> msys-1.0.dll.
>
Yes, but this one shouldn't be used at all, should it?
> So if I want to upgrade the one in /bin I just copy the libraries and
> executables from msys and push the changes to mob?
>
Why do you want to upgrade it in the first place? Have you placed /bin
before /mingw/bin in the path or something?
I'm thinking that we should probably just delete the openssl
installation in /bin, to make sure it doesn't get in the way...
> PS: I've managed compiling openssl 1.0.0d in /src/openssl. It uses the
> standard build system (no more mingw32.bat) as suggested on the openssl
> mailing list [1].
So they finally fixed their make-system to work on MinGW? Nice.
> Only one patch is still necessary (the drop march-486 one).
>
Nice. Does this mean that the issues solved in the other patches are
either fixed or made redundant due to the build-system switch?
> The result looks a bit different, although compiled as shared lib, as the
> main dlls are named ssleay32.dll and libeay32.dll. Nevertheless openssl.exe
> works and "make tests" passes all tests.
>
> [1]: http://www.mail-archive.com/opens...@openssl.org/msg27545.html
>
Nice work. Where can I find the recepie to reproduce this?
Erik Faye-Lund
> On Wed, Apr 13, 2011 at 11:21 AM, Thomas Braun
> <thomas...@virtuell-zuhause.de> wrote:
>> Am 08.04.2011 14:54, schrieb Erik Faye-Lund:
>>>
>>> So yes, at least we have a newer and more secure openssl.
>>
>> I'm back on the openssl topic.
>> We now have openssl version 0.9.8r in /mingw/bin, but we still have 0.9.8e
>> in /bin.
>>
>
> /mingw/bin is what we distribute as /bin for Git for Windows (and link
> to when we create native windows binaries), /bin is our msys
> environment.
>
> In other words, this is exactly as expected.
>
Hmm, it seems this explanation is wrong. We distribute a mixture of
/bin and /mingw/bin, and unless my understanding of
/share/WinGit/copy-files.sh is way off openssl.exe is actually picked
from /bin.
This should probably be fixed so we distribute the new version.
Something like this?
diff --git a/share/WinGit/copy-files.sh b/share/WinGit/copy-files.sh
index 6672172..d953a45 100644
--- a/share/WinGit/copy-files.sh
+++ b/share/WinGit/copy-files.sh
@@ -49,7 +49,7 @@
ssh-agent.exe,ssh.exe,ssh-add.exe,ssh-keygen.exe,ssh-keyscan.exe,\
tail.exe,tar.exe,tee.exe,touch.exe,tr.exe,true.exe,uname.exe,uniq.exe,vi,\
msys-perl5_8.dll,lib{apr,aprutil,expat,neon,z,svn}*.dll,pthreadGC2.dll,\
msys-crypto-0.9.8.dll,msys-regex-1.dll,msys-ssl-0.9.8.dll,msys-minires.dll,msys-z.dll,\
-openssl.exe,vim,wc.exe,which,xargs.exe,start} lib/engines/ \
+vim,wc.exe,which,xargs.exe,start} lib/engines/ \
ssl/ cmd/ lib/perl5/ share/antiword/ share/bison/ share/git* \
share/vim/vimrc share/vim/vim73/{filetype.vim,ftoff.vim,menu.vim,optwin.vim,\
scripts.vim,\
@@ -74,7 +74,7 @@ fi &&
(cd /mingw && tar cf - \
bin/*{tcl,tk,wish,gpg,msmtp,curl.exe,*.crt}* bin/connect.exe \
bin/*{libcurl,libcrypto,libssl,libgsasl,libiconv}* \
- bin/getcp.exe bin/rebase.exe \
+ bin/getcp.exe bin/rebase.exe bin/openssl.exe \
bin/{libpoppler-7.dll,pdfinfo.exe,pdftotext.exe} \
lib/{tcl,tk,dde,reg}* libexec/gnupg/) |
tar xf - &&
Erik Faye-Lund
lib/engines should probably be removed also:
-openssl.exe,vim,wc.exe,which,xargs.exe,start} lib/engines/ \
+vim,wc.exe,which,xargs.exe,start} \
Thomas Braun
> On Wed, Apr 13, 2011 at 11:21 AM, Thomas Braun
> <thomas...@virtuell-zuhause.de> wrote:
>> Am 08.04.2011 14:54, schrieb Erik Faye-Lund:
>>>
>>> So yes, at least we have a newer and more secure openssl.
>>
>> I'm back on the openssl topic.
>> We now have openssl version 0.9.8r in /mingw/bin, but we still have 0.9.8e
>> in /bin.
>>
>
> /mingw/bin is what we distribute as /bin for Git for Windows (and link
> to when we create native windows binaries), /bin is our msys
> environment.
>
> In other words, this is exactly as expected.
Ah okay, that's the missing piece of information I was looking for.
>> The one in /bin is compiled in msys and therefore links implicitly to
>> msys-1.0.dll.
>>
>
> Yes, but this one shouldn't be used at all, should it?
>
>> So if I want to upgrade the one in /bin I just copy the libraries and
>> executables from msys and push the changes to mob?
>>
>
> Why do you want to upgrade it in the first place? Have you placed /bin
> before /mingw/bin in the path or something?
No, just out of the I-don't-want-vulnerable-openssl-lying-around-feeling.
> I'm thinking that we should probably just delete the openssl
> installation in /bin, to make sure it doesn't get in the way...
It is already getting in the way, isn't it?
At least in my local git for windows installation in
"C:\Programme\Git\bin" I have
msys-crypto-0.9.8.dll
msys-ssl-0.9.8.dll
openssl.exe
which, if I understand you correctly, shouldn't be there.
>> PS: I've managed compiling openssl 1.0.0d in /src/openssl. It uses the
>> standard build system (no more mingw32.bat) as suggested on the openssl
>> mailing list [1].
>
> So they finally fixed their make-system to work on MinGW? Nice.
>
>> Only one patch is still necessary (the drop march-486 one).
>>
>
> Nice. Does this mean that the issues solved in the other patches are
> either fixed or made redundant due to the build-system switch?
Exactly.
>> The result looks a bit different, although compiled as shared lib, as the
>> main dlls are named ssleay32.dll and libeay32.dll. Nevertheless openssl.exe
>> works and "make tests" passes all tests.
>>
>> [1]: http://www.mail-archive.com/opens...@openssl.org/msg27545.html
>>
>
> Nice work. Where can I find the recepie to reproduce this?
>
That's still on my local hard drive only.
I will clean it up and push it to mob (or send it to the mailing list if
it is too intrusive for mob).
bye,
thomas
Thomas Braun
> Hmm, it seems this explanation is wrong. We distribute a mixture of
> /bin and /mingw/bin, and unless my understanding of
> /share/WinGit/copy-files.sh is way off openssl.exe is actually picked
> from /bin.
>
> This should probably be fixed so we distribute the new version.
> Something like this?
>
> diff --git a/share/WinGit/copy-files.sh b/share/WinGit/copy-files.sh
> index 6672172..d953a45 100644
> --- a/share/WinGit/copy-files.sh
> +++ b/share/WinGit/copy-files.sh
> @@ -49,7 +49,7 @@
> ssh-agent.exe,ssh.exe,ssh-add.exe,ssh-keygen.exe,ssh-keyscan.exe,\
> tail.exe,tar.exe,tee.exe,touch.exe,tr.exe,true.exe,uname.exe,uniq.exe,vi,\
> msys-perl5_8.dll,lib{apr,aprutil,expat,neon,z,svn}*.dll,pthreadGC2.dll,\
> msys-crypto-0.9.8.dll,msys-regex-1.dll,msys-ssl-0.9.8.dll,msys-minires.dll,msys-z.dll,\
> -openssl.exe,vim,wc.exe,which,xargs.exe,start} lib/engines/ \
> +vim,wc.exe,which,xargs.exe,start} lib/engines/ \
> ssl/ cmd/ lib/perl5/ share/antiword/ share/bison/ share/git* \
> share/vim/vimrc share/vim/vim73/{filetype.vim,ftoff.vim,menu.vim,optwin.vim,\
> scripts.vim,\
> @@ -74,7 +74,7 @@ fi&&
> bin/*{tcl,tk,wish,gpg,msmtp,curl.exe,*.crt}* bin/connect.exe \
> bin/*{libcurl,libcrypto,libssl,libgsasl,libiconv}* \
> - bin/getcp.exe bin/rebase.exe \
> + bin/getcp.exe bin/rebase.exe bin/openssl.exe \
> bin/{libpoppler-7.dll,pdfinfo.exe,pdftotext.exe} \
> lib/{tcl,tk,dde,reg}* libexec/gnupg/) |
> tar xf -&&
>
I also ready copy-files.sh like that. To really remove this old version
of openssl we would also have to not-ship msys-crypto-0.9.8.dll and
msys-ssl-0.9.8.dll from /bin.
But then ssh.exe is not usable any more, which confuses me then even more.
Johannes Schindelin
I agree. Could you squash those changes into one commit and push?
But then, maybe there are things that would not work with ssh anymore, as
lib/enginges/ is incurred by the OpenSSL.
Thanks,
Dscho
Johannes Schindelin
To help your I-do-not-want-buggy-openssl feeling, both OpenSSH (and
OpenSSL) would need to be recompiled at a newer version with MSys, and
OpenSSL with MinGW.
Ciao,
Johannes
Thomas Braun
> Nice work. Where can I find the recepie to reproduce this?
>
6f87459252f6c468aaea94e9b0919bf1993a835f on mob.
I hope I didn't screw up, because today it took me ages to push it properly.
Thomas Braun
Is there anything more I can/should do to get this merged in devel?
bye,
thomas
Erik Faye-Lund
Yes, and sorry for not looking deeply before now:
1) Rebase your work to 'devel'. It doesn't apply cleanly.
2) Create one commit for the changes to /src/openssl/-stuff only.
/src/openssl/release.sh will generate the commits for the upgrading
itself.
3) Don't suggest that 0006-do-not-specify-march-i486.patch has simply
been renamed, it had to be recreated, right? (i.e the patch-file has
new contents)
4) Just force-push the result to mob. There's nothing else interesting
there anyway.