Building recent versions on Windows with VS 2013 and MASM

[Steven Kneizys]

I wanted to be able to build 32 and 64 bit versions of recent OpenSSL
1.0.x production and beta/snapshots using Visual Studio Express
edition. The new Visual Studios include ml/ml64, and while NASM is
great sometimes it is nice to have the option of MASM as well.

I was trying to do things with the built-in translator and other
support code but decided to get the biggest initial return on invested
time that I would do the translation work out the build itself at
first, then integrate later, especially since 1.0.2 is already in beta
and 1.0.1 is winding down. The best way I could see to do that is to
use what is already standard, the NASM flavour (dialect), and go to
MASM from there. So the strategy was to make the OpenSSL build think
it was going to send to NASM, but really I would translate to MASM.
So I created my experimental "fake nasm".

It is available here if anybody wants to play with it:
http://erpdata.com/fakenasm

If run, it will sort of say it is "nasm":
E:\usr_local_src\>nasm -v
NASM version 2.11.02 compiled on Feb 19 2014 (fakeout ver 1.0 03-MAR-2014)

Actually, it is a tiny C program that forwards the request to a ".bat"
file, and a perl script, which then does a translation of the dialect,
creating an intermediate code file ending in ".XMASM" (meaning
translated to MASM) which is then fed to ml/ml64. Here is a sample
from a build:


E:\usr_local_src\openssl-1.0.1f_64>nmake -f ms\ntdll.mak

Microsoft (R) Program Maintenance Utility Version 12.00.21005.1
Copyright (C) Microsoft Corporation. All rights reserved.

Building OpenSSL
set ASM=nasm -f win64 -DNEAR -Ox -g
perl crypto\x86_64cpuid.pl tmp32dll\x86_64cpuid.asm
nasm -f win64 -DNEAR -Ox -g -o tmp32dll\x86_64cpuid.obj
tmp32dll\x86_64cpuid.asm
NASM version 2.11.02 compiled on Feb 19 2014 (fakeout ver 1.0 03-MAR-2014)
Assembling tmp32dll\x86_64cpuid.asm to tmp32dll\x86_64cpuid.obj (via
tmp32dll\x86_64cpuid.asm.XMASM)
ml64 /c /Cp /Cx /Zi /Fotmp32dll\x86_64cpuid.obj tmp32dll\x86_64cpuid.asm.XMASM
Microsoft (R) Macro Assembler (x64) Version 12.00.21005.1
Copyright (C) Microsoft Corporation. All rights reserved.

Assembling: tmp32dll\x86_64cpuid.asm.XMASM

It tells the OpenSSL build it is "nasm", but then passes the request
on to ml or ml64. By keep the translation out of the OpenSSL tree at
first I was able to apply it to a wide variety of OpenSSL snapshots
and releases. I was just able to build 32 and 64 bit version of:

1.0.1f
openssl-1.0.2-stable-SNAP-20140303
openssl-1.0.2-stable-SNAP-20140305

All you have to do is put the "fake nasm" in your path instead of the
real one, then build as usual using Visual Studio 2013 x86 or x64
command tools prompt, and it will have ml/ml64 do the assembly parts.

While I am able to build those versions and they pass the "ms\test", I
do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
anybody want to test it on those processors? :-)

Eventually I hope to put what I am learning into the base openssl so
maybe ml/ml64 might be supported again.

Thanks,

Steve...

On Tue, Mar 4, 2014 at 9:11 AM, Steven Kneizys <skne...@ferrilli.com> wrote:
> I'm working on a little fun thing to see if I can get Visual Studio to
> compile and assemble (64 bit) the 1.0.2 snapshots ... more on that
> later on in the week. I am trying to figure out what the MASM 12.x
> assembler wants that is different from what NASM takes right now. I
> know things are going through the translator on OpenSSL, but the
> current MASM-generated code is not working, and I know it is
> officially unsupported. But I am looking at it :-)
>
> On Fri, Feb 28, 2014 at 5:00 PM, Andy Polyakov <ap...@openssl.org> wrote:
>>> I remembered seeing XMMWORD being taken into consideration if the
>>> opcode args were of the xmm variety, so out of curiosity I added a
>>> little code to my previous patch to modify the opcode's third arg as
>>> well, if present:
>>
>>
>> It will be looked into as time permits, but not for 1.0.2 release. If
>> anything one should also look into AVX2 and AVX512 support
>>
>> ______________________________________________________________________
>> OpenSSL Project http://www.openssl.org
>> Development Mailing List opens...@openssl.org
>> Automated List Manager majo...@openssl.org
>
>
>
> --
> Steve Kneizys
> Senior Business Process Engineer
> Voice: (610) 256-1396 [For Emergency Service (888)864-3282]
> Ferrilli Information Group -- Quality Service and Solutions for Higher Education
> web: http://www.ferrilli.com/
>
> Making you a success while exceeding your expectations.



--
Steve Kneizys
Senior Business Process Engineer
Voice: (610) 256-1396 [For Emergency Service (888)864-3282]
Ferrilli Information Group -- Quality Service and Solutions for Higher Education
web: http://www.ferrilli.com/

Making you a success while exceeding your expectations.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List opens...@openssl.org
Automated List Manager majo...@openssl.org

[Steven Kneizys]

Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
supported or just nasm?

Thanks,

Steve...

[Andy Polyakov]

> Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
> supported or just nasm?

Quick answer: quantum mechanics. Longer answer: state is unknown till
it's measured. Long answer: reports are not rejected, but
when-in-doubt-use-nasm principle applies, and reaction to reports can as
well be avoiding problem.

As for supporting MASM in more general terms. You have to do better than
"it's *nice* to have MASM". You have to show why is it so, what is it
MASM offers that NASM doesn't. Even if there are some/any, they would be
weighed against advantages NASM has to offer (such as unconditional
availability for immediate download, possibility to affect the
functionality, cross-platform support). And even then faking nasm is
hardly appropriate way to solve the problem.

>> While I am able to build those versions and they pass the "ms\test", I
>> do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
>> anybody want to test it on those processors? :-)

http://software.intel.com/en-us/articles/intel-software-development-emulator/

[Steven Kneizys]

>> Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64

>> supported or just nasm?

>Quick answer: quantum mechanics. Longer answer: state is unknown till

>it's measured. Long answer: reports are not rejected, but

>when-in-doubt-use-nasm principle applies, and reaction to reports can as

>well be avoiding problem.

Well then I will definitely send in my more complete analysis/proposed fixes.

>As for supporting MASM in more general terms. You have to do better than

>"it's *nice* to have MASM". You have to show why is it so, what is it

>MASM offers that NASM doesn't. Even if there are some/any, they would be

>weighed against advantages NASM has to offer (such as unconditional

>availability for immediate download, possibility to affect the

>functionality, cross-platform support). And even then faking nasm is

>hardly appropriate way to solve the problem.

OpenSSL is security software.  I have clients who have auditors that make 

demands sometimes.  I just helped a client get on 1.0.1f because of an 

issue with 1.0.1e that a government auditor noted.  What if the auditors 

say "and why is it that it cannot be built for the Microsoft platform with 

the Microsoft platform tools?".  It would be SO nice to not have to have that 

conversation, ever.

And if my analysis is correct, OpenSSL is way over 99% of the way there, all

the hooks are still there, it is just a little tiny bit of tweaking.  I feel like we'd

be making a mistake to let this opportunity slip away, now that the native tool

MASM is installed even on the free "Express Edition" of Visual Studio 2012/2013

in both 32 and 64 bit.  No extra download.  Platform vendor tool.  Supportable by 

vendor.  Free.  Automatic software update mechanism.

And faking nasm, well, it is a nasm emulator ... but as the name denotes, I

really didn't want to call it a "real" tool exactly.  Just one for development

analysis work if we can get OpenSSL to use ml/ml64 natively.  But if not,

I can build it anytime I want to and have control over new features :-)

>>> While I am able to build those versions and they pass the "ms\test", I

>>> do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...

>>> anybody want to test it on those processors?  :-)

>http://software.intel.com/en-us/articles/intel-software-development-emulator/

Yes, I tried that, testing / dev tools like this are the inspiration for the

fake nasm.  Sort of a "just inject this tool in the middle and you can learn

some things" so to speak.

On the SDE, I was seeing the AVX/AVX2 from my lil CPUID program (not the greatest

and the Intel one doesn't seem to work with the Intel SDE!)  AVX512 support didn't

seem to be turned on in the emulator, I tried the edition released yesterday and the 

previous one as  well.  I tried the command line switches but couldn't figure that part out.

When I ran my fake-nasm generated 1.0.2 snapshot 64-bit OpenSSL code 

through the SDE (while AVX/AVX2 seemed to be on) it said all tests passed. 

But I think I'd like to see it on something "real"!