Using fuzzing to improve operating system quality

[Simon Clubley]

The Linux developers use fuzzing as a testing method to find problems
in Linux:

https://www.theregister.co.uk/2017/10/16/linus_torvalds_says_fuzzing_is_improving_linux_kernel_security/

I wonder if the VSI developers have considered doing the same ?

BTW, when I found the problem with nulls in the DCL recall buffer,
I initially realised there was a problem when I fed the output from
the monitor recorder into DCL (just to see what would happen) and then
promptly crashed DCL.

IOW, fuzzing the DCL inputs led me to realise there was a problem
somewhere in DCL. Although it did take me a while to find the pattern
that was causing the crash, fuzzing DCL made me realise there was a
problem in the first place.

Simon.

--
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

[John Reagan]

On Monday, October 16, 2017 at 8:16:52 AM UTC-4, Simon Clubley wrote:
> The Linux developers use fuzzing as a testing method to find problems
> in Linux:
>
> https://www.theregister.co.uk/2017/10/16/linus_torvalds_says_fuzzing_is_improving_linux_kernel_security/
>
> I wonder if the VSI developers have considered doing the same ?
>

Can't speak to the OS itself, but the compilers have used some fuzzing over the years. We have some pseudo-random program generators in some of the language test systems (and some of the C++ template tests look like random line noise to start with).

And to test the testing harness itself, I'll occasionally point the test system to something bogus to test. I remember pointing the Pascal test system at a Fortran compiler and still finding 10 tests out of 9000 that passed. :)

I also broke the Pascal compiler once by doing:

$ pascal sys$system:fortran.exe

[John Reagan]

FYI, the LLVM docs on LLVM fuzzing have recently been updated to show all the tools that are currently in use on LLVM on other platforms.

https://reviews.llvm.org/rL315544

[Richard Maher]

On 16-Oct-17 9:44 PM, John Reagan wrote:
> On Monday, October 16, 2017 at 8:16:52 AM UTC-4, Simon Clubley
> wrote:
>> The Linux developers use fuzzing as a testing method to find
>> problems in Linux:
>>
>> https://www.theregister.co.uk/2017/10/16/linus_torvalds_says_fuzzing_is_improving_linux_kernel_security/
>>
>>
>>
>> I wonder if the VSI developers have considered doing the same ?
>>
>
> Can't speak to the OS itself, but the compilers have used some
> fuzzing over the years. We have some pseudo-random program
> generators in some of the language test systems (and some of the C++
> template tests look like random line noise to start with).
>

And we'd have true hardware supported random generators if you would've
just seen sense and added an optional flags argument to m[a]th$random :-(

[John Reagan]

On Alpha or Itanium? I did ask about the future with x86. I'd like to offer optional support there but was also convinced to be suspicious about the Intel instructions.

[Richard Maher]

Yes, x86 because you mentioned it supported it. Just add an opt-in flags
and it's done.

[Simon Clubley]

And so are you if there turns out to be a backdoor in the generator.

[Simon Clubley]

On 2017-10-16, John Reagan <xyzz...@gmail.com> wrote:
>
> And to test the testing harness itself, I'll occasionally point the test
> system to something bogus to test. I remember pointing the Pascal test system
> at a Fortran compiler and still finding 10 tests out of 9000 that passed. :)
>

:-)

In fairness however, it's possible that some simple examples of source
code could appear to be valid to another language's compiler.

Although I have forgotten the details, there used to be competitions
where you wrote source code in different languages in the same one
physical file and got different language compilers to compile the one
file without error.

> I also broke the Pascal compiler once by doing:
>
> $ pascal sys$system:fortran.exe
>

Yes, it's amazing what you can break when you do the totally unexpected. :-)

[David Froble]

Simon Clubley wrote:
> On 2017-10-16, Richard Maher <maher_rj...@hotmail.com> wrote:
>> On 17-Oct-17 9:46 AM, John Reagan wrote:
>>> On Alpha or Itanium? I did ask about the future with x86. I'd like
>>> to offer optional support there but was also convinced to be
>>> suspicious about the Intel instructions.
>>>
>> Yes, x86 because you mentioned it supported it. Just add an opt-in flags
>> and it's done.
>
> And so are you if there turns out to be a backdoor in the generator.
>
> Simon.
>

There could be back doors in just about anything, so what's so special about this?

[johnwa...@yahoo.co.uk]

The "what if there's something dodgy in the code generator"
subject goes back long before "security reseearch" was a
marketable thing worthy of column inches (or C21 equivalent).

E.g.
Ken Thompson's Reflections on Trusting Trust
That was in 1984 (the year, not the book).
Lots of subsequent comment and analysis too.

No links provided here, readers who aren't familiar can find
their own (assuming the search engines haven't been got at).

[Bill Gunshannon]

I have recommended reading that numerous times. I used to (and still
do when given the chance) recommend it to all my students. But, sadly,
I don't think anyone actually listens.

bill

[David Froble]

What? You don't agree with "everyone else does it"?

Seems like the only people worried about security are those who have already
suffered. And all they seem to worry about is the suffering.

[Bob Gezelter]

John,

I do not have the time to dig up the citation, but I remember an earlier reference, probably in ACM SIGPLAN on the problem of trusting compilers not to insert backdoors. From my recollections, it was pre-1980.

- Bob Gezelter, http://www.rlgsc.com

[Simon Clubley]

Because it's in hardware - that's why the extra concerns are justified.

Software backdoors can be found by security researchers who are motivated
to reveal them. Hardware backdoors generally require the resources of a
nation state to find and they are likely to keep those discoveries to
themselves while leaving everyone vulnerable.

[Bill Gunshannon]

When have I ever put forth that kind of signal? I have long been an
advocate of serious risk management but that, in itself, is a hard
sell. Going beyond that to implementing security is even harder.
Only reason I was able to do what I did at the University was the
fact that my boss gave me full autonomy as long as the faculty and
students could get their work done. We had only one breach since
the days when Windows98 went out the door and that was a single users
account and the reason was he used the same userid and passwords as
his department one on a WordPress site on the Internet.

This as opposed to the current regimes "security" measure which
basically blocks all incoming capabilities except thru their VPN.

>
> Seems like the only people worried about security are those who have
> already suffered.  And all they seem to worry about is the suffering.

I don't agree. I was able to keep our systems locked down while still
allowing users to do anything they needed to do.

bill

[Simon Clubley]

On 2017-10-17, johnwa...@yahoo.co.uk <johnwa...@yahoo.co.uk> wrote:
> On Tuesday, 17 October 2017 13:28:42 UTC+1, Simon Clubley wrote:
>> On 2017-10-16, Richard Maher <maher_rj...@hotmail.com> wrote:
>> > On 17-Oct-17 9:46 AM, John Reagan wrote:
>> >> On Alpha or Itanium? I did ask about the future with x86. I'd like
>> >> to offer optional support there but was also convinced to be
>> >> suspicious about the Intel instructions.
>> >>
>> >
>> > Yes, x86 because you mentioned it supported it. Just add an opt-in flags
>> > and it's done.
>>
>> And so are you if there turns out to be a backdoor in the generator.
>>
>

> The "what if there's something dodgy in the code generator"
> subject goes back long before "security reseearch" was a
> marketable thing worthy of column inches (or C21 equivalent).
>

It's not a code generator but a hardware based random number generator
which is the concern here.

Some reading for those who do not understand the concerns:

https://en.wikipedia.org/wiki/RdRand#Reception

[Simon Clubley]

On 2017-10-17, David Froble <da...@tsoft-inc.com> wrote:
>

> Seems like the only people worried about security are those who have already
> suffered. And all they seem to worry about is the suffering.

No. Some of us are worried about _other_ people suffering and that is
based purely on knowledge gained while trying to keep ourselves up to
date on developments in the security ecosystem.

[Hans Vlems]

John
Can you provide a Pascal example that passed the Fortran test? I can’t think of a program that would...
May be I’m limited by Fortran IV.
Hans

[David Froble]

I think your ability to recognize sarcasm needs a booster shot?

[David Froble]

Simon Clubley wrote:
> On 2017-10-17, David Froble <da...@tsoft-inc.com> wrote:
>> Seems like the only people worried about security are those who have already
>> suffered. And all they seem to worry about is the suffering.
>
> No. Some of us are worried about _other_ people suffering and that is
> based purely on knowledge gained while trying to keep ourselves up to
> date on developments in the security ecosystem.
>
> Simon.
>

Yeah, Ok, but:

What do you do when you hear the following from the customers, you know, the
ones who cough up the money?

"everyone else does it"

and

"my boss doesn't care about security"

Brings to mind something such as "lead a horse to water but cannot make him drink".

Until you get all the users (and you won't) to care about security, you're
doomed to failure.

Yes, do your best, at least you'll know you did your part, but it's like the
humming bird trying to put out the forest fire, one drop at a time.

[David Froble]

Or, the Intel compiler that checked if it was running on Intel or AMD?

Don't know anything about it, just read it somewhere.

[Bob Koehler]

In article <os5kpq$tvr$1...@dont-email.me>, David Froble <da...@tsoft-inc.com> writes:
>
> Brings to mind something such as "lead a horse to water but cannot make him
> drink".
>

Yuo can lead a programer to water, but you're not alowed to throw him in.

[Simon Clubley]

On 2017-10-17, David Froble <da...@tsoft-inc.com> wrote:

> Simon Clubley wrote:
>>
>> No. Some of us are worried about _other_ people suffering and that is
>> based purely on knowledge gained while trying to keep ourselves up to
>> date on developments in the security ecosystem.
>>
>

> Yeah, Ok, but:
>
> What do you do when you hear the following from the customers, you know, the
> ones who cough up the money?
>
> "everyone else does it"
>
> and
>
> "my boss doesn't care about security"
>
> Brings to mind something such as "lead a horse to water but cannot make him drink".
>
> Until you get all the users (and you won't) to care about security, you're
> doomed to failure.
>

In many cases, you don't need to get the users to care, you only need
to get the insurance companies and the regulatory authorities to care.

Responsible disclosure has had a major beneficial effect on the
security ecosystem by forcing vendors to fix problems.

For the next stage (dealing with the end user organisations), I strongly
suspect (if they are not doing it already) that insurance companies are
going to start requiring organisations to keep their systems up to date
as part of the conditions for insuring that organisation.

This is what some regulatory authorities already require and for the
insurance companies this is just an extension of requiring the
organisation to have appropriate physical security measures before
they will insure that organisation.

> Yes, do your best, at least you'll know you did your part, but it's like the
> humming bird trying to put out the forest fire, one drop at a time.

Thanks to various cultural changes and policies like responsible
disclosure, that humming bird is currently getting larger and larger.

[Bill Gunshannon]

I don't know. I can't say my boss didn't care about security
but he definitely never told me to increase ours. He just let
do what I thought was needed. And as for users, I dealt with
several hundred students every semester all of whom couldn't
care less about security. We got by OK.

bill

[John Reagan]

It was a bug in the test harness. Nobody thinks to test the test harness or even attempt to reproduce the failure with the newly added test pointed at an older compiler.

[Stephen Hoffman]

On 2017-10-16 12:16:49 +0000, Simon Clubley said:

> The Linux developers use fuzzing as a testing method...

VSI has to finish the x86-64 port with the production release arriving
circa 2020, and work to secure and to authenticate and to
upgrade-to-current and otherwise contend with the known problems and
known limitations (ftp, telnet, DECnet, down-revision network services,
certificate-handling, libtls and/or libsodium and/or other APIs for
secure transport and distributed authentication, getting security
reporting launched, implementing telemetry,
https://github.com/icing/mod_md and/or additional ACME support,
expunging known-insecure protocols and CDSA, etc), and around the VSI
management and development team and ISVs and end-user developers all
becoming more familiar with modern threats and environments and
attacks, all before expending any particular effort on rummaging for
more problems.

Yes, I'd certainly expect to see fuzzing used and be useful, but I'd
make a very substantial wager that the team at VSI presently doesn't
presently need to go looking for yet more work and for yet more
security-relevant work.

There are various fuzzers around, including for APIs and network
protocols and file formats, and those that target specific
architectures almost always target x86-64.
http://lcamtuf.coredump.cx/afl/ Etc. But if you're running a
down-revision network stack with known holes in (for instance) tcpdump
or Apache or otherwise, why bother rummaging for yet more bugs?

If you're interested in what sorts of files can be handed to some apps,
ponder what could have happened secondary to some proposed changes to
the SVG format: https://www.w3.org/TR/2004/WD-SVG12-20040510/#rawsocket
— other sorts of documents are increasingly routinely executable, too.



From some of the other replies...

BTW and IIRC, Mr Reagan once broke Pascal by installing it on a VAX
9000, with the compiler crashing ~one compilation out of three.

And while on the subject of Pascal:
https://dubst3pp4.github.io/post/2017-10-03-why-i-use-object-pascal/



--
Pure Personal Opinion | HoffmanLabs LLC

[Arne Vajhøj]

On 10/18/2017 2:26 PM, Stephen Hoffman wrote:
> And while on the subject of Pascal:
> https://dubst3pp4.github.io/post/2017-10-03-why-i-use-object-pascal/

Pascal is a very nice language.

And VMS Pascal is a very nice implementation of ISO Pascal.

But the industry trend has gone in a different direction.

And Delphi added a lot of stuff to the language to keep the
language up to date, but not all of that are in the true
Pascal spirit.

Arne

[Arne Vajhøj]

On 10/17/2017 1:29 PM, Simon Clubley wrote:
> On 2017-10-17, johnwa...@yahoo.co.uk <johnwa...@yahoo.co.uk> wrote:
>> On Tuesday, 17 October 2017 13:28:42 UTC+1, Simon Clubley wrote:
>>> On 2017-10-16, Richard Maher <maher_rj...@hotmail.com> wrote:
>>>> On 17-Oct-17 9:46 AM, John Reagan wrote:
>>>>> On Alpha or Itanium? I did ask about the future with x86. I'd like
>>>>> to offer optional support there but was also convinced to be
>>>>> suspicious about the Intel instructions.
>>>>>
>>>>
>>>> Yes, x86 because you mentioned it supported it. Just add an opt-in flags
>>>> and it's done.
>>>
>>> And so are you if there turns out to be a backdoor in the generator.
>>>
>>
>> The "what if there's something dodgy in the code generator"
>> subject goes back long before "security reseearch" was a
>> marketable thing worthy of column inches (or C21 equivalent).
>
> It's not a code generator but a hardware based random number generator
> which is the concern here.
>
> Some reading for those who do not understand the concerns:
>
> https://en.wikipedia.org/wiki/RdRand#Reception

Somehow I think this type of concern actually reduces
internet security.

There are so many real threats that can and should
be protected against.

Spending time thinking about possible possible
backdoor scenarios like this seems like a
distraction to me.

Arne

[Chris]

On 10/18/17 18:26, Stephen Hoffman wrote:

>
> If you're interested in what sorts of files can be handed to some apps,
> ponder what could have happened secondary to some proposed changes to
> the SVG format: https://www.w3.org/TR/2004/WD-SVG12-20040510/#rawsocket
> — other sorts of documents are increasingly routinely executable, too.
>

Any sort of rpc is suspect as well. Needs to be a way to secure script
and rpc execution, as most modern systems depend on it...

Regards,

Chris

[Simon Clubley]

We are talking about an intelligence service (GCHQ) that once tried to
compromise Gemalto. _Gemalto_ for goodness sake. :-( For a reminder:

https://www.theregister.co.uk/2015/02/25/gemalto_spooks_popped_our_lans_not_sim_keys/
https://www.wired.com/2015/02/gemalto-confirms-hacked-insists-nsa-didnt-get-crypto-keys/

We are also talking about an intelligence service (NSA) which wilfully
compromised an Elliptic Curve cryptographic standard:

https://en.wikipedia.org/wiki/Dual_EC_DRBG

This compromise is a part of the NSA's Bullrun program:

https://en.wikipedia.org/wiki/Bullrun_program

|As part of Bullrun, NSA has also been actively working to "Insert
|vulnerabilities into commercial encryption systems, IT systems, networks, and
|endpoint communications devices used by targets".[10] The New York Times has
|reported that the random number generator Dual_EC_DRBG contains a back door,
|which would allow the NSA to break encryption keys generated by the random
|number generator.

This _is_ a real threat and I am surprised that anyone could think
otherwise after reading the Snowden documents.

[Arne Vajhøj]

On 10/19/2017 8:33 AM, Simon Clubley wrote:
> On 2017-10-18, Arne Vajhøj <ar...@vajhoej.dk> wrote:
>> On 10/17/2017 1:29 PM, Simon Clubley wrote:
>>>
>>> It's not a code generator but a hardware based random number generator
>>> which is the concern here.
>>>
>>> Some reading for those who do not understand the concerns:
>>>
>>> https://en.wikipedia.org/wiki/RdRand#Reception
>>
>> Somehow I think this type of concern actually reduces
>> internet security.
>>
>> There are so many real threats that can and should
>> be protected against.
>
> We are talking about an intelligence service (GCHQ) that once tried to
> compromise Gemalto. _Gemalto_ for goodness sake. :-( For a reminder:
>
> https://www.theregister.co.uk/2015/02/25/gemalto_spooks_popped_our_lans_not_sim_keys/
> https://www.wired.com/2015/02/gemalto-confirms-hacked-insists-nsa-didnt-get-crypto-keys/
>
> We are also talking about an intelligence service (NSA) which wilfully
> compromised an Elliptic Curve cryptographic standard:
>
> https://en.wikipedia.org/wiki/Dual_EC_DRBG

If you actually read these, then you will see that it is a bunch
of "possible", "maybe" and "likely".

> This compromise is a part of the NSA's Bullrun program:
>
> https://en.wikipedia.org/wiki/Bullrun_program

According to a lot of rumors and hearsay.

> |As part of Bullrun, NSA has also been actively working to "Insert
> |vulnerabilities into commercial encryption systems, IT systems, networks, and
> |endpoint communications devices used by targets".[10] The New York Times has
> |reported that the random number generator Dual_EC_DRBG contains a back door,
> |which would allow the NSA to break encryption keys generated by the random
> |number generator.
>
> This _is_ a real threat and I am surprised that anyone could think
> otherwise after reading the Snowden documents.

I think we may have slightly different understanding of real.

All this is purely speculation/theories. Good reason for suspecting
some capabilities. Some reasons to suspect specific capabilities.
But nothing certain about what they can or what they have done.

The fact that we may not know about any breaches that way does
obviously not prove that it did not happen.

But we all know case after case of specific data stolen from
specific companies due to less "exotic" reasons. Equifax and
thousands other.

Unless you have unlimited funding for your security work, then
I will recommend a strong focus on the most critical threats.

And all the "what if NSA ..." theories does not seem that
critical.

Arne

[David Froble]

I guess part of the question is, will the NSA sell credit card info, steal your
money?

I'm not saying either way, but, if they are after the bad guys, then perhaps
they are not such a threat to others.

Wonder if they are working for back doors into quantum communications?

[Craig A. Berry]

On 10/19/17 8:46 PM, David Froble wrote:

> Arne Vajhøj wrote:

>> Unless you have unlimited funding for your security work, then
>> I will recommend a strong focus on the most critical threats.
>>
>> And all the "what if NSA ..." theories does not seem that
>> critical.
>>
>> Arne
>
> I guess part of the question is, will the NSA sell credit card info,
> steal your money?
>
> I'm not saying either way, but, if they are after the bad guys, then
> perhaps they are not such a threat to others.

Except when they maintain an arsenal of exploits that fall into the
hands of the bad guys, such as the EternalBlue vulnerability in SMBv1
that was the basis of WannaCry, among others, that brought down the
British national health service, among others.

[Stephen Hoffman]

On 2017-10-20 01:12:03 +0000, Arne Vajh j said:

> If you actually read these, then you will see that it is a bunch of
> "possible", "maybe" and "likely".

Issues of platform and processor trust aside, the use multiple sources
to seed the CPRNG is sensible, if for no other reason than that bugs in
any CPRNG are very bad, irrespective of the details of the bug. Ask
Infineon and Debian and others about that.
https://www.debian.org/security/2008/dsa-1571

If for no other reason than that "VMS Software, Inc. develops, sells
and supports innovative and advanced releases of OpenVMS, the most
secure operating system on the planet."

[Simon Clubley]

On 2017-10-19, Craig A. Berry <craig...@nospam.mac.com> wrote:
>
> Except when they maintain an arsenal of exploits that fall into the
> hands of the bad guys, such as the EternalBlue vulnerability in SMBv1
> that was the basis of WannaCry, among others, that brought down the
> British national health service, among others.

And in addition, what happens when an hostile state power finds
a backdoor inserted by NSA/GCHQ and uses it to attack your country ?

[Simon Clubley]

On 2017-10-19, David Froble <da...@tsoft-inc.com> wrote:
>
> I guess part of the question is, will the NSA sell credit card info, steal your
> money?
>
> I'm not saying either way, but, if they are after the bad guys, then perhaps
> they are not such a threat to others.
>

And what about when the NSA decide they have the right to insert backdoors
into the products of American companies ?

[David Froble]

Agreed.

One would hope that vulnerabilities are found that they are reported and fixed,
not hoarded for whatever NSA would hoard such for. Seems to me they have their
priorities mixed up. If the NSA's purpose is to serve and protect the citizens
of the US, then protecting them from exploits should be their concern.
Apparently some at the NSA, or higher, have decided protecting US citizens is
not their priority.

[David Froble]

Simon Clubley wrote:
> On 2017-10-19, David Froble <da...@tsoft-inc.com> wrote:
>> I guess part of the question is, will the NSA sell credit card info, steal your
>> money?
>>
>> I'm not saying either way, but, if they are after the bad guys, then perhaps
>> they are not such a threat to others.
>>
>
> And what about when the NSA decide they have the right to insert backdoors
> into the products of American companies ?
>
> Simon.
>

Vote "Dave for president", and things will change, for the better ...

:-)