Re: Is there any latest version available MersenneTwister.h C++ class MTRand?

[Öö Tiib]

On Friday, 14 December 2018 08:27:46 UTC+2, jman...@gmail.com wrote:
> We are using MersenneTwister.h V1.1 C++ class MTRand implemented by Richard J. Wagner.

Who are "we"?

>
> We need the clarifications on the below,
>
> 1. Is there any newer version available after MersenneTwister.h V1.1?
> 2. Is there any SV issues reported on this version?

With questions about every particular product it is better idea
to contact the authors not random people on the net. Or why
you think he put his e-mail address into header file?

> If so,
> 3. What is the alternative?
> Note: For the current being we can't move to the ++11 standard mersenne_twister_engine.

Why? When standard library feature is not available (or is defective)
on platform then next best thing is Boost.
https://www.boost.org/doc/libs/1_69_0/doc/html/boost_random/reference.html#boost_random.reference.generators

Boost version often works on C++98 compiler, is usually
well-tested and supported and resulting usage code is
easier to migrate to standard later since its interface is
close.

[Manikandan J]

Thanks for the update.

The intention is to use the generator which doesn't have any vulnerability issues.

it was last updated 9 years ago. If any vulnerability there, cannot continue with this.

[David Brown]

And you think that posting to a general C++ discussion group is the best
way to do a security audit on old libraries?

If the original code was posted as part of a live, on-going project then
look on that project's side - there will be issue lists, updated
development, etc.

If the original code was posted "as is", then you got it "as is" - and
it is /your/ responsibility to do a security audit, and fix the code if
necessary.

If neither of these suit, then you need a different random number
generator library. Boost is usually a fine starting point. Or you can
write one yourself.

[Vir Campestris]

On 14/12/2018 09:05, Manikandan J wrote:
> Thanks for the update.
>
> The intention is to use the generator which doesn't have any vulnerability issues.
>
> it was last updated 9 years ago. If any vulnerability there, cannot continue with this.

It generates random numbers.

Perhaps they aren't random enough for your needs, which is why your
concern. But it shouldn't be public facing.

Perhaps if you tell us a bit more about why you are worried we can help.

Andy

[Jorgen Grahn]

On Mon, 2018-12-17, Vir Campestris wrote:
> On 14/12/2018 09:05, Manikandan J wrote:
>> Thanks for the update.
>>
>> The intention is to use the generator which doesn't have any
>> vulnerability issues. it was last updated 9 years ago. If any
>> vulnerability there, cannot continue with this.
>
> It generates random numbers.
>
> Perhaps they aren't random enough for your needs, which is why your
> concern. But it shouldn't be public facing.

I don't understand the "public facing" part.

> Perhaps if you tell us a bit more about why you are worried we can help.

Yes. "Vulnerability" can mean many different things. I note that
https://en.wikipedia.org/wiki/Mersenne_Twister (unsurprisingly) says
the algorithm is "not cryptographically secure", so the implementation
doesn't matter in that regard.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Manfred]

Also in http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/efaq.html

>
> /Jorgen
>

[David Brown]

On 18/12/18 00:51, Jorgen Grahn wrote:
> On Mon, 2018-12-17, Vir Campestris wrote:
>> On 14/12/2018 09:05, Manikandan J wrote:
>>> Thanks for the update.
>>>
>>> The intention is to use the generator which doesn't have any
>>> vulnerability issues. it was last updated 9 years ago. If any
>>> vulnerability there, cannot continue with this.
>>
>> It generates random numbers.
>>
>> Perhaps they aren't random enough for your needs, which is why your
>> concern. But it shouldn't be public facing.
>
> I don't understand the "public facing" part.

As you say, "vulnerability" can mean many things. For software, it can
mean "vulnerable to buffer overflows, special characters, SQL injection,
malicious input data, etc." I am guessing that this is what Vir meant
here - code like this should not be in a "public facing" part of the
system, meaning you are not going to pass data to it directly from the
outside world. So if the code has a function that expects an integer
between 1 and 100000000, and a vulnerability causing a buffer overflow
if it is called with negative values, that will not matter because the
code will not be used with unchecked values.

>
>> Perhaps if you tell us a bit more about why you are worried we can help.
>
> Yes. "Vulnerability" can mean many different things. I note that
> https://en.wikipedia.org/wiki/Mersenne_Twister (unsurprisingly) says
> the algorithm is "not cryptographically secure", so the implementation
> doesn't matter in that regard.
>

True.

> /Jorgen
>

[Chris M. Thomasson]

On 12/13/2018 10:27 PM, jman...@gmail.com wrote:
> Hi,

>
> We are using MersenneTwister.h V1.1 C++ class MTRand implemented by Richard J. Wagner.
>

> We need the clarifications on the below,
>
> 1. Is there any newer version available after MersenneTwister.h V1.1?

> 2. Is there any SV issues reported on this version? If so,

> 3. What is the alternative?
> Note: For the current being we can't move to the ++11 standard mersenne_twister_engine.
>
>

> // MersenneTwister.h
> // Mersenne Twister random number generator -- a C++ class MTRand
> // Based on code by Makoto Matsumoto, Takuji Nishimura, and Shawn Cokus
> // Richard J. Wagner v1.1 28 September 2009 wag...@umich.edu
>
> // The Mersenne Twister is an algorithm for generating random numbers. It
> // was designed with consideration of the flaws in various other generators.
> // The period, 2^19937-1, and the order of equidistribution, 623 dimensions,
> // are far greater. The generator is also fast; it avoids multiplication and
> // division, and it benefits from caches and pipelines. For more information
> // see the inventors' web page at
> // http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html
>
> // Reference
> // M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-Dimensionally
> // Equidistributed Uniform Pseudo-Random Number Generator", ACM Transactions on
> // Modeling and Computer Simulation, Vol. 8, No. 1, January 1998, pp 3-30.
>
> // Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura,
> // Copyright (C) 2000 - 2009, Richard J. Wagner
> // All rights reserved.
> //

Fwiw: It really does not create "true" random numbers. Instead, it
outputs pseudo random numbers. If you are interested in a crypto safe
(e.g., wrt a good crypto safe hash, sha384?) HMAC based generation of
pseudo random numbers, look here:

http://funwithfractals.atspace.cc/ct_cipher

Here is a sample C impl with a hardcoded in-memory secret key of
"Password" and sha256 for the hash used with the HMAC:

https://groups.google.com/d/topic/comp.lang.c/a53VxN8cwkY/discussion
(please, read _all_ if interested...)

The ciphertext can actually be used for crypto safe pseudo-random numbers.

The PRNG stream would be password protected such that different
passwords will produce radically different streams. Also, the random
numbers are actually encrypting files. So, we can encrypt files created
from the output of an actual TRNG. The seed for the CSPRNG is the
password and the hash algo.

Interested?

[David Brown]

On 18/12/18 22:30, Chris M. Thomasson wrote:
> On 12/13/2018 10:27 PM, jman...@gmail.com wrote:

> Fwiw: It really does not create "true" random numbers. Instead, it
> outputs pseudo random numbers.

/All/ deterministic algorithms generate pseudo random sequences.

> If you are interested in a crypto safe
> (e.g., wrt a good crypto safe hash, sha384?) HMAC based generation of
> pseudo random numbers, look here:
>
> http://funwithfractals.atspace.cc/ct_cipher

First, /you/ are not qualified to judge what is a "crypto safe"
algorithm - you want your algorithm to be studied deeply by a whole lot
of experts looking for weak points.

Secondly, your algorithm does not generate true random numbers either -
assuming your encryption algorithm is good enough, you have a
cryptographically secure pseudo random number generator. (Of course, a
CSPRNG is usually just as good as a TRNG.)

I don't mean to disparage your work, and I don't really doubt that your
algorithm is secure - but in this field, you need serious justification
and confirmation from experts before you can make claims about being secure.

[Juha Nieminen]

Chris M. Thomasson <invalid_chr...@invalid.invalid> wrote:
> Fwiw: It really does not create "true" random numbers.

I wish people stopped saying this, because it causes confusion and
misunderstandings. When this is repeated over and over for PRNGs,
even cryptographically strong ones, the vast majority of people get
the false impression that the "randomness" of these PRNGs is somehow
"poorer" than that of a "real" random number source.

In reality, the output of a (cryptographically strong) PRNG is essentially
indistinguishable from a true source of randomness, with the exception that
the PRNG will give you the same number stream for the same initial seed.
Otherwise, however, they are indistinguishable.

If you are given two very large streams or random numbers (like millions
or even billions of numbers), one from a (cryptographically strong) PRNG
and another from a true source of randomness, there is no test you can
apply to tell for certain which one was produced by which type of
generator. (This may be the case for non-strong PRNGs, but one of the
requisites for cryptographic strength is that there really is no
disinguishable pattern nor predictability, just like true randomness.)

Thus for all intents and purposes, for pretty much any practical
application, it makes exactly zero difference that a PRNG "does not
create true random numbers". For all intents and purposes, in practice,
it does.

I wish people stopped claiming it doesn't.

[james...@alumni.caltech.edu]

On Wednesday, December 19, 2018 at 7:13:34 AM UTC-5, Juha Nieminen wrote:
> Chris M. Thomasson <invalid_chr...@invalid.invalid> wrote:
> > Fwiw: It really does not create "true" random numbers.
>
> I wish people stopped saying this, because it causes confusion and
> misunderstandings. When this is repeated over and over for PRNGs,

...

> Thus for all intents and purposes, for pretty much any practical
> application, it makes exactly zero difference that a PRNG "does not
> create true random numbers". For all intents and purposes, in practice,
> it does.
>
> I wish people stopped claiming it doesn't.

"true random numbers" are a theoretical concept; appealing to
"practicality" doesn't make the distinction disappear. The right
solution is not to pretend that pseudo-random numbers are truly random,
but rather to understand that the fact that they aren't truly random
isn't a problem for most practical purposes.

[Jorgen Grahn]

On Tue, 2018-12-18, David Brown wrote:
> On 18/12/18 00:51, Jorgen Grahn wrote:
>> On Mon, 2018-12-17, Vir Campestris wrote:
>>> On 14/12/2018 09:05, Manikandan J wrote:
>>>> Thanks for the update.
>>>>
>>>> The intention is to use the generator which doesn't have any
>>>> vulnerability issues. it was last updated 9 years ago. If any
>>>> vulnerability there, cannot continue with this.
>>>
>>> It generates random numbers.
>>>
>>> Perhaps they aren't random enough for your needs, which is why your
>>> concern. But it shouldn't be public facing.
>>
>> I don't understand the "public facing" part.
>
> As you say, "vulnerability" can mean many things. For software, it can
> mean "vulnerable to buffer overflows, special characters, SQL injection,
> malicious input data, etc." I am guessing that this is what Vir meant
> here - code like this should not be in a "public facing" part of the
> system, meaning you are not going to pass data to it directly from the
> outside world. So if the code has a function that expects an integer
> between 1 and 100000000, and a vulnerability causing a buffer overflow
> if it is called with negative values, that will not matter because the
> code will not be used with unchecked values.

I guessed something like that, but PRNGs tend to be created once, with
limited user-provided input, and then only be stepped based on
user-provided stimuli, with no actual user input. There's not a lot the
user can do to cause a buffer overflow.

(Contradicting myself: if you can predict the random number sequence,
you can perhaps overload something, just like you can misbalance a
hash table by knowing the hash function and feeding it doctored data.)

[David Brown]

Agreed.

> (Contradicting myself: if you can predict the random number sequence,
> you can perhaps overload something, just like you can misbalance a
> hash table by knowing the hash function and feeding it doctored data.)
>

That seems unlikely - PRNGs rarely have dynamic memory in the first place.

Of course, if you can predict the random number sequence, you can find
vulnerabilities in whatever you have that /uses/ the PRNG, and that is
usually much more interesting to the attacker.

[Jorgen Grahn]

On Thu, 2018-12-20, David Brown wrote:
> On 19/12/18 23:30, Jorgen Grahn wrote:

...

>> (Contradicting myself: if you can predict the random number sequence,
>> you can perhaps overload something, just like you can misbalance a
>> hash table by knowing the hash function and feeding it doctored data.)
>>
>
> That seems unlikely - PRNGs rarely have dynamic memory in the first place.
>
> Of course, if you can predict the random number sequence, you can find
> vulnerabilities in whatever you have that /uses/ the PRNG, and that is
> usually much more interesting to the attacker.

Yes, it was this latter thing I meant.

[Robert Wessel]

While true, in many cryptographic applications it's required that the
CSPRNG is sufficiently unpredictably seeded. Which just moves the
issue to the seeding mechanism

[Chris M. Thomasson]

Agreed. Fwiw, the multi-part secret key, or seed, in my encryption
scheme advises the user to create a large HMAC password derived from a TRNG.

http://funwithfractals.atspace.cc/ct_cipher

Section 2
_________________________
2. The Secret Key

Both properties of my cipher rely on a good cryptographic hash function.
HMAC is an abstract layer on top of a hash algorithm that allows for a
secret key. Let us refer to this HMAC key as SK.hmac_key from now on. We
also need to allow Alice and Bob to choose a hash to use with HMAC. Let
us refer to this as SK.hash_algo. The size of HMAC's digest in bytes is
based on the digest size of SK.hash_algo. For example, SHA-256 has
32-byte digests.

Another aspect involves prepending random bytes from a TRNG to the front
of a plaintext. Let us refer to the number of these random bytes as
SK.rand_n. Okay, the secret key SK used by Alice and Bob is comprised of:
_________________________________
SK.hmac_key = Key for HMAC.
SK.hash_algo = The Hash Algorithm.
SK.rand_n = The Number of TRNG bytes.
_________________________________

SK.hmac_key must be a cryptographically secure password, e.g. comprised
of 1024 bytes from a TRNG.

SK.hash_algo must be a cryptographically secure hash, e.g. SHA-384.

SK.rand_n must be equal to, or ideally larger than the digest size of
SK.hash_algo.
_________________________


If _all_ of those rules are followed, my HMAC cipher should produce
crypto safe streams.

[Juha Nieminen]

Robert Wessel <robert...@yahoo.com> wrote:
> While true, in many cryptographic applications it's required that the
> CSPRNG is sufficiently unpredictably seeded. Which just moves the
> issue to the seeding mechanism

Yes, but my objection to the oft-repeated "PRNGs do not produce true
random numbers" is that it easily gives the impression that the stream
of numbers produced by PRNGs are "less random", ie. of "lesser quality"
than some true source of randomness, which in turns gives the impression
that if someone wants "high-quality" random numbers, they should look
for some sort of source that produced "true random numbers" instead of
using a PRNG, even though the latter may well be more than good enough
for their application (and, quite often, thousands of times more
efficient).

[Vir Campestris]

On 21/12/2018 11:32, Juha Nieminen wrote:
> Yes, but my objection to the oft-repeated "PRNGs do not produce true
> random numbers" is that it easily gives the impression that the stream
> of numbers produced by PRNGs are "less random", ie. of "lesser quality"
> than some true source of randomness, which in turns gives the impression
> that if someone wants "high-quality" random numbers, they should look
> for some sort of source that produced "true random numbers" instead of
> using a PRNG, even though the latter may well be more than good enough
> for their application (and, quite often, thousands of times more
> efficient).

Which gets back to my question to the OP up-thread:

"Perhaps if you tell us a bit more about why you are worried we can help. "

Andy

[Chris M. Thomasson]

On 12/19/2018 12:44 AM, David Brown wrote:
> On 18/12/18 22:30, Chris M. Thomasson wrote:
>> On 12/13/2018 10:27 PM, jman...@gmail.com wrote:
>
>> Fwiw: It really does not create "true" random numbers. Instead, it
>> outputs pseudo random numbers.
>
> /All/ deterministic algorithms generate pseudo random sequences.

Agreed.

>> If you are interested in a crypto safe
>> (e.g., wrt a good crypto safe hash, sha384?) HMAC based generation of
>> pseudo random numbers, look here:
>>
>> http://funwithfractals.atspace.cc/ct_cipher
>
> First, /you/ are not qualified to judge what is a "crypto safe"
> algorithm - you want your algorithm to be studied deeply by a whole lot
> of experts looking for weak points.

Agreed. In my defense, my cipher is an abstraction around a form of HMAC
that does not mutate its internal state when taking a digest. So, if
HMAC uses a crypto-safe hash (e.g., sha-384), then it should be okay. I
mean, can you get at any part of a megabyte sized HMAC key derived from
a TRNG with a single hash digest? Fwiw, Python uses a native HMAC that
does not destroy its internal state due to the simple act of a user
taking a digest of the current state...

> Secondly, your algorithm does not generate true random numbers either -
> assuming your encryption algorithm is good enough, you have a
> cryptographically secure pseudo random number generator. (Of course, a
> CSPRNG is usually just as good as a TRNG.)

Agreed. Actually, my system encrypts a plaintext into a ciphertext. The
ciphertext is not that bad wrt programs like ent. I am thinking that
multiple ciphertexts could be used as a CSPRNG stream, without
decrypting the plaintext in mind. In this case, a plaintext can be part
of the secret key itself. Btw, have you tried to run:

https://pastebin.com/raw/feUnA3kP
(raw text, no ads)

Yet? The usage is:
__________________________
Usage: program in_file out_file mode_flag

mode_flag -e is encrypt where the in_file gets encrypted as out_file

mode_flag -d is decrypt where the in_file gets decrypted as out_file

Example:

program plaintext ciphertext -e
program ciphertext plaintext_decrypt -d
__________________________

It has a hardcoded HMAC key at "Password". ;^)

> I don't mean to disparage your work, and I don't really doubt that your
> algorithm is secure - but in this field, you need serious justification
> and confirmation from experts before you can make claims about being secure.

Agreed. It is possible to pay somebody to try to destroy it, wrt
cracking it wide open?

[Chris M. Thomasson]

Imho it can be useful and prudent, to run the stream through some tests.
A simple program that does this is ent:

http://www.fourmilab.ch/random

And there are many others.

[Chris M. Thomasson]

Fwiw, I always thought of a TRNG as a non-repeatable source of entropy.
Think of zooming in on every individual snowflake produced by a hardcore
blizzard that happened to dump 6+ feet of snow. Can we ever perfectly
repeat the conditions contained therein? Well, think of the idea that a
single flake contains uniquely self-similar patterns in and of itself...