"shift count is too large" & little-endian issue in file asn1.cpp

[Alex Vinokur]

Hi,

I have some questions related to program from file asn1.cpp (Copyright
(c) 1999 Hewlett-Packard Company)


---------------------------
unsigned long mask;
...
mask = 0x1FF << ((8 * (sizeof(long) - 1)) - 1);
/* mask is 0xFF800000 on a big-endian machine */
---------------------------

First of all, we have got the following warning on 64-bits model
warning: shift count is too large
unsigned long mask = 0x1FF << ((8 * (sizeof(long) - 1)) - 1);
So, result of the shift may be undefined

Second. What about little-endian machine?
Should mask be equal 0xFF800000?
Should mask be equal 0?

Thanks,

Alex



#######################################
Fragment of file asn1.cpp
################ BEGIN ################

Copyright (c) 1999
Hewlett-Packard Company

[--snip--]

A S N 1. C P P

ASN encoder / decoder implementation

VERSION 2.8

DESIGN:
Peter E. Mellquist

AUTHOR:
Peter E Mellquist


LANGUAGE:
ANSI C++

OPERATING SYSTEM(S):
MS-Windows Win32
BSD UNIX

[--snip--]



/*
* asn_build_int - builds an ASN object containing an integer.
* On entry, datalength is input as the number of valid bytes
following
* "data". On exit, it is returned as the number of valid bytes
* following the end of this object.
*
* Returns a pointer to the first byte past the end
* of this object (i.e. the start of the next object).
* Returns NULL on any error.
*/
unsigned char * asn_build_int( unsigned char *data,
int *datalength,
unsigned char type,
long *intp,
int intsize)
{
/*
* ASN.1 integer ::= 0x02 asnlength byte {byte}*
*/

long integer;
unsigned long mask;

if (intsize != sizeof (long))
return NULL;
integer = *intp;
/*
* Truncate "unnecessary" bytes off of the most significant end of
this
* 2's complement integer. There should be no sequence of 9
* consecutive 1's or 0's at the most significant end of the
* integer.
*/
mask = 0x1FF << ((8 * (sizeof(long) - 1)) - 1);
/* mask is 0xFF800000 on a big-endian machine */
while((((integer & mask) == 0) || ((integer & mask) == mask))
&& intsize > 1){
intsize--;
integer <<= 8;
}
data = asn_build_header(data, datalength, type, intsize);
if (data == NULL)
return NULL;
if (*datalength < intsize)
return NULL;
*datalength -= intsize;
mask = 0xFF << (8 * (sizeof(long) - 1));
/* mask is 0xFF000000 on a big-endian machine */
while(intsize--){
*data++ = (unsigned char)((integer & mask) >> (8 * (sizeof(long) -
1)));
integer <<= 8;
}
return data;
};

################ END ################

[Frank DG1SBG]

Alex Vinokur <alex.v...@gmail.com> writes:

> Hi,
>
> I have some questions related to program from file asn1.cpp (Copyright
> (c) 1999 Hewlett-Packard Company)

A newer version of the net-snmp library (some of the source code) can be
found at

http://net-snmp.sourcearchive.com/documentation/5.3.1/group__asn1__packet__parse.html

> ---------------------------
> unsigned long mask;
> ...
> mask = 0x1FF << ((8 * (sizeof(long) - 1)) - 1);
> /* mask is 0xFF800000 on a big-endian machine */
> ---------------------------

Looking at the code there is a very significant line:

> if (intsize != sizeof (long))
> return NULL;

What does intsize say ?

> First of all, we have got the following warning on 64-bits model
> warning: shift count is too large
> unsigned long mask = 0x1FF << ((8 * (sizeof(long) - 1)) - 1);
> So, result of the shift may be undefined
>
> Second. What about little-endian machine?
> Should mask be equal 0xFF800000?
> Should mask be equal 0?

I have made myself a little test program:

#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main(int argc, const char * argv[])
{
int input = 1;
int intsize = sizeof( int );

int *intp = &input;
long integer = 0L;
unsigned long mask = 0UL;

unsigned long shiftsize = 0UL;

fprintf( stderr, "%s: Integer and Long Sizes:\n", "MASKTEST" );
fprintf( stderr, "*** Size of Integer: %ld\n", sizeof( int ));
fprintf( stderr, "*** Size of Long: %ld\n", sizeof( long ));
fprintf( stderr, "*** Size of Long Long: %ld\n", sizeof( long long ));

integer = *intp;

shiftsize = (( 8 * (sizeof( long ) -1)) -1);

fprintf( stderr, "\n*** Shift Size: %ld\n", shiftsize );

mask = 0x1FF << shiftsize;

fprintf( stderr, "*** Mask: 0x%lx\n", mask );

while((((integer & mask) == 0) || ((integer & mask) == mask))
&& intsize > 1){
intsize--;
integer <<= 8;
}

fprintf( stderr, "*** Resulting Integer: %ld/ 0x%lx\n", integer, integer );

return 0;
}


If I run this on my little endian, Mac OS X-based machine I get:

MASKTEST: Integer and Long Sizes:
*** Size of Integer: 4
*** Size of Long: 8
*** Size of Long Long: 8

*** Shift Size: 55
*** Mask: 0xffffffffff800000
*** Resulting Integer: 16777216/ 0x1000000

What does this test program say on your architecture?

Cheers
Frank