convert byte array to hex string using BigInteger

[Laura Schmidt]

Hi,

I try to convert a byte array to a hex string like this:

private static String hex_encode (byte [] val)
{
BigInteger b = new BigInteger(val);

String t = b.toString(16);

return (t);
}

For a long byte array it returns a "negative" hex string, i. e. starting
with a "-" sign.

But I want just the bytes in the array converted to hex representation,
each one ranging from "00" to "FF". There should be no minus sign then.

Can you help?
Thanks!

Laura

[Eric Sosman]

// Untested:
private static String hexEncode(byte[] val) {
StringBuilder buff = new StringBuilder(2 * val.length);
for (byte b : val) {
buff.append("0123456789ABCDEF".charAt((b >> 4) & 0xF));
buff.append("0123456789ABCDEF".charAt(b & 0xF));
}
return buff.toString();
}

There are other ways, too.

--
Eric Sosman
eso...@comcast-dot-net.invalid

[rossum]

On Thu, 20 Jun 2013 14:03:21 +0200, Laura Schmidt <l...@mailinator.com>
wrote:

This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
that it expects the byte array in two's complement representation, so
a leading 1 bit is interpreted as a negative number. You need a
different constructor:

public BigInteger(int signum, byte[] magnitude)

This lets you set the sign separately.

rossum

[Laura Schmidt]

On 06/20/2013 03:09 PM, rossum wrote:

> This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
> that it expects the byte array in two's complement representation, so
> a leading 1 bit is interpreted as a negative number. You need a
> different constructor:
>
> public BigInteger(int signum, byte[] magnitude)

You are right, sorry.

Now I get a hex string without sign. But now the decode method does not
return the original bytes anymore:

private byte [] hex_decode (String val)
{
BigInteger b = new BigInteger (val,16);
byte [] t = b.toByteArray();

return (t);
}

There is no other constructor for byte arrays.
And I don't really understand why it doesn't return the original byte array.

Laura

[rossum]

On Thu, 20 Jun 2013 15:32:56 +0200, Laura Schmidt <l...@mailinator.com>
wrote:

Again, you need to look at the Javadoc. That tells you toByteArray()
returns the two's complement representation of the BigInt.

Either convert the two's complement or write your own hex_decode(), it
isn't that difficult.

As a minor point, you should follow the Java naming conventions:
hexEncode() and hexDecode(). Underscores are more C++ than Java.

rossum

[lipska the kat]

I'm not sure what all this twos compliment stuff is about
twos compliment is just a number representation scheme
There should be no need to 'convert' anything

Anyway

The following program works with the following observation

If the most significant byte is positive e.g >= 1 && <= 127 or 0x7F
the conversion works both ways

If the most significant byte is negative or 0 e.g <= 0 && >= -128 or
0x80 then the conversion works with one proviso

The contract for BigInteger#toByteArray() contains the following text

"The array will contain the minimum number of bytes required to
represent this BigInteger, including at least one sign bit"

so, if the MSB is negative there will be an additional byte in the MSG
position after calling toByteArray set to 0 which indicates that the
following is a positive number, if the MSB is positive there is no need
for an additional byte as the first bit in the MSB is 0 thereby marking
the following number as positive ... interesting, never seen this
before... apart from that it seems to work.

Try changing the byte variable a to a positive number to see the
difference.


<code compiles='yes'>

import java.math.BigInteger;

public class Converter {

private static void printBytes(byte[] val){
for(byte b : val){
System.out.print(b);
System.out.print(' ');
}
System.out.print("\n");
}


private static String hexEncode (byte[] val){
BigInteger b = new BigInteger(1, val);
String t = b.toString(16);
return t;
//Just Say No to this :-)
//return new BigInteger(1, val).toString(16);
}

private static byte[] hexDecode (String val){

BigInteger b = new BigInteger (val, 16);
byte[] t = b.toByteArray();

return t;
}

public static void main(String[] args) {
byte a = -1; //0xFF
byte b = 127; //0x7F
byte c = -128; //0x80
byte d = 64; //0x40
byte e = 16; //0x10
byte f = 99; //0x5A

byte[] val = {a, b, c, d, e, f};

printBytes(val);

String encoded = hexEncode(val);

System.out.println(encoded);

byte[] decoded = hexDecode(encoded);

printBytes(decoded);

encoded = hexEncode(decoded);

System.out.println(encoded);

}

}

</code>

lipska

--
Lipska the Kat©: Troll hunter, sandbox destroyer
and farscape dreamer of Aeryn Sun

[markspace]

On 6/20/2013 11:22 AM, lipska the kat wrote:

> public static void main(String[] args) {
> byte a = -1; //0xFF
> byte b = 127; //0x7F
> byte c = -128; //0x80
> byte d = 64; //0x40
> byte e = 16; //0x10
> byte f = 99; //0x5A

Laura, if you're still reading, please note especially this part of the
sample program. If you want to know if your idea is going to work, you
have to test it. This list of test vectors is excellent, and also an
excellent example of how to unit test a method.

You'll need to do testing like this not only for yourself, now, to
assure you that your method really works, but also for posterity, by
which I mean also yourself, but three months from now when you have
completely forgotten the details of any method you implement today.

(Or if you're like me, posterity means about two weeks from now. Or less.)

[lipska the kat]

Heh, two weeks sounds about right.

[Joerg Meier]

If you are not married to reinventing the wheel, such as by a homework
assignment or whatever, you could always use:

DatatypeConverter.printHexBinary(val);

instead.

Liebe Gruesse,
Joerg

--
Ich lese meine Emails nicht, replies to Email bleiben also leider
ungelesen.

[Lew]

lipska the kat wrote:

> Laura Schmidt wrote:
>> rossum wrote:
>>> This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
>>> that it expects the byte array in two's complement representation, so
>>> a leading 1 bit is interpreted as a negative number. You need a
>>> different constructor:
>>>
>>> public BigInteger(int signum, byte[] magnitude)
>>
>> You are right, sorry.
>>
>> Now I get a hex string without sign. But now the decode method does not
>> return the original bytes anymore:
>>
>> private byte [] hex_decode (String val)
>> {
>> BigInteger b = new BigInteger (val,16);
>> byte [] t = b.toByteArray();
>>
>> return (t);
>> }
>>
>> There is no other constructor for byte arrays.
>> And I don't really understand why it doesn't return the original byte
>> array.
>
> I'm not sure what all this twos compliment stuff is about

That's spelled "two's-complement".

> twos compliment is just a number representation scheme

And the purpose of that constructor is just to convert a representation
in that scheme to the correct number.

> There should be no need to 'convert' anything

If there isn't, then don't use that constructor.

--
Lew

[lipska the kat]

How incredibly interesting.

[Roedy Green]

On Thu, 20 Jun 2013 14:03:21 +0200, Laura Schmidt <l...@mailinator.com>
wrote, quoted or indirectly quoted someone who said :

Here is a low level direct way to convert byte[] to a hex String. It
should be quite a bit faster than using high level routines.

/**
* Fast convert a byte array to a hex string
* with possible leading zero.
* @param b array of bytes to convert to string
* @return hex representation, two chars per byte.
*/
public static String toHexString ( byte[] b )
{
StringBuilder sb = new StringBuilder( b.length * 2 );
for ( int i=0; i<b.length; i++ )
{
// look up high nibble char
sb.append( hexChar [( b[i] & 0xf0 ) >>> 4] );

// look up low nibble char
sb.append( hexChar [b[i] & 0x0f] );
}
return sb.toString();
}

/**
* table to convert a nibble to a hex char.
*/
static char[] hexChar = {
'0' , '1' , '2' , '3' ,
'4' , '5' , '6' , '7' ,
'8' , '9' , 'a' , 'b' ,
'c' , 'd' , 'e' , 'f'};
--
Roedy Green Canadian Mind Products http://mindprod.com
Getting information off the Internet is
like taking a drink from a fire hydrant.
~ Mitch Kapor 1950-11-01

[Joerg Meier]

On Fri, 21 Jun 2013 14:51:09 -0700, Roedy Green wrote:

> On Thu, 20 Jun 2013 14:03:21 +0200, Laura Schmidt <l...@mailinator.com>
> wrote, quoted or indirectly quoted someone who said :

> Here is a low level direct way to convert byte[] to a hex String. It
> should be quite a bit faster than using high level routines.

> /**
> * Fast convert a byte array to a hex string
> * with possible leading zero.
> * @param b array of bytes to convert to string
> * @return hex representation, two chars per byte.
> */
> public static String toHexString ( byte[] b )
> {
> StringBuilder sb = new StringBuilder( b.length * 2 );
> for ( int i=0; i<b.length; i++ )
> {
> // look up high nibble char
> sb.append( hexChar [( b[i] & 0xf0 ) >>> 4] );
>
> // look up low nibble char
> sb.append( hexChar [b[i] & 0x0f] );
> }
> return sb.toString();
> }

Note that this is pretty much a copy and paste version of the converter
that already comes with the JRE:

from javax.xml.bind.DatatypeConverterImpl:

public String printHexBinary(byte[] data) {
StringBuilder r = new StringBuilder(data.length * 2);
for (byte b : data) {
r.append(hexCode[(b >> 4) & 0xF]);
r.append(hexCode[(b & 0xF)]);
}
return r.toString();
}

For a reason I was unable to discern it was actually a teensy bit slower
than the Oracle implementation.

[Roedy Green]

On Sat, 22 Jun 2013 00:26:49 +0200, Joerg Meier <joerg...@arcor.de>

wrote, quoted or indirectly quoted someone who said :

>For a reason I was unable to discern it was actually a teensy bit slower
>than the Oracle implementation.

I forgot to make my static array final. That might have done it.
The corrected version is posted at http://mindprod.com/jgloss/hex.html

How did you think to look for such a method in that package by that
name?

[Roedy Green]

On Sat, 22 Jun 2013 00:26:49 +0200, Joerg Meier <joerg...@arcor.de>

wrote, quoted or indirectly quoted someone who said :

>For a reason I was unable to discern it was actually a teensy bit slower
>than the Oracle implementation.

The code looks almost identical. Probably the difference was caused by
different "warmup" times before starting to measure. You have to give
it time to apply the hotspot.

[Joerg Meier]

On Sun, 23 Jun 2013 08:49:12 -0700, Roedy Green wrote:

> On Sat, 22 Jun 2013 00:26:49 +0200, Joerg Meier <joerg...@arcor.de>
> wrote, quoted or indirectly quoted someone who said :
>>For a reason I was unable to discern it was actually a teensy bit slower
>>than the Oracle implementation.
> I forgot to make my static array final. That might have done it.
> The corrected version is posted at http://mindprod.com/jgloss/hex.html

It has not made any difference in my test case, but I will readily admit
that I only jotted down a very primitive one, so it's entirely possible
that the issue was on my end. Either way, it was only around 5-10%, so I
didn't bother to look into it too much.

> How did you think to look for such a method in that package by that
> name?

I must admit that I don't recall how I first came upon it, but ever since I
did, I retained to memory that Java "has that", and the few times I needed
it, I just used Google until I found "the one in the JRE". It's not the
most obvious of places, I agree.

[Joerg Meier]

On Sun, 23 Jun 2013 12:40:29 -0700, Roedy Green wrote:

> On Sat, 22 Jun 2013 00:26:49 +0200, Joerg Meier <joerg...@arcor.de>
> wrote, quoted or indirectly quoted someone who said :
>>For a reason I was unable to discern it was actually a teensy bit slower
>>than the Oracle implementation.
> The code looks almost identical. Probably the difference was caused by
> different "warmup" times before starting to measure. You have to give
> it time to apply the hotspot.

I looped both versions multiple times, and while times varied, yours was
consistently slower. But, like I said in my other post, not nearly by
enough as to prefer one over the other.

For what it's worth, I just noticed that my code is from OpenJDK, but I
tested the code from Oracle. Is there still a difference with these
classes, or are they identical now ? I'm not near my dev PC with the Oracle
JDK/sources right now.

[Arne Vajhøj]

On 6/20/2013 8:03 AM, Laura Schmidt wrote:

You have already received several working solutions.

I will strongly recommend you drop the idea about using
BigInteger for this.

It becomes extremely tricky to get it right - you have already
found the sign problem - next problem will be the leading zero
problem.

See below for coding examples (and I am not even sure that I got the
BigInteger hack correct).

Arne

====

import java.math.BigInteger;
import java.util.Arrays;

import javax.xml.bind.DatatypeConverter;

public class ToHex {
private static void test(byte[] ba, ByteArrayHex cvt) {
String s = cvt.encode(ba);
System.out.println(s);
byte[] ba2 = cvt.decode(s);
if(ba2.length != ba.length) {
System.out.println("Length does not match: " + ba2.length + " != " +
ba.length);
}
for(int i = 0; i < Math.min(ba2.length, ba.length); i++) {
if(ba2[i] != ba[i]) {
System.out.println("Byte " + i + " does not match: " + ba2[i] + " !=
" + ba[i]);
}
}
}
private static void testAll(byte[] ba) {
test(ba, new ManualWithInteger());
test(ba, new ManualWithString());
test(ba, new ManualWithCalc());
test(ba, new JAXBTrick());
test(ba, new BigIntegerHack());

}
public static void main(String[] args) {

testAll(new byte[] { 1, 2, 3, 4, 5 });
testAll(new byte[] { -1, -2, -3, -4, -5 });
testAll(new byte[] { 0, 0, 0 });
}
}

interface ByteArrayHex {
public String encode(byte[] ba);
public byte[] decode(String s);
}

class ManualWithInteger implements ByteArrayHex {
public String encode(byte[] ba) {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < ba.length; i++) {
sb.append(Integer.toHexString((ba[i] >> 4) & 0x0F));
sb.append(Integer.toHexString(ba[i] & 0x0F));
}
return sb.toString();
}
public byte[] decode(String s) {
int n = s.length() / 2;
byte[] res = new byte[n];
for(int i = 0; i < n; i++) {
res[i] = (byte)(Integer.parseInt(s.substring(2 * i, 2 * i + 2),
16));
}
return res;
}
}

class ManualWithString implements ByteArrayHex {
private static final String HEX = "0123456789abcdef";
public String encode(byte[] ba) {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < ba.length; i++) {
sb.append(HEX.charAt((ba[i] >> 4) & 0x0F));
sb.append(HEX.charAt(ba[i] & 0x0F));
}
return sb.toString();
}
public byte[] decode(String s) {
int n = s.length() / 2;
byte[] res = new byte[n];
for(int i = 0; i < n; i++) {
res[i] = (byte)(HEX.indexOf(s.charAt(2 * i)) << 4 |
HEX.indexOf(s.charAt(2 * i + 1)));
}
return res;
}
}

class ManualWithCalc implements ByteArrayHex {
private char encode(int v) {
if(v < 10) {
return (char)('0' + v);
} else if(v < 16) {
return (char)('a' + v - 10);
} else {
throw new RuntimeException("Invalid hex value");
}
}
public String encode(byte[] ba) {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < ba.length; i++) {
sb.append(encode((ba[i] >> 4) & 0x0F));
sb.append(encode(ba[i] & 0x0F));
}
return sb.toString();
}
private int decode(char c) {
if('0' <= c && c <= '9') {
return c - '0';
} else if('a' <= c && c <= 'f') {
return c - 'a' + 10;
} else {
throw new RuntimeException("Invalid hex value");
}
}
public byte[] decode(String s) {
int n = s.length() / 2;
byte[] res = new byte[n];
for(int i = 0; i < n; i++) {
res[i] = (byte)(decode(s.charAt(2 * i)) << 4 |
decode(s.charAt(2 * i + 1)));
}
return res;
}
}

class JAXBTrick implements ByteArrayHex {
public String encode(byte[] ba) {
return DatatypeConverter.printHexBinary(ba);
}
public byte[] decode(String s) {
return DatatypeConverter.parseHexBinary(s);
}
}

class BigIntegerHack implements ByteArrayHex {
private String leftPad(String s, char c, int totlen) {
StringBuilder sb = new StringBuilder();
for(int i = s.length(); i < totlen; i++) sb.append(c);
sb.append(s);
return sb.toString();
}
private byte[] leftTrim(byte[] ba, int len) {
return len < ba.length ? Arrays.copyOfRange(ba, ba.length - len,
ba.length) : ba;
}
private byte[] leftPad(byte[] ba, byte b, int totlen) {
byte[] res = new byte[totlen];
int padlen = totlen - ba.length;
for(int i = 0; i < padlen; i++) res[i] = b;
System.arraycopy(ba, 0, res, padlen, ba.length);
return res;
}
public String encode(byte[] ba) {
BigInteger bi = new BigInteger(1, ba);
String s = bi.toString(16);
return leftPad(s, '0', ba.length * 2);
}
public byte[] decode(String s) {
BigInteger bi = new BigInteger(s, 16);
return leftPad(leftTrim(bi.toByteArray(), s.length() / 2), (byte)0,
s.length() / 2);
}
}

[Arne Vajhøj]

On 6/20/2013 2:22 PM, lipska the kat wrote:
> On 20/06/13 14:32, Laura Schmidt wrote:
>> On 06/20/2013 03:09 PM, rossum wrote:
>>> This is a case of RTFM. The Javadocs for BigInteger(byte[]) tell you
>>> that it expects the byte array in two's complement representation, so
>>> a leading 1 bit is interpreted as a negative number. You need a
>>> different constructor:
>>>
>>> public BigInteger(int signum, byte[] magnitude)
>>
>> You are right, sorry.
>>
>> Now I get a hex string without sign. But now the decode method does not
>> return the original bytes anymore:
>>
>> private byte [] hex_decode (String val)
>> {
>> BigInteger b = new BigInteger (val,16);
>> byte [] t = b.toByteArray();
>>
>> return (t);
>> }
>>
>> There is no other constructor for byte arrays.
>> And I don't really understand why it doesn't return the original byte
>> array.
>
> I'm not sure what all this twos compliment stuff is about
> twos compliment is just a number representation scheme
> There should be no need to 'convert' anything

> The following program works with the following observation
>
> If the most significant byte is positive e.g >= 1 && <= 127 or 0x7F
> the conversion works both ways
>
> If the most significant byte is negative or 0 e.g <= 0 && >= -128 or
> 0x80 then the conversion works with one proviso
>
> The contract for BigInteger#toByteArray() contains the following text
>
> "The array will contain the minimum number of bytes required to
> represent this BigInteger, including at least one sign bit"
>
> so, if the MSB is negative there will be an additional byte in the MSG
> position after calling toByteArray set to 0 which indicates that the
> following is a positive number, if the MSB is positive there is no need
> for an additional byte as the first bit in the MSB is 0 thereby marking
> the following number as positive ... interesting, never seen this
> before... apart from that it seems to work.

You will also get into problems with leading zero bytes.

Arne

[Arne Vajhøj]

On 6/26/2013 10:12 PM, Arne Vajh�j wrote:
> On 6/20/2013 8:03 AM, Laura Schmidt wrote:
>> I try to convert a byte array to a hex string like this:
>>
>> private static String hex_encode (byte [] val)
>> {
>> BigInteger b = new BigInteger(val);
>>
>> String t = b.toString(16);
>>
>> return (t);
>> }
>>
>> For a long byte array it returns a "negative" hex string, i. e. starting
>> with a "-" sign.
>>
>> But I want just the bytes in the array converted to hex representation,
>> each one ranging from "00" to "FF". There should be no minus sign then.
>
> You have already received several working solutions.
>
> I will strongly recommend you drop the idea about using
> BigInteger for this.
>
> It becomes extremely tricky to get it right - you have already
> found the sign problem - next problem will be the leading zero
> problem.

One of the problems is that toString(radix) and toHexString()
are not the same method.

Demo:

public class SimpleHex {

public static void main(String[] args) {

for(int i = -1; i <= 1; i++) {
System.out.println(Integer.toString(i, 16));
System.out.println(Integer.toHexString(i));
}
}
}


Arne

[Laura Schmidt]

On 06/27/2013 04:12 AM, Arne Vajh�j wrote:
> On 6/20/2013 8:03 AM, Laura Schmidt wrote:

> I will strongly recommend you drop the idea about using
> BigInteger for this.

Thank you! I agree. In general, I try not to duplicate code that already
exists in standard libraries, but in this case there were too many hacks
needed when using BigInteger.

Laura

[Joerg Meier]

Thankfully, as I pointed out in
<16xd6kqk55a7s$.1w2vnmqt...@40tude.net>, you still don't need to
duplicate code that already exists.

[Stanimir Stamenkov]

Wed, 26 Jun 2013 22:12:30 -0400, /Arne Vajhøj/:

> You have already received several working solutions.
>
> I will strongly recommend you drop the idea about using
> BigInteger for this.
>
> It becomes extremely tricky to get it right - you have already
> found the sign problem - next problem will be the leading zero
> problem.
>

> [... bunch of code...]

I may be dense, but I didn't got why using BigInteger is not good?
What's wrong with the following, for example:

byte[] val;
...
String.format("%064x", new BigInteger(1, val));

I guess one could come up with a method like:

String toHexString(byte[] val, int len) {
BigInteger num = new BigInteger(1, val);
return (len > 0) ? String.format("%0" + len + "x", num)
: String.format("%x", num);
}

Doesn't it appear sufficient?

--
Stanimir

[Arne Vajhøj]

On 6/29/2013 12:10 PM, Stanimir Stamenkov wrote:
> Wed, 26 Jun 2013 22:12:30 -0400, /Arne Vajhøj/:
>
>> You have already received several working solutions.
>>
>> I will strongly recommend you drop the idea about using
>> BigInteger for this.
>>
>> It becomes extremely tricky to get it right - you have already
>> found the sign problem - next problem will be the leading zero
>> problem.
>>
>> [... bunch of code...]
>
> I may be dense, but I didn't got why using BigInteger is not good?
> What's wrong with the following, for example:
>
> byte[] val;
> ...
> String.format("%064x", new BigInteger(1, val));

You will not be able to get back to the same byte array with that.

> I guess one could come up with a method like:
>
> String toHexString(byte[] val, int len) {
> BigInteger num = new BigInteger(1, val);
> return (len > 0) ? String.format("%0" + len + "x", num)
> : String.format("%x", num);
> }
>
> Doesn't it appear sufficient?

It seems to work.

I would probably drop the len argument and use 2*val.length.

But it is still not as simple as the DatatypeConverter.

Arne

[aydin....@gmail.com]

Hi
I`m wondering, how I can implement java constructor in C++ and what private member I need ?
Thanks Aydin

[aydin....@gmail.com]

Hi
in c++ ,there is no val.length to check the size of val similar to Binint(byte [] val) in java, can anybody let me know,please,what exactly I need to do.


Regards

Aydin


On Thursday, June 20, 2013 1:03:21 PM UTC+1, Laura Schmidt wrote:

[Andreas Leitgeb]

Not sure, if I understand you correctly, but why use a BigInteger or even C++ at all?
The thread you replied to was quite long already, and I think there were a couple
of alternatives posted in it to do this rather simple task (byte[] -> hexstring)
in a loop, even without BigInteger.

Anyway, in C++ you would either pass the length of the array as a separate
argument, or create a wrapper class that packs pointer and length into one
entity (thus, mimicking Java).