Bitwise xor of two bytes does not work as I expected

[Mohsen Owzar]

Hi guys,
I have to check some data in the Labview programming for the RS 232 interface.
In that protocol I must build the checksum of the data to be sent.

For example, for the given data:
03, 58, 43 and 30:
I have to take from left to right two data (03 and 58) and build the xor of them as shown below
0000 0011
0101 1000
========
0101 1011
Or
03 xor 58  5B
This result should be taken with the next data (43) to build the Xor and the result of them should be xored with the last data (30) as shown below:
0101 1011  5B
0100 0011  43
===============
0001 1000  18

0001 1000  18
0011 0000  30
===============
0010 1000  28

So, to check this with AWK, I've written this one liner, but this seems not to work:
awk 'BEGIN {a = ARGV[1]; DELETE ARGV[1]; a = ARGV[2]; DELETE ARGV[2]; print (xor(a, b))}' 1011 1100
1100
Instead of giving me
0101 1011 = 5B
It gives only
1100 = C
Why? Where I'm doing something wrong?

[Mack The Knife]

In article <93c55d4c-2b3c-42eb...@googlegroups.com>,

Gawk converts values as decimal. You want to use hex.
Something like

gawk 'BEGIN {
a = strtonum(ARGV[1])
b = strtonum(ARGV[2])
printf("%#x\n", xor(a, b))
}' 0x03 0x58

Mack The Knife

[Janis Papanagnou]

Probably in many places. First, please tell us what awk version you are
using that supports binary operations; is that GNU awk? Does this awk
support binary string literals (like 1011)? (You seem to xor the binary
representation of the *decimal* numbers 1011 and 1100 here.) Does this
awk support DELETE with all caps? (Using 'delete' is anyway unnecessary
here.) Then the question whether overwriting 'a' (a second time) is
intentional? (b=... is probably meant.)

PS: In what environment are you working? Doing that (here [OT]) in a
shell, ksh, might be simpler that using awk for binary base numbers.

Janis

[Mohsen Owzar]

Janis Papanagnou schrieb am Freitag, 15. Juli 2022 um 11:40:44 UTC+2:

> Probably in many places. First, please tell us what awk version you are
> using that supports binary operations; is that GNU awk? Does this awk
> support binary string literals (like 1011)? (You seem to xor the binary
> representation of the *decimal* numbers 1011 and 1100 here.) Does this
> awk support DELETE with all caps? (Using 'delete' is anyway unnecessary
> here.) Then the question whether overwriting 'a' (a second time) is
> intentional? (b=... is probably meant.)
>
> PS: In what environment are you working? Doing that (here [OT]) in a
> shell, ksh, might be simpler that using awk for binary base numbers.
>
> Janis

Hi Janis,

Excuse me for unprecise question.
I'm using GAWK under CYGWIN.
The Version of GAWK is:
m.owzar@ /cygdrive/c/cygwin/home/m.owzar/BinMo
$> gawk --version
GNU Awk 5.1.1, API: 3.1 (GNU MPFR 4.1.0, GNU MP 6.2.1)
Copyright (C) 1989, 1991-2021 Free Software Foundation.

In the manual "A User’s Guide for GNU Awk, Edition 4, June, 2011" on page 167, 168
Under "9.1.6 Bit-Manipula:ion Functions" I found:
xor(v1, v2) Return the bitwise XOR of the values provided by v1 and v2.

[Mohsen Owzar]

@ Mack the Knife
Thanks a lot for your correction of the code.
It works well.
I haven't used awk for a long tine and forgot to convert the read argument into a number.
Stupid mistake

Regards Mohsen

[Janis Papanagnou]

On 15.07.2022 12:52, Mohsen Owzar wrote:
> @ Mack the Knife
> Thanks a lot for your correction of the code.
> It works well.

And it works with binary literals as you requested in your sample?
Can you show us the GNU awk program that works with such binary
arguments, please?

Janis

[Mohsen Owzar]

Janis Papanagnou schrieb am Freitag, 15. Juli 2022 um 14:23:49 UTC+2:
> And it works with binary literals as you requested in your sample?
> Can you show us the GNU awk program that works with such binary
> arguments, please?

No, unfortunately not.
It was a mistake.
You were right with your assumption.
Thanks a lot for your hint.

Regards
Mohsen

[Mohsen Owzar]

Now to calculate all the checksums of the sent data, I have put two set of data as an example in a file "data.txt" as below:
--------------------------------------------
7F # Begin pattern
03 # Length of the coming data
58
43
30
28
06 # End pattern
7F # Begin pattern
07 # Length of the coming data
58
53
30
56
34
30
30
5E # This checksum should be calculated
06 # End pattern
--------------------------------------------
7F and 06 are the begin and end pattern of each data, which must be ignored.
My program should ignore 7F and xor 03 and 58 which has the result 28.
Then should take this result (28) and xor it with the next data (43) to get the result 18.
This result (18) after xoring with the last data (30) should give the value 28 like the next value after 30.
--------------------------------------------
And I have written this AWK code two calculate and print the checksum values of the two data sets.
--------------------------------------------
#! /usr/bin/awk -f
# Calculate_Checksum_for_Serial_Interface.awk
# 2022_07_15
#####################################################
$1 == "7F", $1 == "06" {
CheckSum = 0
if ($1 != "7F" && $1 != "06") {
Data[++N] = $0
printf "%s\n", Data[N]
CheckSum = xor(CheckSum, strtonum(Data[N]))
printf "CheckSum = %X\n", CheckSum
}
}

But somehow, I cannot get it managed.
It prints something else than I described above.
I haven't written for long time AWK scripts more.

Do you have any idea?

Regards
Mohsen

[Janis Papanagnou]

On 15.07.2022 15:21, Mohsen Owzar wrote:
> CheckSum = xor(CheckSum, strtonum(Data[N]))

>

> But somehow, I cannot get it managed.
> It prints something else than I described above.

Could it be that you need: strtonum("0x"Data[N])

Janis

[Keith Thompson]

Mohsen Owzar <mohsen...@gmail.com> writes:
[...]

> So, to check this with AWK, I've written this one liner, but this seems not to work:
> awk 'BEGIN {a = ARGV[1]; DELETE ARGV[1]; a = ARGV[2]; DELETE ARGV[2]; print (xor(a, b))}' 1011 1100
> 1100

[...]

Awk is case-sensitive. There's nothing called "DELETE". Apparently
"DELETE ARGV[1]" does nothing (I haven't taken the time to figure out
why it doesn't produce an error message). The behavior is the same
when I replace "DELETE" by a random identifier.

--
Keith Thompson (The_Other_Keith) Keith.S.T...@gmail.com
Working, but not speaking, for Philips
void Void(void) { Void(); } /* The recursive call of the void */

[Ed Morton]

On 7/15/2022 11:40 AM, Keith Thompson wrote:
> Mohsen Owzar <mohsen...@gmail.com> writes:
> [...]
>> So, to check this with AWK, I've written this one liner, but this seems not to work:
>> awk 'BEGIN {a = ARGV[1]; DELETE ARGV[1]; a = ARGV[2]; DELETE ARGV[2]; print (xor(a, b))}' 1011 1100
>> 1100
> [...]
>
> Awk is case-sensitive. There's nothing called "DELETE". Apparently
> "DELETE ARGV[1]" does nothing (I haven't taken the time to figure out
> why it doesn't produce an error message). The behavior is the same
> when I replace "DELETE" by a random identifier.
>

DELETE ARGV[1] concatenates the value of the unset variable DELETE with
the value of ARGV[1] then does nothing with the resulting string.
Perfectly valid, just useless in this context.

Ed.

[Kenny McCormack]

In article <dc99d701-8822-4a0c...@googlegroups.com>,

Mohsen Owzar <mohsen...@gmail.com> wrote:
>Now to calculate all the checksums of the sent data, I have put two set of data
>as an example in a file "data.txt" as below:
>--------------------------------------------

I think this what you're after:

Data file:

7F # Begin pattern
03 # Length of the coming data
58
43
30
28
06 # End pattern
7F # Begin pattern
07 # Length of the coming data
58
53
30
56
34
30
30
5E # This checksum should be calculated
06 # End pattern

Program:
# Run as: gawk -f thisfile datafile

# Calculate_Checksum_for_Serial_Interface.awk
# 2022_07_15
#####################################################

$1 == "7F" { flag = 1 ; CheckSum = 0 ; next }
$1 == "06" { flag = 0 ; next }
flag {
printf "%#X\t", val = strtonum("0x" $1)
CheckSum = xor(CheckSum, val)
printf "CheckSum = %#X\n", CheckSum
}

Note: I didn't bother to actually verify the numerical accuracy of the
results, because I don't really understand the underlying goal of the
exercise.

--
If you don't have faith, it's because you are reading the Bible with an
honest, truthful, real-answer seeking heart.

- Rick C Hodgin -

[Mohsen Owzar]

Kenny McCormack schrieb am Samstag, 16. Juli 2022 um 00:57:49 UTC+2:

> Note: I didn't bother to actually verify the numerical accuracy of the
> results, because I don't really understand the underlying goal of the
> exercise.

Hi all,
@ Keith and Ed
You are right.
This DELETE command does nothing as you explained.
It was my fault after having not written programs in AWK for a long time.

@ Kenny
Yes, it is. It was my intention I wanted to achieve.
I have rewritten my code and ran it on the whole data.
It has generated correct values for all datasets.

And the answer to your question about the sense of my data protocol, if you are interested in is as the following:
I have an IO-Device which controls some LEDs to turn them on or off and a camera to take some pictures of the LED spots on a screen.
The firmware in the IO-Device in the micro controller is already given by the firmware programmer.
For RS 232 interface we have to use VISA and send (write) data onto device and get back (read) the answer from the device.
These string-ins and string-outs for a command like “GetVersion” (version of the firmware in the micro controller) looks like the following:

String-in
7F 03 58 43 30 28 06
SYNC LEN X C 0 RLC EOF

String-out
7F 07 58 53 30 56 34 30 30 5E 06
SYNC LEN X S 0 V 4 0 0 RLC EOF

7F (Sync) and 06 (End Of Frame) are the boundary of each data set.
LEN is the length of the data to be sent and RLC is the checksum of the sent or read data.
It means that
28 is the checksum of four values (03, 58, 43 and 30) for string-in and
5E is the checksum of 8 values (07, 58, 53, 30, 56, 34, 30, 30)

And your corrected version of my program does this perfectly.
The only thing for getting the firmware version I have to send “XC0” as command surrounded with prefix 7F and LEN and suffix RLC and 06.
The received data from the device has the same boundary elements (prefix 7F and LEN respect. RLC and 06) with the data in between (XS0V400).
XS0 stands for operation OK and V400 is the version of the firmware.

With best regards to all of you
Mohsen

[Kpop 2GM]

if you wanna avoid have to call strtonum( ) all the time, use the -n flag at startup :

gawk -n -b -e '$++NF = sprintf("0x %.8X", xor( +$1 , +$2 ))' OFS='\n' <<< ' 0x88776655 0x311DD993 '

0x 88776655
0x 311DD993
0x B96ABFC6

[Mohsen Owzar]

Thanks a lot

Best regards
Mohsen

[Kenny McCormack]

In article <7ad10c4b-4f50-434c...@googlegroups.com>,

Note that this depends on the bash-ism of <<<
(I don't know [or care] if any other shells implement it)

I think the -n (non-decimal-data) option was sort of only partially
implemented, and then got abandoned/deprecated by the developers.
The fact that it only works on input data, not on strings contained in the
program, is kind of a deal-breaker for me.

--
The randomly chosen signature file that would have appeared here is more than 4
lines long. As such, it violates one or more Usenet RFCs. In order to remain
in compliance with said RFCs, the actual sig can be found at the following URL:
http://user.xmission.com/~gazelle/Sigs/Reaganomics

[Kpop 2GM]

he -n (non-decimal-data) option was sort of only partially
> implemented, and then got abandoned/deprecated by the developers.
> The fact that it only works on input data, not on strings contained in the
> program, is kind of a deal-breaker for me.

0xABCDEF1234567
3022415481488743
3022415481488743

0xFFFFFBBBB1111
4503598482002193
4503598482002193


# gawk profile, created Wed Jul 27 16:23:06 2022
# BEGIN rule(s)

gawk -d- -p- -n -b -e '
BEGIN {

1 OFS = ORS
1 _ = "0xABCDEF1234567"
1 print "", _, +_, strtonum(_), ""
1 __ = 0xFFFFFBBBB1111
1 print "", __, +__, strtonum(__), ""
}

_: "0xABCDEF1234567"
__: 4503598482002193

You don't need to contemplate about tradeoffs - you have simultaneous access to both.
(note that first test case in a string,
2nd is a source code constant),

The only reason Arnold tried warning people to stay away from it is because leading zeros gets octal-decoded instead, so as long as you know exactly what you're doing, it's a great feature that's one of the most underrated.

From everything I've seen so far, it works just like s2n. Using just this tiny bit of syntax (next section), you can even directly decode arbitrarily large hex input without any function calls


gawk -nMbe '$++NF = +$_'

<<< $'0xFFFFFBBBB11FFFFFBBBB1111FFFFFBBBB11FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111111\n0xABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF128888888888888888888834567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567'

0xFFFFFBBBB11FFFFFBBBB1111FFFFFBBBB11FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111FFFFFBBBB1111111

792410481646616505940015000922854762744740671457867320743901473455933528744978850062048762862883942445525635692307798461905632480501012011096400210064677766678999906102075532691817396150908877637911699936390025554616004702352634076014122288896340204968302468430833810028641892689528530396772919229032922120676191493949322801021779981817073411777529139765810172013836465217011970968434212005035841316398621556889190978730637998199560973318985274390342067388654875762277420079317807823056337881279321136369340204985282842983017647888357332662597023472432476204933821245498817882242982792948216196147421085112855567884413125558670444123899473660999012831952983059532300226833

0xABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF128888888888888888888834567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567ABCDEF1234567

147756656133133375059351666795447984278054882222533236595909293890685589866885373840348521196663726563576447415924545810526238707724997382127366903652083960255538003615196549414054013680043219084492803632778613732247026252292747211855616100560906066384901785190067087361163419350457120581150307988552198430060501148970441659114216572552701453237232433617072788364057940404735915402922707276024780803259994084711965740498550069678887837257707432535593118909370394246796395560873743559698062561227554788546397543


One more thing - standard mode gives you nothing for negative hex, but -n mode handles it just fine :

gawk -be 'BEGIN { OFS=ORS=RS RS } ($++NF = strtonum($_))^_' <<< $'-0xBBBBBCCCCCCCD'

-0xBBBBBCCCCCCCD
0

gawk -nbe 'BEGIN { OFS=ORS=RS RS } ($++NF = strtonum($_))^_' <<< $'-0xBBBBBCCCCCCCD'

-0xBBBBBCCCCCCCD
-3302640013069517

[Kpop 2GM]

> Note that this depends on the bash-ism of <<<
> (I don't know [or care] if any other shells implement it)

I use that only cuz i'm too lazy to be thumping the POSIX-Bible. Feel free to change that to command line variable assignment or echo or printf or whatever you think would make Richard Stallman happy.

[Kenny McCormack]

In article <40753535-d20a-4e26...@googlegroups.com>,

w/e

Anyway, I'm the last person you'll meet that would get hung up on any sort
of "standards compliance". If you examine my posting history, that will be
borne out. In any case, I use bash exclusively in my shell coding, and am
of the opinion that using any other shell is silly.

That said, I only included the comment quoted above because it relates to
the subsequent comment about "non-decimal-data" not working in program
code. Note that I did not bother to read your recent long post - it didn't
seem worth the trouble to try to parse it.

--
Faced with the choice between changing one's mind and proving that there is
no need to do so, almost everyone gets busy on the proof.

- John Kenneth Galbraith -

[Janis Papanagnou]

Don't worry, the prominent powerful shells (ksh, zsh, bash) all support
that here-string redirection. Users of plain POSIX shells or worse will
know how to work around it; here-documents were long existing for that
purpose with only little overhead.

Janis