SWIG, cdata memmove - wrong number of bytes memmoved'

[scotty...@gmail.com]

I'm writing a Tcl interface to a communications library using Swig as the library generator, and I found Swig's cdata and memmove functions very useful for getting access to binary structures like data packets, mac fields in addresses and that kind of thing. But I did find a problem with the conversion between the Tcl string object and the byte array. This is caused by Swig's use of the Tcl_GetStringFromObj() and Tcl_NewStringObj().

Here is the test results. In the code I was trying to fill in the 7 byte mac field in a BACNET_ADDRESS structure with a simple single byte mac - 0x80. The results was packets going out with a 0xC2 mac address which is what lead me on the search for the source of the problem.

The Tcl code looks like this:

proc sendReadPropertyRequest {mac object instance property} {
set dest_addr [::bacnet::BACnet_Device_Address]
::bacnet::memmove [$dest_addr cget -mac] [binary decode hex [format %2.2x $mac]]
.......

After passing in a mac address of 0x80 I found later the code was pulling out a 0xC2. So I did some tests examining the byte array that is the -mac object. I also was able to reproduce the fault using a malloc'ed byte array so I used that for the test:

set p_mac [::bacnet::malloc_uint8_t 100]
puts "p_mac=$p_mac"
puts "mac_ascii=[set mac_ascii [format %2.2x $mac]]"

puts "TEST #1: binary decode hex---------------------"
puts "asciimac=[set asciistr ${mac_ascii}]"
set binmac [binary decode hex $asciistr]
puts "binmac length=[set length [string length $binmac]]"
::bacnet::memmove $p_mac $binmac
dumpbytestring $p_mac [expr [string length $asciistr] / 2]
.........

dumpbytestring just shows me the bytes in the byte array pulling them out one at a time:
proc dumpbytestring {uint8_object maxlen} {
puts " Dumping uint8_t array------------------"
for {set i 0} {$i <= $maxlen} {incr i} {
puts -nonewline " [format %2.2x $i]: "
if {$i == 0} {
set binread [::bacnet::cdata $uint8_object]
} else {
set binread [::bacnet::cdata $uint8_object $i]
}
puts -nonewline " cdata:[string length $binread] bytes "
puts "ascii=[binary encode hex $binread]"
}
}

And here is the results from the console:

p_mac=_9017b902_p_uint8_t
mac_ascii=80
TEST #1: binary decode hex---------------------
asciimac=80
binmac length=1
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2

The byte is definitely not 0x80! Three more tests followed:

# Test a longer mac string
puts "TEST #2: binary decode hex (longer)------------"
puts "asciimac=[set asciistr ${mac_ascii}1020304050]"
set binmac [binary decode hex $asciistr]
puts "binmac length=[set length [string length $binmac]]"
::bacnet::memmove $p_mac $binmac
dumpbytestring $p_mac [expr [string length $asciistr] / 2]

puts "TEST #3: binary format--------------------------"
puts "asciimac=[set asciistr ${mac_ascii}]"
set binmac [binary format H* $asciistr]
puts "binmac length=[string length $binmac]"
::bacnet::memmove $p_mac $binmac
dumpbytestring $p_mac [expr [string length $asciistr] / 2]

puts "TEST #4: string---------------------------------"
set binmac "\x80\x10\x20\x30\x40\x50"
puts "binmac ($binmac) length=[string length $binmac]"
::bacnet::memmove $p_mac $binmac
dumpbytestring $p_mac [string length $binmac]
puts "------------------------------------------------"

And the results -

TEST #2: binary decode hex (longer)------------
asciimac=801020304050
binmac length=6
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
02: cdata:1 bytes ascii=80
03: cdata:2 bytes ascii=8010
04: cdata:3 bytes ascii=801020
05: cdata:4 bytes ascii=80102030
06: cdata:5 bytes ascii=8010203040
TEST #3: binary format--------------------------
asciimac=80
binmac length=1
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
TEST #4: string---------------------------------
binmac (? 0@P) length=6
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
02: cdata:1 bytes ascii=80
03: cdata:2 bytes ascii=8010
04: cdata:3 bytes ascii=801020
05: cdata:4 bytes ascii=80102030
06: cdata:5 bytes ascii=8010203040
------------------------------------------------

So the first byte is corrupted when reading less than 2 bytes, and on reads of 2 bytes or more the result is truncated by 1 byte. I then wanted to know exactly what memmove was copying, so into the xxxx_wrap file in the memmove function I inserted (in between the #ifdef and #endif) -

arg2 = (void *)(buf2);
arg3 = (size_t)(size2);
#ifdef DEBUG_MEMMOVE_CDATA
{
char buffer[100];
sprintf (buffer, "puts \"memmove: pto=%x pfrom=%x len=%d alloc2=%d\"\n", (unsigned long)arg1, (unsigned long)arg2, arg3, alloc2);
Tcl_Eval(interp,buffer);
}
#endif
memmove(arg1,(void const *)arg2,arg3);

And then I saw this -

TEST #1: binary decode hex---------------------
asciimac=80
binmac length=1
memmove: pto=2aca830 pfrom=2926be0 len=3 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
TEST #2: binary decode hex (longer)------------
asciimac=801020304050
binmac length=6
memmove: pto=2aca830 pfrom=2a63d68 len=8 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
02: cdata:1 bytes ascii=80
03: cdata:2 bytes ascii=8010
04: cdata:3 bytes ascii=801020
05: cdata:4 bytes ascii=80102030
06: cdata:5 bytes ascii=8010203040
TEST #3: binary format--------------------------
asciimac=80
binmac length=1
memmove: pto=2aca830 pfrom=2926be0 len=3 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
TEST #4: string---------------------------------
binmac (? 0@P) length=6
memmove: pto=2aca830 pfrom=2a648a8 len=8 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=c2
01: cdata:1 bytes ascii=c2
02: cdata:1 bytes ascii=80
03: cdata:2 bytes ascii=8010
04: cdata:3 bytes ascii=801020
05: cdata:4 bytes ascii=80102030
06: cdata:5 bytes ascii=8010203040
------------------------------------------------

The "memmove: pto=...." report in each case lists the pointers and the length of the strings. In each case the string that memmove is moving is LONGER than the number of bytes!

And this longer string length is the value reported by Tcl_GetStringFromObj(...). Here is where the fault lies. And the solution I found was to replace the calls to the Tcl_xxxStringxxxx() functions in SWIG_AsCharPtrAndSize() and SWIG_FromCharPtrAndSize() with calls to Tcl_xxxByteArrayxxx() functions.

Actually I took _wrap_memmove(), _wrap_cdata() and bundled them into my *.i file along with a new typedef SWIGBYTEDATA to replace SWIGCDATA, function bytedata_void() to replace cdata_void(), SWIG_FromBytePtrAndSize() to replace SWIG_FromCharPtrAndSize() and SWIG_AsCharPtrAndSize().

That solved the problems and now memmove and cdata work properly. I wanted to patch directly into Swig but that looks way to complicated for me.

Good test results:
p_mac=_e86bc902_p_uint8_t
mac_ascii=80
TEST #1: binary decode hex---------------------
asciimac=80
binmac length=1
memmove: pto=2c96be8 pfrom=2c57fe0 len=1 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=80
01: cdata:1 bytes ascii=80
TEST #2: binary decode hex (longer)------------
asciimac=801020304050
binmac length=6
memmove: pto=2c96be8 pfrom=2c740e0 len=6 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=80
01: cdata:1 bytes ascii=80
02: cdata:2 bytes ascii=8010
03: cdata:3 bytes ascii=801020
04: cdata:4 bytes ascii=80102030
05: cdata:5 bytes ascii=8010203040
06: cdata:6 bytes ascii=801020304050
TEST #3: binary format--------------------------
asciimac=80
binmac length=1
memmove: pto=2c96be8 pfrom=2c579e0 len=1 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=80
01: cdata:1 bytes ascii=80
TEST #4: string---------------------------------
binmac (? 0@P) length=6
memmove: pto=2c96be8 pfrom=2c72040 len=6 alloc2=0
Dumping uint8_t array------------------
00: cdata:1 bytes ascii=80
01: cdata:1 bytes ascii=80
02: cdata:2 bytes ascii=8010
03: cdata:3 bytes ascii=801020
04: cdata:4 bytes ascii=80102030
05: cdata:5 bytes ascii=8010203040
06: cdata:6 bytes ascii=801020304050
------------------------------------------------


If you have problems with these functions and are struggling with a solution please email me and I'll send you a full patch to include in your *.i file.