LCM porting, removing glib

[Cospan]

Hello,

I'm working on a project that requires marshaling. After looking at
CORBA, and Ice, I've found that LCM is perfect for my application.
Thank you for this.

I would like to communicate between two devices. one is a development
board with Android on it, it communicates through a socket to a daemon
in C, and that daemon communicates to the other board using the serial
port. My other processor is attached directly to the serial port, its
like a microcontroller with a limited amount of libraries, and no
chance of getting any sort of Linux kernel up and running on it.

I have a few questions about my situation.

if I am using the code found in root/lcm (the c runtime) can I get rid
of lcm_file.c, lcm_udp.c, lcm_eventlog.c and replace it with a new
provider lcm_serial.c (I would write this), or is there something
fundamental that I'm missing?

I can't use glib... or perhaps I can after fiddling with the
ridiculously long config file, and my stupid toolchain that only seems
to work inside my IDE, but to make things 'simpler' I decided to try
and replace all glib commands with my own.


looking at lcm.c in root/lcm directory
I'm a little new to threads but after reading some tutorials, and
other stuff I think I can start replacing some of the
g_static_rec_mutex_lock with pthread_mutex_lock, and so on, but I'm
confused as to the purpose of g_static_rec_mutex_unlock_full, and its
counterpart. I know a resource can be locked multiple times by the
same thread, so you can have 5 locks on a resource, but why?
The command g_static_rec_mutex_unlock_full returns a depth... is this
acting sort of like saving the stack pointer of the current thread, so
you can do other stuff with it, then when you done you re-establish
the depth? with g_static_rec_mutex_lock_full

can I accomplish the same ends with a counter and continuously calling
unlock until I get an error message?

Sorry about the flood of questions, thanks again.

Dave

[Cospan]

>
> looking at lcm.c in root/lcm directory
> I'm a little new to threads but after reading some tutorials, and
> other stuff I think I can start replacing some of the
> g_static_rec_mutex_lock with pthread_mutex_lock, and so on, but I'm
> confused as to the purpose of g_static_rec_mutex_unlock_full, and its
> counterpart. I know a resource can be locked multiple times by the
> same thread, so you can have 5 locks on a resource, but why?
> The command g_static_rec_mutex_unlock_full returns a depth... is this
> acting sort of like saving the stack pointer of the current thread, so
> you can do other stuff with it, then when you done you re-establish
> the depth? with g_static_rec_mutex_lock_full
>
> can I accomplish the same ends with a counter and continuously calling
> unlock until I get an error message?
>

I think I figured out this part. The mutex type is
PTHREAD_MUTEX_RECURSIVE so a lock by the same thread can be called on
it many times, since the function that this is in is buried deep
within the lcm functions it may be the case that it is locked 2, or 3
times by the functions that called it. So in order to allow the
callbacks to access the handlers_map the mutex must be completely
unlocked.

.
.
.

about right?

[Albert Huang]

Hi Dave,

I'm glad you're finding LCM useful!

You are correct about the recursive mutexes. It's actually pretty rare
that the recursive-ness of the mutexes is needed (we might not even need
them anymore) so you can try this with just normal mutexes if you like.

You are also correct in that you can remove the lcm_udp and lcm_file
code without adverse consequences if you don't need them in your
application. Additionally, you can add your own provider that has a
serial port backend.

Good luck!!

-albert

[Cospan]

Hi,

Thanks very much for your input, so far LCM is compiling for my
device, I've gotten rid of UDP, and file support, and have been using
lcm_udp.c as a reference to create my own lcm_serial.c. I am doing
pretty well so far... that is until I found this.

in lcm_udp.c there are the message headers

typedef struct _lcm2_header_short lcm2_header_short_t;
struct _lcm2_header_short {
uint32_t magic;
uint32_t msg_seqno;
};

typedef struct _lcm2_header_long lcm2_header_long_t;
struct _lcm2_header_long {
uint32_t magic;
uint32_t msg_seqno;
uint32_t msg_size;
uint32_t fragment_offset;
uint16_t fragment_no;
uint16_t fragments_in_msg;
};

With tcp, and udp the protocol will handle the message size. If I were
using one of the two, and I received a short header I would know that
I got the entire message, but with serial I don't know if I got the
entire message or just a few bytes before something got stopped,
basically I don't know if the data came into me is finished.

I was thinking of a couple of ways to approach this problem.

-set a timer for something like a millisecond, hopefully I'll get any
short messages from there (long messages are not a problem).

-because I'm not worried about message sequence, I can use uint32_t
msg_seqno; as my message size.
in my particular design I'm using a daemon in Linux to translate
serial to sockets (and vice versa) so I can communicate with a Java
application, I can modify the sequence number in the daemon.

-create a new header (I don't like this one).

-always send large packets.

Any feedback or ideas are very much appreciated.


Dave

[Cospan]

I've been thinking about this for a couple of days, and I came up with
a few ideas, this is my favorite one.

because serial is a communication protocol between two devices then I
can make a special header for it.

basically it would look like

magic number:
LCMSERIAL_MAGIC_LONG 0x4c433034 // hex repr of ascii "LC04"

typedef struct _lcm_serial_header_short lcm_serial_header_short_t;
struct _lcm_serial_header_short {
uint32_t magic;
uint32_t msg_size;
};

The benefit of this is that it would allow LCM to be extended to
smaller devices such as microcontrollers.

If you want to use the normal LCM protocol a proxy would be needed
that would change a directed communication to something like udp, a
daemon could accomplish this.


Dave

[David Moore]

Yes, that sounds like a good solution. I probably wouldn't name the
struct with "_short" since there is no "_long". The "_long" version is
only needed with udp which has a maximum packet size.

You probably also want some sort of checksum to be transmitted at the
end of your packets to ensure data integrity, since that also came "for
free" with UDP.

-David

> --
>
> You received this message because you are subscribed to the Google Groups "Lightweight Communications and Marshalling" group.
> To post to this group, send email to lcm-...@googlegroups.com.
> To unsubscribe from this group, send email to lcm-users+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/lcm-users?hl=.
>
>

[Cospan]

So far so... meh

I've since gotten a provider written for my device. Since it is a
serial port I can attach the TX to the RX, and basically send myself
messages, which has allowed me to exercise my code.

For now I've implemented the serial header, and an 8 bit checksum at
the end (to be improved). I can see the data go out, and then come
back in (success!).

I've since compiled the program lcmgen on my Linux box (I can't
compile it for my device, I don't know if this is an issue), and
created a simple structure called test_t, which has a single int16_t
called value.

here is the content of my test.lcm file:

struct test_t
{
int16_t value;
}


I've created a handler to read the data by following the c tutorial,
and listener.c, and I've reached the point where I am trying to
dispatch a handler.

in the lcm.c file I'm in the function:
lcm_dispatch_handlers (...) calling: the h->handler (buf, channel, h-
>userdata); which then calls (by a vector offset table I assume)
test_t_handler_stub(...) within the file test_t.c, this then calls
test_t_decode(...)

I get a -1 from that function, so I've started to look inside of it.
Inside this function the variable "thislen" returns a -1; It is
returned by a function called __int64_t_decode_array(...)

This is in a header file which makes it a little tricky because I'll
have to move it to a C file in order to debug it, but I think the
problem lies in "total_size" in the function __int64_t_decode_array
(...) within the file lcm_coretypes.h.

here is some stuff I've noticed:

int total_size = sizeof(int64_t) * elements;
sizeof(int64_t) = 4
elements = 1

so the total size = 4

my data size is 2 (sizeof (int16_t))

translated into this function

total size = total_size = 4
data size = maxlen = 2

the problem is in this line:

if (maxlen < total_size)
return -1;


This seems as though LCM has a minimum size limit of 64bits, is this
because lcmgen was compiled on a Ubuntu Linux box, and thus configured
to use 64bit processor?. The runtime is running in a soft processor
inside of an FPGA (32bits running at about 50Mhz, with a microkernel
(I have a scheduler, a couple of IPC possibilities, and thats about
it))Is there any suggestions for this. My hope is that I don't have to
modify any of the automatically generated files from lcmgen. I know
I'm working with LCM in an area where I don't think it was originally
designed, but I can see promise in it. Any suggestions would be
greatly appreciated.

Sincerely,

Dave

[Albert Huang]

Hi Dave,

> here is some stuff I've noticed:
>
> int total_size = sizeof(int64_t) * elements;
> sizeof(int64_t) = 4

This definitely seems to be _a_ problem. LCM is usable on 32-bit
platforms (e.g., 32-bit Linux), but the compiler must support int64_t
It looks like the compiler you're using is aliasing int64_t to int32_t,
which is why sizeof(int64_t) is appearing as 4 instead of 8. This is
messing up the LCM marshalling protocol. It might be worth checking the
compiler manual/documentation to see this support can be enabled
somehow?

> my data size is 2 (sizeof (int16_t))

Every marshalled LCM message has an 8-byte fingerprint. So, the
marshalled size of your struct is 8 + 2 = 10, not 2. Where are you
getting the 2 from?

> This seems as though LCM has a minimum size limit of 64bits, is this
> because lcmgen was compiled on a Ubuntu Linux box, and thus configured
> to use 64bit processor?. The runtime is running in a soft processor

see above.

-albert

[Cospan]

IT WORKS!!!!
Thanks very much for your help!
My brain glitched last night and for some reason I thought 64 bits
was
4 bytes. The main problem was that I called lcm_publish(...) and not
test_t_publish(...) so now the hash data can go through.
Dave

[Cospan]

Hello,

I wrote a little about porting LCM over to my design, and included
some of my files.

http://www.cospandesign.com/?p=71

Dave