Best way to pass c++ pointer to tcl
thibault.daoulas
I have read several topics around this, but quite old and different.
That's why I would like an "updated" piece of advice about my issue :
I have in my c++ code a pointer to u_char's (u_char *), and I want
this to be reachable from my Tcl/Tk code. Everything is pretty
straightforward with simple types like int, long, etc., but I do not
see clearly how this can be efficiently done for u_char *, or whatever
c++ pointer.
I was thinking about creating a new tcl command, like "$d getdata"
which would return the u_char* corresponding to my variable $d, via a
tcl.result()-like method in my c++ class, but it does not seem to be
as easy as I thought.
Do you have a clue about this?
Cheers,
Thibault.
Georgios Petasis
You should take a look at swig:
It does all these automatically...
George
Alexandre Ferrieux
wrote:
> Hi everybody,
> I have read several topics around this, but quite old and different.
> That's why I would like an "updated" piece of advice about my issue :
> I have in my c++ code a pointer to u_char's (u_char *), and I want
> this to be reachable from my Tcl/Tk code.
Are you sure you need link-level coupling between C and Tcl ?
Please comment on how the suggestion below applies to your case:
-Alex
GPS
Creating and Using Tcl Handles in C Extensions:
http://wiki.tcl.tk/13881
Be aware of the problems at the end of the page with the code.
I originally used that type of code, but I've found better performance by
using integers that refer to offsets in a structure.
So my Ext_Init allocates a struct like:
struct ext {
void **structures;
int allocated;
int used;
};
I also allocate a few ext->structures to start with in Ext_Init.
Then I clean that up with Tcl_CallWhenDeleted() when the interp gets
destroyed.
The Tcl_Obj command Ext_Create is passed the struct ext * via its client
data.
The bulk of the code might look something like this:
int i;
ext->used++;
if(ext->used >= ext->allocated) {
size_t newsize = ext->allocated * 2;
ext->structures = (void *)Tcl_Realloc((void *)ext->structures,
sizeof(*(ext->structures)) * newsize);
/* Initialize the new pointers to NULL. */
for(i = ext->allocated; i < newsize; ++i) {
ext->structures[i] = NULL;
}
ext->allocated = newsize;
}
/* Now find the first free slot.
* This could be a bit smarter if the table just grew.
*/
for(i = 0; i < ext->allocated; ++i) {
if(NULL == ext->structures[i]) {
ext->structures[i] = ext_alloc_something();
Tcl_SetObjResult(interp, Tcl_NewIntObj(i));
return TCL_OK;
}
}
Tcl_Panic("invalid accounting for ext data structure");
return TCL_ERROR;
Then in void *Ext_GetSomePointer(interp, struct ext *ext, int i); that takes
an integer and returns NULL or a valid pointer, you do something like this:
void *ptr;
/*Validate the offset.*/
if(i >= 0 && i < ext->allocated && ext->structures[i]) {
ptr = ext->structures[i];
return ptr;
}
Tcl_SetObjResult(interp, Tcl_ObjPrintf("invalid handle: %d", i));
return NULL;
The performance with this method is superior to a hash table, and there are
never any hash collisions to worry about, or hash attacks you might be
vulnerable to. It's also more efficient in terms of memory usage in
comparison to a Tcl hash table too. When I've explained this before, it's
fallen on deaf ears. I hope this makes more sense today to the rest of the
Tcl community.
You can also have the pretty "file#%d" type of descriptors quite easily with
a bit of string manipulation, while retaining the O(1) behavior.
-George
thibault.daoulas
Thanks everyone, I will give some feedback as soon as I get something
from your advice. I also saw that the tclX extension could do some
great things about this. I will let you know anyway..
Thibault.
thibault.daoulas
wrote:
Okay, so the first thing is that using SWIG does not suit me, because
the (u_char *) that I want to be reachable from tcl is a frame that is
updated around 15 times per second...
Then, Alexandre, I am not quite aware about if I need a link-level
coupling, but now that you know better my situation, maybe you are
able to tell me ?
And finally, George, it seems that the Tcl Handles would suit indeed
to my case. But one question, maybe a bit stupid, but how do I consult
the handle from the tcl interface?
Thanks again for your help,
Thibault.
Bezoar
I wrote an example using SWIG just to get to the elements in a
structure. see
http://wiki.tcl.tk/17047
Carl
Alexandre Ferrieux
wrote:
>
> Then, Alexandre, I am not quite aware about if I need a link-level
> coupling, but now that you know better my situation, maybe you are
> able to tell me ?
Fifteen updates per second is a rate that is within easy reach of the
script level nowadays.
So, the idea is to leave your C part completely inside a separate
process, and do I/O with it from the Tcl side.
For example, if the C part is the interface to some device or
complicated network protocol, and just outputs updates periodically,
you can simply hook a fileevent to its output and read updates as they
come.
If it is more interactive and also needs stimuli from the script, you
can also write to its input.
Bottom line: going to the details of the C-Tcl API is something that
should only be done when "process-level coupling" is too slow for the
job.
-Alex
thibault.daoulas
wrote:
If I understand well, the fileevent makes sense when it comes to files
or network sockets, correct me if I'm wrong. In my case I implemented
a decoder in C, and now I want to implement a renderer that would only
get the pointer to that video data after it has been decompressed by
the codec. The tcl handles seem to do the job, even easier with tclX,
but the point for me (as a newbie, as you may have guessed..) is that
I do not manage to get this pointer from tcl. I indeed create in C and
get from tcl the string representation of my video data, but from tcl,
should I be able to manipulate this string (let's say Video0) directly
as a pointer?
Sorry for maybe obvious questions, but reading docs and newsgroups I
wasn't able to solve my issue.
Otherwise, I am not quite in favor for using swig, since it seems to
cause some linking problems, and that I'm not sure if it suits to a
very frequently updated data (once more, correct me if I'm wrong...).
Alexandre Ferrieux
wrote:
> On Nov 18, 8:30 pm, Alexandre Ferrieux <alexandre.ferri...@gmail.com>
> wrote:
>
>
>
> > On Nov 18, 5:08 am, "thibault.daoulas" <thibault.daou...@gmail.com>
> > wrote:
>
> > > Then, Alexandre, I am not quite aware about if I need a link-level
> > > coupling, but now that you know better my situation, maybe you are
> > > able to tell me ?
>
> > Fifteen updates per second is a rate that is within easy reach of the
> > script level nowadays.
> > So, the idea is to leave your C part completely inside a separate
> > process, and do I/O with it from the Tcl side.
> > For example, if the C part is the interface to some device or
> > complicated network protocol, and just outputs updates periodically,
> > you can simply hook a fileevent to its output and read updates as they
> > come.
> > If it is more interactive and also needs stimuli from the script, you
> > can also write to its input.
>
> > Bottom line: going to the details of the C-Tcl API is something that
> > should only be done when "process-level coupling" is too slow for the
> > job.
>
> > -Alex
>
> If I understand well, the fileevent makes sense when it comes to files
> or network sockets, correct me if I'm wrong.
Not files, pipes.
> In my case I implemented
> a decoder in C, and now I want to implement a renderer that would only
> get the pointer to that video data after it has been decompressed by
> the codec.
Don't talk about "pointers", talk about what data must cross the
boundaries, and how it is used on either side.
If your decoder is put inside a child process writing to a pipe, it
can simply write binary data chunks (like frames or GOPs or RTP
packets, you choose), that Tcl will happily read from the pipe
whenever they are available. Then what is Tcl supposed to do with the
frame to display it ? If it is a Jpeg image it could be directly
rendered as a Tk image. Otherwise more work may be due. Please give
some details.
-Alex
thibault.daoulas
Thanks for the correction, it makes me think closer to the right way!
What my decoder gives as an output is a (u_char *) which represents
the data of each frame stored in YUV format. I do not want to render
it directly, but get this raw data through tcl and use it as a texture
on a tcl3d primitive. Does it make sense?
Thibault.
Alexandre Ferrieux
wrote:
>
> > Don't talk about "pointers", talk about what data must cross the
> > boundaries, and how it is used on either side.
> > If your decoder is put inside a child process writing to a pipe, it
> > can simply write binary data chunks (like frames or GOPs or RTP
> > packets, you choose), that Tcl will happily read from the pipe
> > whenever they are available. Then what is Tcl supposed to do with the
> > frame to display it ? If it is a Jpeg image it could be directly
> > rendered as a Tk image. Otherwise more work may be due. Please give
> > some details.
>
> What my decoder gives as an output is a (u_char *) which represents
> the data of each frame stored in YUV format. I do not want to render
> it directly, but get this raw data through tcl and use it as a texture
> on a tcl3d primitive. Does it make sense?
You bet it does :-)
It turns out Tcl3D's script-level API uses Tk photo images for
textures, with functions like [tcl3dVectorFromPhoto].
So, the best path for you would be
package require Img
# assume $ch is the pipe from C process
# and $len is the length of frame data
# (known in advance or transmitted)
set data [read $ch $len]
set im [image create photo -format XXX -data $data]
set tex [tcl3dVectorFromPhoto $im]
...
glTexImage2D GL_TEXTURE_2D .... $tex
Note that above XXX is the name of one of the formats at the
intersection of what Img understands and what your decoder can
produce.
I am not aware of a format in YUV colorspace in today's Img, but
correct me if I wrong.
But there are several in RGB, so maybe you could make a slight effort
to generate RGB instead (ffmpeg has very fast colorspace conversion
functions).
See how easy it is to stay _away_ from the C-Tcl API ?
-Alex
thibault.daoulas
wrote:
Actually I thought that instead of creating a photo from the data, I
could use directly the data with a tcl3dVectorFromString or
tcl3dVectorFromByteArray, so thanks for solving my problems in
advance!
The point is precisely that I do not manage to retrieve the pipe from
the C process. I do have this clear at all still, but I thought using
tclx and its handle methods would do the job, but I actually get the
string representation, and nothing more... I am sorry for what seems
to be a very basic issue since you assumed that I got it already!
Thanks a lot for your help,
Thibault.
Alexandre Ferrieux
wrote:
>
> The point is precisely that I do not manage to retrieve the pipe from
> the C process. I do have this clear at all still, but I thought using
> tclx and its handle methods would do the job, but I actually get the
> string representation, and nothing more... I am sorry for what seems
> to be a very basic issue since you assumed that I got it already!
Sorry, I am completeley confused by the above three sentences (both
syntax and semantics).
What I can do though is re-describe the architecture I'm suggesting:
(1) You have your GUI written in Tcl, with packages Img and Tcl3D.
(2) It spawns a child process running a C program.
(3) This C program writes binary frames to its stdout
(4) The Tcl script picks them up through [fileevent] and [read]
(5) It then passes the data to Tcl3D through [image -data]
Please tell me on which of 1-5 you need help.
-Alex
thibault.daoulas
wrote:
> On Nov 19, 9:33 am, "thibault.daoulas" <thibault.daou...@gmail.com>
> wrote:
>
>
>
> > The point is precisely that I do not manage to retrieve the pipe from
> > the C process. I do have this clear at all still, but I thought using
> > tclx and its handle methods would do the job, but I actually get the
> > string representation, and nothing more... I am sorry for what seems
> > to be a very basic issue since you assumed that I got it already!
>
> Sorry, I am completeley confused by the above three sentences (both
> syntax and semantics).
>
> What I can do though is re-describe the architecture I'm suggesting:
>
> (2) It spawns a child process running a C program.
> (3) This C program writes binary frames to its stdout
> (5) It then passes the data to Tcl3D through [image -data]
>
> Please tell me on which of 1-5 you need help.
>
> -Alex
Oh, sorry, (and forgot a "not" : "I do not have this clear at
all"... )
I just meant that I am not able to get the output from the C method
needed, transfered to tcl. So, in the cases you're suggesting, it
would fit in (4). Thanks for making it simple, with numbers, we all
speak the same language :).
Thibault.
Alexandre Ferrieux
wrote:
Here it goes:
# This is (2) but better be sure we get it right ;-)
# spawn the child and prepare for binary IO
set pi [open "|mydecoder arg arg ..." r]
fconfigure $pi -translation binary
# This is (4):
# register a fileevent handler
fileevent $pi readable [list gotframe $pi]
proc gotframe ch {
set data [read $ch SOME_FIXED_LENGTH]
$::im configure -data $data -format XXX
set tex [tcl3dVectorFromPhoto $im]
...
glTexImage2D GL_TEXTURE_2D .... $tex
...
$tex delete
}
Note that above I'm assuming you're reading frames of
SOME_FIXED_LENGTH (constant and known in advance).
This makes sense for decoded frames, but for genericity (if you need
to use the method in a different context), you can easily add
"packetizing" in your pipe. My favourite is just to alternate text and
binary, text giving the number of bytes of binary:
/* C side */
fprintf(stdout,"%d\n",len);
fwrite(buf,1,len,stdout);
fflush(stdout);
# Tcl side
proc gotframe ch {
gets $ch len
set data [read $ch $len]
...
}
-Alex
thibault.daoulas
Quite impressive! I did not expect that much, I am sincerely grateful!
I going to have a deeper look tomorrow morning, but I have just
another point I do not really get. I have indeed the GUI written in
tcl, but this is called by the main C process, which manages all the
"technical" aspects such as decoding video data and rendering. What I
do not get, mainly, is how the fileevent works to know in which
channelId to read, since the video data is generated by one of the C
methods. That's why I am a bit confused by your point (2).. But things
are getting clearer and clearer!
Alexandre Ferrieux
wrote:
>
> Quite impressive! I did not expect that much, I am sincerely grateful!
> I going to have a deeper look tomorrow morning, but I have just
> another point I do not really get. I have indeed the GUI written in
> tcl, but this is called by the main C process, which manages all the
> "technical" aspects such as decoding video data and rendering. What I
> do not get, mainly, is how the fileevent works to know in which
> channelId to read, since the video data is generated by one of the C
> methods. That's why I am a bit confused by your point (2).. But things
> are getting clearer and clearer!
The key is I'm proposing to replace your current architecture with a
new one.
Your current architecture is called "embedding" in Tcl circles: that
means a C app linking with libtcl.{so,dll}, with the C being the boss.
That was indeed the spirit of softawre-building-with-Tcl in the very
first day of its existence. However, simpler alternatives have emerged
since then.
What I'm proposing is "extending", "external process" variety (the
other variety being "loadable extension"): the process interpreting
the Tcl scripts is just the vanilla tclsh or wish, *not* one you
wrote. The scripted part *is* the boss. Then, when something is not
provided by the core Tcl, it uses its "extension" to implement it.
Here this means spawning an external process and doing I/O with it
(just like you would do with a webservice). In your case (or at least
my simplified understanding of it) it is even simpler since the Tcl
side just needs to *read*, not write anything to it.
In yet other words: the child process spawned in my code above via
[open "|..."] is (a) the *only* C code you have to write in the whole
app and (b) completely agnostic of Tcl or Tcl-C APIs. It can even be
tested without Tcl, or run on another machine with a different OS. Its
only contract here is to write the frames to its output in a timely
fashion.
Is that clear ?
-Alex
thibault.daoulas
wrote:
Ok, I think I understood what you mean. If I'm right, I create this
new tcl command, which runs the C process that just gets and returns
the frame data periodically.And that's where the fileevent and read,
previously prepared from tcl, get the data. Maybe it's not the correct
terms, but is it the idea?
Oh, and yes I will try to generalize it for variable length data
frames, in a second time..
Thanks!
Thibault.
Paul@Tcl3D
If you can transform your C main program into a library, it would be
quite easy to get the decoded video data directly into an OpenGL texture.
Look at the function tcl3dReadRedBookImage in file
tcl3dDemoUtil/tcl3dReadRedBookImg.c as an example. It loads an image
file (already in ready-to-use OpenGL format) and returns a GLubyte pointer:
GLubyte* tcl3dReadRedBookImage (...)
This would be your function returning the pointer to the video data.
Then add the include file into swigfiles/demoutil.i.
Recompile Tcl3D and you can use your function in Tcl3D like if you would
have called tcl3dVectorFromPhoto to convert a Tk photo into a tcl3dVector.
Note that this implies that your decoded image data should be stored in
a format understandable by OpenGL.
See function glTexImage2D at http://www.opengl.org/sdk/docs/man/
Interesseting parameters for you are "format" and "type".
Greetings,
Paul
Alexandre Ferrieux
wrote:
No "new tcl command", just use [open] to create the pipe and launch
the C process.
Then [fileevent] prepares the callback, and you're ready to drop to
the event loop.
There, every time the C program writes a frame, the callback is woken
up, does the [read], and stores the data in a Tk image for use by
Tcl3D.
If you have problems setting all this up, post the code so far.
-Alex
thibault.daoulas
wrote:
> On Nov 19, 9:32 pm, "thibault.daoulas" <thibault.daou...@gmail.com>
> wrote:
>
>
>
> > On Nov 20, 2:25 am, Alexandre Ferrieux <alexandre.ferri...@gmail.com>
> > wrote:
>
> > > On Nov 19, 12:20 pm, "thibault.daoulas" <thibault.daou...@gmail.com>
> > > wrote:
>
> > > > Quite impressive! I did not expect that much, I am sincerely grateful!
> > > > I going to have a deeper look tomorrow morning, but I have just
> > > > another point I do not really get. I have indeed the GUI written in
> > > >tcl, but this is called by the main C process, which manages all the
> > > > "technical" aspects such as decoding video data and rendering. What I
> > > > do not get, mainly, is how the fileevent works to know in which
> > > > channelId to read, since the video data is generated by one of the C
> > > > methods. That's why I am a bit confused by your point (2).. But things
> > > > are getting clearer and clearer!
>
> > > The key is I'm proposing to replace your current architecture with a
> > > new one.
>
> > > means a C app linking with libtcl.{so,dll}, with the C being the boss.
> > > first day of its existence. However, simpler alternatives have emerged
> > > since then.
>
> > > What I'm proposing is "extending", "external process" variety (the
> > > other variety being "loadable extension"): the process interpreting
> > > wrote. The scripted part *is* the boss. Then, when something is not
> > > Here this means spawning an external process and doing I/O with it
> > > (just like you would do with a webservice). In your case (or at least
> > > my simplified understanding of it) it is even simpler since theTcl
> > > side just needs to *read*, not write anything to it.
>
> > > In yet other words: the child process spawned in my code above via
> > > [open "|..."] is (a) the *only* C code you have to write in the whole
> > > tested withoutTcl, or run on another machine with a different OS. Its
> > > only contract here is to write the frames to its output in a timely
> > > fashion.
>
> > > Is that clear ?
>
> > > -Alex
>
> > Ok, I think I understood what you mean. If I'm right, I create this
> > the frame data periodically.And that's where the fileevent and read,
> > terms, but is it the idea?
> > Oh, and yes I will try to generalize it for variable length data
> > frames, in a second time..
>
> the C process.
> Then [fileevent] prepares the callback, and you're ready to drop to
> the event loop.
> There, every time the C program writes a frame, the callback is woken
> up, does the [read], and stores the data in a Tk image for use by
> Tcl3D.
>
> If you have problems setting all this up, post the code so far.
>
> -Alex
Alex,
One question about what you suggest,
When I do :
set pi [open "|mydecoder arg arg ..." r]
I guess that mydecoder refers to an added tcl command via c++ side,
that gets and returns the data, am I right?
Bruce
>
>
> It's a bit strange, it seems that my post doesn't show up... Another
> try :
Actually, I see it 4 times ;)
> when you mention
> set pi [open "|mydecoder arg arg ..." r]
> I guess mydecoder is an extended tcl command built via c++, which gets
> and returns the frame data right ?
>
> Thibault.
no mydecoder is a stand alone program that is run and the output of that
program is read into the "pi" file descriptor
Bruce
thibault.daoulas
Hi Bruce, thanks for the reply, I finally saw my post 4 times as
well !..but have the feeling that I deleted it 5 times...so I may have
removed a post that wasn't mine...
thibault.daoulas
wrote:
The point with the open command, is that it gets the output from a
main
C process, whereas my main C process does not return the frame data
that I need. It's one of the methods in one of the C classes
that gets and manipulates the frame data. That's
why ideally in the tcl code, I would have something like :
set pi [open "|$renderer getframe" r]]
where $renderer is my current renderer, and getframe is a tcl extended
built command via c++, in my renderer class, that just returns the
video frame data from this renderer...
Is there something basic that I am omitting, makes me confusing, and
makes me do it a bad way?
Another way I have been thinking about, is using Tcl_GlobalEval, which
executes a tcl script.
Let's say in the tcl side, I have a proc getData { dat } { ... } which
sets a global variable to the argument of the proc dat. Is it
feasible, in the C side, to call Tcl_GlovalEval(interp*, "getData
datInC"), being dataInC the pointer to the data, so that it's
transmitted as an argument to getData?
It may be completely utopian, but still, I would like to have it
clear..
Thibault.
Alexandre Ferrieux
wrote:
>
> > > no mydecoder is a stand alone program that is run and the output of that
> > > program is read into the "pi" file descriptor
> > > Bruce
>
Yes, Bruce is right :-)
> The point with the open command, is that it gets the output from a
> main
> C process, whereas my main C process does not return the frame data
> that I need. It's one of the methods in one of the C classes
> that gets and manipulates the frame data.
Hmm, say, are you a C programmer or not ?
Is your C main provided on a CD-ROM or EPROM-but-you-forgot-your-UV-
lamp ?
Seriously, I'm completely at a loss understanding why you're
systematically dodging the obvious simplest path...
First, by "C main" I mean "C/C++ main"... hope it was clear ?
Second, the skeleton for this program is:
int main(int argc,char **argv)
{
X *x;
/* various initializations */
x = Constructor_of_class_X_of_interest_(some args);
/* other preparations */
while(1)
{
uchar *f;
f = x->getframe();
write(1,f,FRAMELENGTH);
}
}
Please explain what in the above departs from what you can do...
-Alex
thibault.daoulas
wrote:
> On Nov 20, 6:11 am, "thibault.daoulas" <thibault.daou...@gmail.com>
> wrote:
>
>
>
> > > > no mydecoder is a stand alone program that is run and the output of that
> > > > program is read into the "pi" file descriptor
> > > > Bruce
>
> Yes, Bruce is right :-)
>
> > The point with the open command, is that it gets the output from a
> > main
> > C process, whereas my main C process does not return the frame data
> > that I need. It's one of the methods in one of the C classes
> > that gets and manipulates the frame data.
>
> Hmm, say, are you a C programmer or not ?
> Is your C main provided on a CD-ROM or EPROM-but-you-forgot-your-UV-
> lamp ?
> Seriously, I'm completely at a loss understanding why you're
> systematically dodging the obvious simplest path...
>
...so I guess I do not have enough knowledge yet in some cases to
realize which path is the simplest!
> First, by "C main" I mean "C/C++ main"... hope it was clear ?
> Second, the skeleton for this program is:
>
> int main(int argc,char **argv)
> {
> X *x;
> /* various initializations */
> x = Constructor_of_class_X_of_interest_(some args);
> /* other preparations */
>
> while(1)
> {
> uchar *f;
>
> f = x->getframe();
> write(1,f,FRAMELENGTH);
> }
> }
>
> Please explain what in the above departs from what you can do...
>
> -Alex
Thanks a lot for this skeleton, sincerely, hopefully I will be fine
with this, and otherwise I will have a deeper look into my c classes
and manuals ;-)
Cheers,
Thibault.
Bezoar
When you use open with a "|" you are creating a new process outside of
your current script process. What Alex is suggesting is that you
modify your C++ program to output to stdout the frame data in binary
format so the TCL script can read it. "open" does not invoke a
library command! Modify your main in your C++ program to accept
command line arguments to modify how the program behaves. When you
want the frames you can get them by "set fd [ open "|<program name>
getframes" ]" for instance. You can also put communications
parameters on the command line to open a socket or named pipe to read
the data thus leaving the stdout alone.
if { [ catch { exec <program name> -socket 2345 & } err ] != 0 }
{ ....
or
if { [ catch { exec <program name> -fifo /tmp/out[pid] & } err ] !=
0 } { ...
Point is you could open up a shared memory area or use memory mapped
files, or
any other type of InterProcess Communication (IPC) to get the
information to the tcl script. You can avoid the library part and get
several advantages. If your library locks up your GUI will become
unresponsive. With this architecture this can be avoided. And as Alex
indicated you can test both the GUI and the frame emitting program
independently.
Carl
thibault.daoulas
Ok! Thanks Carl, now I get it! What made me hesitant what modifying
the already existing main, but now I finally got it clear, I think,
and ready to go on! Thanks a lot for this advice and clearing
confusing things away ! Good to learn from you guys.
Thibault.