Simple example Mongo C Driver - fetch/insert docs parallel

[Waitman Gobble]

Hi,

Here is a simple example using the MongoDB C Client, pthreads and libcurl to fetch documents by URL and insert into MongoDB.

If you wish to load and process multiple docs using the C programming language then this example may be interesting to you.

When I find another moment I'll expand the example to do stuff with the data.

https://github.com/creamy/parallel-proc-url-mongodb

Thank you,

Waitman Gobble

San Jose California USA

[Kyle Banker]

This is very cool.

Quick question: why do you free the bson object in line 71?

https://github.com/creamy/parallel-proc-url-mongodb/blob/master/tmo.c#L71

This doesn't appear necessary (or correct).

[Waitman Gobble]

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/bd9NDEmxljQJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com

Hi,

indeed. oops, looks like it needs an update.

Thanks,

Waitman Gobble
San Jose, California USA

[Waitman Gobble]

I made some updates.

Update 2012-04-30

remove free w/o malloc

add status check block to avoid compiler warnings

add build.sh to avoid compiler warnings about comments

add script to insert test data

remove abuff / memcopy and use buffer from curl callback

https://github.com/creamy/parallel-proc-url-mongodb

I tried with latest mongo c client from github. Since I'm running on 

$ uname -a

FreeBSD hunny.waitman.net 10.0-CURRENT FreeBSD 10.0-CURRENT #0: Fri Apr 20 20:56:16 PDT 2012     root@:/usr/obj/usr/src/sys/KEYSHIA  amd64

(it's also tested on CentOS 5.8/SMP if that makes any difference to anyone!)

on my machine i deleted env_win32.c and env_standard.c so gcc src/*.c would work.. :-) that's actually in the build.sh so definitely check that out before running. maybe there's a nicer way(?) 

[Waitman Gobble]

more updates, generating run id to associate docs to run and beginning libxml2 doc parsing code

change init.c to strip trailing newline

Update 2012-05-01

6). added curl error handling code

7). added response header store

8). added lastModifiedTime store*

9). added unique run id store

10). initial libxml2 doc parsing code

Waitman Gobble

San Jose California USA