MongoDB and Julia

[Kevin Liu]

Hi, 

I have Julia 0.3, Mongodb-osx-x86_64-3.0.4, and Mongo-c-driver-1.1.9 installed, but can't get Julia to access the Mongo Client through this 'untestable' package https://github.com/pzion/Mongo.jl, according to  http://pkg.julialang.org/. 

I have tried Lytol/Mongo.jl and the command require("Mongo.jl") can't open file Mongo.jl, or the auto-generated deps.jl. 

Is anyone having similar problems trying to make Julia work with Mongo? 

Thank you

Kevin

[Kevin Liu]

Any help would be greatly appreciated. I am even debating over the idea of contributing to the development of this package because I believe so much in the language and need to use MongoDB. 

[Stefan Karpinski]

Have you tried opening issues on the relevant packages? Most people here (myself included) won't know much about mongoDB or these packages.

[Kevin Liu]

Hey Stefan, thanks for replying. I have not opened an issue on Github's pzion/Mongo.jl. I will, and I will attempt to debug it. Thank you. Kevin

[Jacob Quinn]

You may also try Pkg.add("ODBC") if you can find a working ODBC driver for mongo. I feel like I've heard of people going this route.

-Jacob

[Kevin Liu]

Hey Jacob, thanks for the suggestion. ODBC just doesn't sound like the optimal way to go for being too generic. I am studying its implications and alternatives, but probably won't follow with ODBC. I appreciate the help. 

[Jacob Quinn]

No worries. I realize it's a bit of a square peg-round hole there.

[Kevin Liu]

This seems like a path I could take http://docs.mongodb.org/meta-driver/latest/tutorial/ which I just found out about. 

[Jeff Waller]

On Monday, July 13, 2015 at 3:27:49 AM UTC-4, Kevin Liu wrote:

Any help would be greatly appreciated. I am even debating over the idea of contributing to the development of this package because I believe so much in the language and need to use MongoDB. 

I think this is why it's untestable.  

https://travis-ci.org/pzion/Mongo.jl/jobs/54034564

Lytol/Mongo.jl looks abandoned.  It has a bunch of issues created over the past 2 years and

the last update was in 2013 the pzion repo is a fork which was updated 4 months ago, maybe

it's abandoned too and you'll have to fork.  But it's at least work contacting him.

[Kevin Liu]

Thanks Jeff, I will look into it and see if that's the case. I will review it carefully because I want the driver to run smoothly. 

In my last post, there was a typo, so I'm just pasting the source http://www.slideshare.net/NorbertoLeite/how-mongodb-drv for slide 28/29. 

[Kevin Liu]

Hi, I just spoke to Jesse Davis, the author and maintainer of the C Mongo driver, and he gave me some precious advice. The C driver passes the YAML tests. Using this driver's public API will also get me the implementation of two specs right away and for free: the Server Selection Spec and the Server Discovery and Monitoring Spec, which is probably the most complicated part about MongoDB and allows an app using it to be highly available. 

He pointed out to the Lua driver, which is a wrapper around the C driver https://github.com/10gen-labs/mongorover

He said the Julia driver won't need to test against YAML tests or the two specs mentioned. He told me to pass a connection string to the C driver from Julia, and the C driver will implement the specs. 

[cdm]

would a MongoDB JDBC driver paired with JDBC.jl

be feasible for your purposes ... ?

   https://groups.google.com/forum/#!topic/julia-users/qqOTu4XL1HI

good luck,

cdm

[Kevin Liu]

Probably not since it's oriented towards RDBMS. Thanks

[tim....@multiscalehn.com]

I have just made pull requests to pzion/LibBSON.jl and pzion/Mongo.jl to fix the driver in v0.4. Works fine for me, after adding @compats

[Kevin Liu]

Hi Tim, did it pass specs 1, 2, and part of 3? 4 hasn't been started yet. 

These new MongoDB drivers conform to published specifications.

1. Server Selection - Deciding which server to send database operations to in a MongoDB deployment.
2. Server Discovery and Monitoring - All the logic required to make a MongoDB application highly available.
3. CRUD API - The API for how we Create, Read, Update and Delete data from MongoDB.
4. Authentication - The rules for how to authenticate to MongoDB servers.

https://www.mongodb.com/blog/post/announcing-next-generation-drivers-mongodb

[Kevin Liu]

I'm sorry Tim, check this out https://github.com/10gen-labs/mongorover/issues/16

Could you share how you made it work properly? 

[tim....@multiscalehn.com]

https://github.com/pzion/LibBSON.jl/pull/4

https://github.com/pzion/Mongo.jl/pull/6

[Kevin Liu]

Thanks

[Kevin Liu]

Hi Julia Users, feel free to contribute to the Julia wrapper of the C Mongo Driver, maintained by Mongo 

https://github.com/tenthdimension/Julia-C-Mongo

This Julia wrapper is based on the Lua wrapper of the C Mongo Driver. Jesse Davis from MongoDB recommended I used it as a reference. 

[Kevin Squire]

Hi Kevin,

If you plan to make this a Julia package (and I encourage you to do so), it would be good to look at the Julia package naming conventions.  You might consider choosing a different name, generating a package skeleton, and moving the files in this repo there.  Alternatively, renaming that repo shouldn't be hard.

(This isn't mentioned explicitly there, but dashes also won't work for Julia package names.)

Cheers!

   Kevin

[Kevin Liu]

Hey Kevin, 

That's great. Thanks for the advice. On it right now. 

Cheers! 

[Kevin Liu]

Any suggestions for the name? I just want to remember this will be a wrapper around C Mongo. 

[Kevin Liu]

I'll name it CMONGO.jl

[Kevin Squire]

I would suggest CMongo.jl (when renaming this, be careful if you're on a case-insensitive filesystem).  Of the fully capitalized names in Julia, most of them imply that they are acronyms.

Suggestion: at this point, there isn't much code in the repo.  Since not many people have (publicly) responded to your posts, IMHO, it would be good to try to get a basic working system in place, and post your progress when something basic is working--this will make it easier for people to contribute.

One more thing you should look at is Clang.jl, which makes wrapping C libraries easier.

Cheers!

   Kevin

[Kevin Liu]

Thanks for the valuable advice and modesty, there is no code in the reps. I'm learning how to do this properly. From the Mongo side I have received a little bit of guidance but from the Julia side, very little. Changed it to CMongo.jl. Clang.jl will be of great help (I had bookmarked it, but didn't realize it was a wrapper). 

[Kevin Squire]

Hi Kevin,

I'm sorry to hear (and see) that you haven't gotten much help on the Julia mailing list.  It's probably just that your request hasn't fallen on the ears of anyone who has interest in both Mongo and Julia and the time to help.  I'm almost in that category, in that I have a small interest in Mongo (and a lot of interest in Julia), but only so much time.

I will say that once you get something working, the mailing list will usually be a good resource, but it works best if you have specific questions or problems (e.g., I'm trying to do this with the following code, but it's not working--what am I doing wrong) vs. general requests (e.g., asking for help wrapping mongo with a mostly empty repo).

I'm assuming this is your first major Julia project?  If so, my first suggestion is to ignore everything written below (for now), and try to get the Mongo.jl library running on a modern Julia first, and make sure you understand everything that it's doing.  Only after that would I try anything below.

----

In truth, I don't know the best way forward, but I can give you a little more information, and maybe it will help you decide. 

Clang.jl is a general framework for wrapping C libraries.  It's never necessary--any C library can be called directly using ccall.  But ccalls can be kind of clunky, so Clang.jl provides an API which very closely mimics the C API provided in some header file: functions have very similar signatures, and structs become julia types.  For a large library with lots of functions, or complicated structs with lots of members, this can make wrapping relatively easy.

While the output of Clang.jl is usually reasonably nice, it's still a very low level of abstraction.  Actually, very little abstraction, because it matches the C library, which is usually very low level, and still somewhat inconvenient in Julia (but more convenient than ccalls).  To be useful, you'll often want to add a higher level API on top of that, which adds functions and/or types that encompass or simplify the lower level calls.

It's also often the case that Clang.jl doesn't give you exactly what you need--e.g., it doesn't know how to wrap certain things, such as unions and C macros.  In those cases, you'll have to edit the output by hand, or spend some time programmatically filtering/modifying the results.  I do this a bit in the wrapper for VideoIO (and I still have to edit some files by hand at the end).

The main alternative is to wrap a subset of useful functions by hand (or simply use ccall directly, which amounts to pretty much the same thing).  This is probably closer to what the Mongo folks had in mind when they directed you to the Lua driver.  If you only need access to a few functions, or if your code is highly specialized in a way that Clang.jl has trouble with, wrapping by hand can be the way to go.  For this, you're basically writing Julia functions which ccall out to external library functions (such as those in mongo), and returns the result (or some modification thereof that matches what would normally be done in Julia).

Hopefully this was useful.  Please feel free to post back here with questions, and I (and maybe others, if the question is right) will try to answer as we have time.

Cheers!

   Kevin

[Kevin Liu]

Haha this is my first major project, period! Thanks a lot for putting in the time and effort into guiding. I come from the finance world but became interested in Julia and MongoDB for what they can do with science. It's a hand into understanding so much. Focused on Mongo.jl and will digest your comments after that. I got ahead of myself from Mongo.jl after seeing there was so much more to be done to make the stack fully functioning. Thank you! Will keep you and the community posted. Cheers mate!

[Kevin Liu]

I'm also sharing the stack design in the hopes of speeding the process. Comments are welcome. 

https://docs.google.com/spreadsheets/d/1rgqtCay8HhnVuYR4UCZi9IGa0bzjbc3VupPxRUT9DS0/edit?usp=sharing

[Kevin Squire]

Good luck!

[Kevin Liu]

Thanks! Will need it! 

[Kevin Liu]

Hey Kevin and Community, I got Pzion's Mongo.jl to work (see picture). I had to Pkg.add("Mongo") manually for it to enter the required packages list in Pkg.status(). 

Question now: i plan to sketch the technical specifications of the stack design on https://en.wikipedia.org/wiki/Unified_Modeling_Language for other developers to use. Is there any other gp modeling language you or the Julia users community would recommend instead? 

Thanks!

[Kevin Squire]

Hi, Kevin,

Great that you got Mongo.jl to work!  My suggestion would be that you play with that package a little, and try extending it by adding a method or two, fixing warnings, etc. 

Regarding modeling language, I wouldn't recommend it here. It's great that you want a complete package, and it's fine if you want to learn some modeling language. But my opinion is that modeling languages like UML are targeted at projects in OO languages with much more overhead than Julia (e.g., Java or C++) or projects which need a strong specification (e.g., government projects), and where that spec needs to be shared among people who may not need, want, or be able to talk with one another.  I don't have the impression that this project fits any of those requirements, and most people reading this list won't be familiar with them.  (If you're doing it just to learn that tool, fine.)

That doesn't mean you shouldn't be organized--it really means you shouldn't worry about designing and implementing the project all at once. 

I recommend incremental implementation. That either means 1. continue working with the Mongo.jl code, adding functionality (and tests) incrementally, or 2. wrap the library using Clang.jl, and add tests for each function incrementally. 

Anyway, just my opinion. 

Cheers,

   Kevin 

[Kevin Liu]

Hi Kevin! Thank you immensely for the advice. I read about libclang's limitations and I decided I will work on CMongo.jl and make it right and complete. Will keep the community posted. Cheers 

[Ferenc Szalma]

Kevin,

I also managed to get Pzion's Mongo.jl to work in Julia v0.3. Now, I am trying to make it work in v0.4 but getting an error message while trying to insert:

oid = insert(collection, {"name"=>"time series"})

WARNING: deprecated syntax "{a=>b, ...}" at In[36]:1. Use "Dict{Any,Any}(a=>b, ...)" instead.

LoadError: MethodError: `convert` has no method matching convert(::Type{Ptr{Void}}, ::Array{UInt8,1})
This may have arisen from a call to the constructor Ptr{Void}(...),
since type constructors fall back to convert methods.
Closest candidates are:
 call{T}(::Type{T}, ::Any)
 convert{T}(::Type{Ptr{T}}, !Matched::UInt64)
 convert{T}(::Type{Ptr{T}}, !Matched::Int64)
 ...
while loading In[36], in expression starting on line 1
 

 in insert at /Users/szalmaf/.julia/v0.4/Mongo/src/MongoCollection.jl:42

Did you try Mongo.jl in Julia v0.4? Do you have any suggestions as to how to go about getting rid of the LoadError above? It seems like a generic problem when switching from v0.3 to v0.4.

Cheers

[Tim Lebel]

Did you try running Pkg.checkout("Mongo")? I believe that I fixed this, but the maintainer may not have pushed a new tag.

[Ferenc Szalma]

Yes, I also tried running Pkg.checkout("Mongo"). I also checked that the original MongoCollection.jl had find(...), while the checked-out version has Base.find(...) methods. I also restarted the kernel to make sure it picked up the changes. 

Could you test that Mongo with the Julia v0.4 works properly on your side? It'd also be nice to get a hint on what this convert(::Type{Ptr{Void}}, ::Array{UInt8,1}) is and what's the problem with. I am pretty new to Julia.

[Ferenc Szalma]

I just tried `delete` and `find` and these two operations work. `insert` however does not work.

On Tuesday, September 1, 2015 at 3:41:27 PM UTC-4, Tim Lebel wrote:

[Ferenc Szalma]

Alright. So I had to checkout LibBSON and build it separately form Mongo, then restarting the kernel insert works in v0.4, too.

[Tim Lebel]

Sorry didn't have time to check this out yesterday... Yes, I had to fix the pointer conversion in LibBSON as well to get insert working. Wish the package maintainers would push new tags!

[Kevin Liu]

Hi Ferenc. Thanks for posting this. I won't be able to get back to you until later. Hope you get help from the community, which is great. Take care.