Magento + NoSQL! (MongoDb)

[Colin M]

I've written a complete Magento resource library which fully integrates MongoDb as a Magento resource model which uses XML schema files (merged like config.xml files), supports advanced features like embedded objects as models, collections that are magento grid-compatible, profiling, etc.. The MongoDb module is open-source and is decently stable. Check it out: Cm_Mongo

Also, I'm using Magento (well Mage_Lite) and Cm_Mongo for a team-oriented hotel booking system. It is lacking a lot of cool features that we have planned such as Twilio integration, ElasticSearch-based search and more over the next 3-4 months. If anyone is interested in working on this project and other Magento projects with our company, please contact me!

[Christoph Petersen]

Hi Colin,

On Fri, 29 Apr 2011 14:37:07 -0700 (PDT), Colin M wrote:
> I've written a complete Magento resource library which fully
> integrates MongoDb as a Magento resource model which uses XML schema
> files (merged like config.xml files), supports advanced features like
> embedded objects as models, collections that are magento
> grid-compatible, profiling, etc.. The MongoDb module is open-source

> and is decently stable. Check it out: Cm_Mongo [1]
> [2]

this sounds very interesting. Whole NoSQL movement is interesting of
course. I can see some advantages especially for EAV structure of
Magento over classic SQL.

Do you have any real world experience with your Cm_Mongo especially
with large installations? If not I'd be happy to help setup some but I
think I would need some help migrating the data to Mongo if that's
actually possible.

>
> Also, I'm using Magento (well Mage_Lite [3]) and Cm_Mongo for a

> team-oriented hotel booking system. It is lacking a lot of cool
> features that we have planned such as Twilio integration,
> ElasticSearch-based search and more over the next 3-4 months. If
> anyone is interested in working on this project and other Magento
> projects with our company, please contact me!
>
>

Thanks & BR
Christoph

[Colin M]

Hi Christoph,

The project I'm using it for now is small with respect to number of
documents so I haven't put it through a real world "stress test" of
any sort yet. The primary goal in designing the module was really more
about rapid development since I'm pretty confident that MongoDb will
scale well as long as the data is indexed properly. The next step is
to concoct some standard convention for indexing the documents into
ElasticSearch and augmenting the collection class to use ElasticSearch
for any queries that Mongo can't handle. I think combining the search
power of ElasticSearch with MongoDb as a data store is going to be a
great combination.

Note, in case it wasn't clear, Cm_Mongo does not replace any core
Magento code. It provides a feature-complete and very familiar
interface for you to implement in your own models. It supports
embedded objects and referenced objects, addFieldToFilter
compatibility (works with Mage_Adminhtml_Block_Widget_Grid), etc.. See
here for the sample mongo.xml file which essentially defines the
schema, but like all other Magento xml files, they are merged from
each module's etc directory so they can be extended.
https://github.com/colinmollenhour/magento-mongo/blob/master/code/etc/mongo.xml

Some sample code:

$id = 1;
$foo = Mage::getModel('foo/bar')->load($id); //db.foobar.find({_id:
1});
$city = $foo->getAddress()->getCity(); // address is embedded object
$foo->op('$push','roles','contributor'); // operation pending until
save()
$foo->setStatus('active')->save(); // db.foobar.update({_id:1},{$set:
{"status":"active"},$push:{"roles":"contributor"}})
$fooCollection = Mage::getResourceModel('foo/bar_collection');
$fooCollection->addFieldToFIlter('status','eq','active');
$fooCollection->setOrder('created_at','asc');
$count = $fooCollection->count(); //
db.foobar.find({status:"active"}).sort({created_at:1}).count()

Thanks,
Colin

[kassner]

Hi Colin,

It is awesome to listen about this. I do not tested yet, but I think that is a great work.

Anyway, this port suggestion was submitted to Magento Dev Team? I meant this because the new 1.6 version installer comes with option to use another databases adapter, perhaps only MySQL was disponible yet.

Thanks,

Rafael Kassner.

[Colin M]

While the 1.6 database adapter is designed to allow different
databases to be used besides MySQL, it is only for SQL compatible
databases. Mongo is fundamentally different enough that to run Magento
on it would require major non-backwards compatible changes. This Mongo
adapter for Magento is intended for new projects, not as a drop-in
adapter for any of the existing Magento code.

Thanks,
Colin

[Max]

Hi Colin,

I'm not a PHP developer. So please bear with me. I want to deploy your
module and do some performance test to see if Magento would perform
better under huge load. Question is, are there any deployment
instructions that I could follow?

Thanks,
Max.

[Colin M]

Max and Paul,

It sounds like you guys are trying to squeeze more performance and
scalability out of Magento, but Cm_Mongo is intended only for new
modules, not as a db adapter for existing modules. Due to
fundamentally different database designs, changing Mage_Catalog to use
Cm_Mongo would be an enormous undertaking that I wouldn't even
recommend attempting. The API is familiar and largely compatible
(extends Mage_Core_Model_Abstract, Varien_Data_Collection, etc..), but
MongoDb lacks things like transactions and joins which will make it
very difficult to simply port without redesigning a lot. If you need
better performance out of Magento 1.6 as an ecommerce platform I
recommend looking into full-page caching instead.

Thanks,
Colin

On Nov 23, 10:09 pm, Paul Aan <weburni...@gmail.com> wrote:
> Hi Colin,
>
> I do appreciate your works on Mongodb and Magento. Thats pretty cool.
> for now, I m going to review Product's Tables which I will try to migrate
> most of them to Mongodb, so maybe I will need ur help in near future.
>
> God bless u all.

> > --
> > You received this message because you are subscribed to the Google Groups
> > "Magento Development List" group.
> > To post to this group, send email to magent...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > magento-deve...@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/magento-devel?hl=en.
>
> --
> Hoài Ân
> Skype: mrgluexv
> 0903744334

[tbu...@omitsis.com]

Hello Collin,

firts of all, great job.

I'm a Magento stores developer and I've just started a new project that's sent me here.

Until now all projects I've done was only about thousands of products as much, and the standard Magento was enough.

Now I'm in a new complex project.

This project requires two groups of products:

 - Some known products (thousands) that could be manage with MYSQL standard Magento.

 - Another group of products (more than 1M and growing) that I need to save in another system because of Magento limits. This products belongs to a third party business and I need to treat by side.

The finale website will have a complex filter that search users searches in both DDBB returning a combination of two queries.

So that, I've thought in tree options.

1) Put it all together in Magento MYSQL. Probably Magento runs wrong. Advantatges -> all products were catchable by Magento front-end an back-end system.

2) Put it the first group in Magento and the second in a MongoDB. I uppset about using the mongo products in front-end and back-end Magento system. I need that mongo was transparent to users and admin users. The idea is use mongo to holds the big group of products. Then when I show products on the web I can use that products too plus the Magento products.

3) Put it all together in a MongoDB using some tool like your cm_mongo. I think that this could be the best solution, but some things like a complete integration (extensions, sap integration, etc.) upsets me.

I know you'll probably need much more details of the project to choose one option. But I would like to know would you do in my case. And if it was option 3, if you think your module could helps me.

Thank you very much.

[enno...@gmail.com]

Hello!

This is an awesome idea!

I'm trying to set up the Cm_Mongo in my magento but I've a problem while setting up. I've copied the files from Cm_mongo as is said in the modman file and also copied the files from mongodb-php-odm-master to /lib/mongodb-php-odm, but when installing magento it appears he following error:

 Fatal error: Class 'Mongo_Database' not found in C:\wamp\www\app\code\community\Cm\Mongo\Model\Resource\Type\Mongo.php on line 17

Seems like Mongo.php is trying to get the class Mongo_Database from mongodb-php-odm/classes but don't finds it.

Actually, I didn't do this from modman file:

# Map file names to be autoload friendly

lib/mongodb-php-odm/classes/json.php                      lib/JSON.php

lib/mongodb-php-odm/classes/mongo/database.php            lib/Mongo/Database.php

lib/mongodb-php-odm/classes/mongo/collection.php          lib/Mongo/Collection.php

Because I don't know what means (maybe symlinks?). I'm working on Windows.

Any help would be appreciated,

Kind regards,

Pere

[Colin M]

That sounds like a really difficult project.. None of the options are good really and I think you might be better off buying the biggest server you can find and simplifying your catalog as much as possible and profiling and fixing specific MySQL problems. The problem with using MongoDb is that all of the other modules depend on Mage_Catalog and the product table so you have to modify checkout, sales, payments, url rewrites, etc.. Cm_Mongo would be only a small step towards doing this. It was really intended to be used for implementing new features, not to replace MySQL for existing features. I did start some work on an EAV-replacement in Cm_Mongo but it is not even in an alpha stage.

[Colin M]

The mongodb-php-odm repo doesn't use the proper file naming conventions to work with the autoloader so I use modman to symlink to the files with auto-loader friendly paths. You can just copy the files using the same mapping in Windows and it should work just fine.

[tbu...@omitsis.com]

Thank you Colin.

According with your point of view, what about keeps the mysql db for the rest on the Magento store (transactions, etc.) and only use Mongo for my catalog?

My only problem is the big number of sku's to manage, nothing else. So, I only need some way to holds this.

Best regards.

[Colin Mollenhour]

There is just a *lot* of Magento code that expects Mage_Catalog_Model_Product and Mage_Catalog_Model_Resource_Product_Collection to be a model for a MySQL database and changing everything to work with MongoDb instead will be a huge task. If I were to try this I would probably leave the models themselves alone and disable all of the existing indexers (flat, price, stock, etc..) and write my own indexers that copied everything into MongoDb. Then I'd rewrite the frontend code to use the MongoDb backed models for the catalog but still use the MySQL models for checkout. In the end I don't know if it would really be that much more efficient or not because I don't know anything about your catalog or where your performance problems lies. MySQL is a very capable database so don't just give up on it entirely; it *can* handle millions of rows but the current indexers in CE are just terribly inefficient (a little bird told me this should be fixed in 1.9).

Colin

tbu...@omitsis.com

Tuesday, May 21, 2013 10:30 AM

You received this message because you are subscribed to a topic in the Google Groups "Magento Development List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/magento-devel/oRTEtYSEExU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to magento-deve...@googlegroups.com.

To post to this group, send email to magent...@googlegroups.com.

Visit this group at http://groups.google.com/group/magento-devel?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Colin M

Tuesday, May 21, 2013 10:16 AM

The mongodb-php-odm repo doesn't use the proper file naming conventions to work with the autoloader so I use modman to symlink to the files with auto-loader friendly paths. You can just copy the files using the same mapping in Windows and it should work just fine.

On Tuesday, May 21, 2013 8:07:50 AM UTC-4, enno...@gmail.com wrote: --
You received this message because you are subscribed to a topic in the Google Groups "Magento Development List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/magento-devel/oRTEtYSEExU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to magento-deve...@googlegroups.com.

To post to this group, send email to magent...@googlegroups.com.

Visit this group at http://groups.google.com/group/magento-devel?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

enno...@gmail.com

Tuesday, May 21, 2013 8:07 AM

Hello!

This is an awesome idea!

I'm trying to set up the Cm_Mongo in my magento but I've a problem while setting up. I've copied the files from Cm_mongo as is said in the modman file and also copied the files from mongodb-php-odm-master to /lib/mongodb-php-odm, but when installing magento it appears he following error:

 Fatal error: Class 'Mongo_Database' not found in C:\wamp\www\app\code\community\Cm\Mongo\Model\Resource\Type\Mongo.php on line 17

Seems like Mongo.php is trying to get the class Mongo_Database from mongodb-php-odm/classes but don't finds it.

Actually, I didn't do this from modman file:

# Map file names to be autoload friendly

lib/mongodb-php-odm/classes/json.php                      lib/JSON.php

lib/mongodb-php-odm/classes/mongo/database.php            lib/Mongo/Database.php

lib/mongodb-php-odm/classes/mongo/collection.php          lib/Mongo/Collection.php

Because I don't know what means (maybe symlinks?). I'm working on Windows.

Any help would be appreciated,

Kind regards,

Pere

On Tuesday, May 21, 2013 11:38:42 AM UTC+2, tbu...@omitsis.com wrote:

--

You received this message because you are subscribed to a topic in the Google Groups "Magento Development List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/magento-devel/oRTEtYSEExU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to magento-deve...@googlegroups.com.

To post to this group, send email to magent...@googlegroups.com.

Visit this group at http://groups.google.com/group/magento-devel?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

tbu...@omitsis.com

Tuesday, May 21, 2013 5:38 AM

Hello Collin,

firts of all, great job.

I'm a Magento stores developer and I've just started a new project that's sent me here.

Until now all projects I've done was only about thousands of products as much, and the standard Magento was enough.

Now I'm in a new complex project.

This project requires two groups of products:

 - Some known products (thousands) that could be manage with MYSQL standard Magento.

 - Another group of products (more than 1M and growing) that I need to save in another system because of Magento limits. This products belongs to a third party business and I need to treat by side.

The finale website will have a complex filter that search users searches in both DDBB returning a combination of two queries.

So that, I've thought in tree options.

1) Put it all together in Magento MYSQL. Probably Magento runs wrong. Advantatges -> all products were catchable by Magento front-end an back-end system.

2) Put it the first group in Magento and the second in a MongoDB. I uppset about using the mongo products in front-end and back-end Magento system. I need that mongo was transparent to users and admin users. The idea is use mongo to holds the big group of products. Then when I show products on the web I can use that products too plus the Magento products.

3) Put it all together in a MongoDB using some tool like your cm_mongo. I think that this could be the best solution, but some things like a complete integration (extensions, sap integration, etc.) upsets me.

I know you'll probably need much more details of the project to choose one option. But I would like to know would you do in my case. And if it was option 3, if you think your module could helps me.

Thank you very much.

El jueves, 24 de noviembre de 2011 06:42:04 UTC+1, Colin M escribió: --
You received this message because you are subscribed to a topic in the Google Groups "Magento Development List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/magento-devel/oRTEtYSEExU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to magento-deve...@googlegroups.com.

To post to this group, send email to magent...@googlegroups.com.

Visit this group at http://groups.google.com/group/magento-devel?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[enno...@gmail.com]

Thank you Collin!

[ra...@librawebsolutions.com]

how to connect mongodb on magento framework guide me step by step thanks in advance

[Colin Mollenhour]

lol

[Mark Neil Borromeo]

This is interesting, I will keep an eye on this project and might fork it soon.. :) thanks Collin.

[TheKhangHuynh]

Dear Colin,

I have install follow modman file, but when I create new product, I've this error

Class 'Mongo' not found in C:\xampp\htdocs\magento demo\lib\Mongo\Database.php on line 182

Can you give me a suggestion??

Vào 04:37:07 UTC+7 Thứ bảy, ngày 30 tháng tư năm 2011, Colin M đã viết:

[Colin Mollenhour]

Mongo is provided by the official MongoDb driver which is still
required. Install it either with pecl "pecl install mongo" or with your
package manager (e.g. "apt-get install php5-mongo").

Colin

[mahesh babu Bokkisam]

How MongoDB is useful with magento?

--

You received this message because you are subscribed to the Google Groups "Magento Development List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to magento-deve...@googlegroups.com.

To post to this group, send email to magent...@googlegroups.com.

Visit this group at http://groups.google.com/group/magento-devel.

For more options, visit https://groups.google.com/groups/opt_out.

--

Thanks and Regards
Mahesh babu. B
+919663776401

[TheKhangHuynh]

After I integrate mongoDb into my magento, I get stuck a error when I clicked Enumerators on admin menu, so I've already created database enums on mongodb:

Fatal error: Call to a member function selectCollection() on a non-object in C:\xampp\htdocs\magentodemo\app\code\community\Cm\Mongo\Model\Resource\Collection\Abstract.php on line 49

Anybody help me, please

Vào 04:37:07 UTC+7 Thứ bảy, ngày 30 tháng tư năm 2011, Colin M đã viết:

I've written a complete Magento resource library which fully integrates MongoDb as a Magento resource model which uses XML schema files (merged like config.xml files), supports advanced features like embedded objects as models, collections that are magento grid-compatible, profiling, etc.. The MongoDb module is open-source and is decently stable. Check it out: Cm_Mongo