IndexedRedis (python ORM)

[Tim Savannah]

Hey,

I'd like to share my python redis-backed ORM. It follows similar API as other python ORMs, like flask or django.

https://pypi.python.org/pypi/indexedredis
or
https://github.com/kata198/indexedredis

It's special in that it's optimized and makes simple storing objects in a relational way; searchable on selected indexed fields (similar to SQL).

I've tested equivalent models with MySQL and Redis, at a performance increase between 600-1200%!

It also supports atomic replacement of entire datasets. This can make it simple to increase the performance of your application: Have a cron job read your SQL stables and do an atomic reset of the Redis storage every five minutes. Or use it instead of SQL for all your data.

A model is defined like this:

class Song(IndexedRedisModel):

        FIELDS = [ \
                        'artist',
                        'title',
                        'album',
                        'track_number',
                        'duration',
                        'description',
                        'copyright',
        ]

        INDEXED_FIELDS = [ \
                                'artist',
                                'title',
                                'track_number',
        ]

        KEY_NAME = 'Songs'


Objects can be saved like this:

            songObj = Song(artist='The Unhappy Folk',
                            title='Sadly she waits',
                            album='Misery loses comfort',
                            track_number=1,
                            duration='15:44',
                            description='A sad song',
                            copyright='Copyright 2014 (c) Cheese Industries')
            newSongs.append(songObj)

And searched like this:

someSongs = Song.objects.filter(artist='The Unhappy Folk').all()

Underlying data is stored using hashes and sets.

Check out https://raw.githubusercontent.com/kata198/indexedredis/master/test.py for a full-working example, or the pages listed above for more methods, usage, etc.

Thanks and enjoy! Keep using Redis!

[Josiah Carlson]

For those who don't already know, there's also my Redis object mapper for Python - rom [1], which offers typed columns, custom indexes (unique, numeric, full text, ...), custom index key generators, and queries spelled similarly. Rom table definitions are a bit different:

class Song(rom.Model):

    artist = rom.String(index=True)

    title = rom.String(index=True)

    album = rom.String()

    track_number = rom.Integer(index=True)

    duration = rom.String()

    description = rom.String()

    copyright = rom.String()

    unique_together = [

        ('artist', 'album', 'title')

    ]

And queries can be chained/combined:

# should return the same results

Song.query.filter(artist='someone', title='woo').all()

Song.query.filter(artist='someone').filter(title='woo').all()

You can also get enforced uniqueness on single (set unique=True during column definition) or multiple columns (see the unique_together list on the definition above).

 - Josiah

[1] - https://pypi.python.org/pypi/rom/

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Tim Savannah]

I actually wrote this to replace usage of ROM within an application I was developing..

ROM did not have the performance needed by the application (that situation may have changed), IndexedRedis has almost no overhead. IndexedRedis does not try to convert your data, or enforce constraints; it aims to be a high-performing machine. Think of it as an advanced interface to the Redis hash, with O(1) searches. It also has the ability to do atomic replacements of entire datasets, which is efficient for being a "live" front.
An example usage is to have a process consolidating and aggregating hundreds of thousands of individual SQL records into an interface. You may have IndexedRedis models which copy the SQL models, for a "drilled-in" detail view, as well as other models to represent the aggregation. Your system can have displayable" data a few moments behind  the live stream.

Both of them have their uses for sure. If you're looking to run very SQL operations within Redis, ROM is may be better for you. If you're looking to combine high-performance with a very simple interface, IndexedRedis may be a better bet.

[Josiah Carlson]

On Tue, Jul 7, 2015 at 8:58 PM, Tim Savannah <kat...@gmail.com> wrote:

I actually wrote this to replace usage of ROM within an application I was developing..

Awesome :)

ROM did not have the performance needed by the application (that situation may have changed), IndexedRedis has almost no overhead. IndexedRedis does not try to convert your data, or enforce constraints; it aims to be a high-performing machine. Think of it as an advanced interface to the Redis hash, with O(1) searches. It also has the ability to do atomic replacements of entire datasets, which is efficient for being a "live" front.

Searches can only be O(1) if they only access O(1) keys/values, O(1) items within values, and return O(1) amounts of data. Is there some type of query aside from primary-key lookup or accidentally unique filters that are O(1) in your library?

 

An example usage is to have a process consolidating and aggregating hundreds of thousands of individual SQL records into an interface. You may have IndexedRedis models which copy the SQL models, for a "drilled-in" detail view, as well as other models to represent the aggregation. Your system can have displayable" data a few moments behind  the live stream.

This is a good use-case. There was another poster here at least a year ago (IIRC) who was parsing a MySQL replication stream to do the same thing (though not in Python). I've also used rom for this.

Both of them have their uses for sure. If you're looking to run very SQL operations within Redis, ROM is may be better for you. If you're looking to combine high-performance with a very simple interface, IndexedRedis may be a better bet.

That's pretty fair :)

I hope that IndexedRedis continues to serve your use-cases :)

 - Josiah