Bulk data loading into Redis
1,061 views
Skip to first unread message
Karim
Oct 14, 2011, 9:27:31 AM10/14/11
to Redis DB
Hello !
For now, to load data into Redis, I'm following this method :
From an existing RDBMS, I managed to create a simple BCP that generate
Redis protocol out of the data, then I replace the AOF file and start
my Redis instance.
Is there any best way doing this (Transforming data to Redis
protocol) ? As I'm loading millions of rows, the generated AOF file
becomes too big !
Any option to tell Redis to load data from AOF each 15 min for
example ?
Thanks.
For now, to load data into Redis, I'm following this method :
From an existing RDBMS, I managed to create a simple BCP that generate
Redis protocol out of the data, then I replace the AOF file and start
my Redis instance.
Is there any best way doing this (Transforming data to Redis
protocol) ? As I'm loading millions of rows, the generated AOF file
becomes too big !
Any option to tell Redis to load data from AOF each 15 min for
example ?
Thanks.
Josiah Carlson
Oct 14, 2011, 1:58:08 PM10/14/11
to redi...@googlegroups.com
Pipe your commands into the redis-cli instead of using AOF. Then there
is no file on disk to grow too large, and you are limited basically to
the same performance as the AOF reading.
is no file on disk to grow too large, and you are limited basically to
the same performance as the AOF reading.
For example...
your command to generate redis commands | redis-cli > /dev/null
If you want to see what is happening, you can have another console
running the MONITOR command, or whatever else you want.
Regards,
- Josiah
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>
Jak Sprats
Oct 15, 2011, 12:19:04 PM10/15/11
to Redis DB
Or do half and half ... half w/ the rewrite to AOF and half like
Josiah said.
I have to ask, how big is your harddrive that you can fit the data
into RAM but cant fit it on your harddrive
Karim
Oct 17, 2011, 8:43:12 AM10/17/11
to Redis DB
Thanks for your ideas !
@Josiah : Pipe commands are not limited on size ? I mean if I have a
very big string to set, it will be truncated ? (Especially on
Windows). My BCP tool just generate raw data file (formatted to redis
syntax).
@Jak : I'm using Redis especially as a cache to a big database (more
than 2 Tb); clients are connected to cache in order to have the
optimal response time. The server is running under Windows Server 2008
with 32 Gb of memory.
@Josiah : Pipe commands are not limited on size ? I mean if I have a
very big string to set, it will be truncated ? (Especially on
Windows). My BCP tool just generate raw data file (formatted to redis
syntax).
@Jak : I'm using Redis especially as a cache to a big database (more
than 2 Tb); clients are connected to cache in order to have the
optimal response time. The server is running under Windows Server 2008
with 32 Gb of memory.
Sergei Tulentsev
Oct 17, 2011, 9:00:04 AM10/17/11
to redi...@googlegroups.com
offtopic: is Redis on windows supported? :)
I know many projects that don't care about windows, and also a few that chose specifically not to run on it. :-)--
Best regards,
Sergei Tulentsev
I know many projects that don't care about windows, and also a few that chose specifically not to run on it. :-)
Best regards,
Sergei Tulentsev
Hampus Wessman
Oct 17, 2011, 9:21:55 AM10/17/11
to redi...@googlegroups.com
Redis is not officially supported on Windows (see
http://redis.io/topics/introduction), so it's not really recommended
to run it on Windows. Apparently it works well for some people.
One could run a virtual machine with Linux inside Windows to get around the problem, I guess. Shouldn't waste that much performance on a big machine. Certainly a lot more complicated, though.
One could run a virtual machine with Linux inside Windows to get around the problem, I guess. Shouldn't waste that much performance on a big machine. Certainly a lot more complicated, though.
Karim
Oct 17, 2011, 9:48:34 AM10/17/11
to Redis DB
@Sergei : Yes :) Check this : https://github.com/ServiceStack/ServiceStack.Redis
@Hampus : Great idea ! well, for now the average response time is less
than 200ms, wich is much faster than doing a query directly on the
RDBMS (Sysbase IQ); all this is running under Windows.
My question was about data loading into Redis from an existing RDBMS,
and how to refresh it every 10 min for example :)
Thanks.
On 17 oct, 15:21, Hampus Wessman <hampus.wess...@gmail.com> wrote:
> Redis is not officially supported on Windows (seehttp://redis.io/topics/introduction), so it's not really recommended to
> > <mailto:redis-db%2Bunsu...@googlegroups.com>.
> > <http://groups.google.com/group/redis-db?hl=en>.
> > <mailto:redis-db%2Bunsu...@googlegroups.com>.
@Hampus : Great idea ! well, for now the average response time is less
than 200ms, wich is much faster than doing a query directly on the
RDBMS (Sysbase IQ); all this is running under Windows.
My question was about data loading into Redis from an existing RDBMS,
and how to refresh it every 10 min for example :)
Thanks.
On 17 oct, 15:21, Hampus Wessman <hampus.wess...@gmail.com> wrote:
> Redis is not officially supported on Windows (seehttp://redis.io/topics/introduction), so it's not really recommended to
> run it on Windows. Apparently it works well for some people.
>
> One could run a virtual machine with Linux inside Windows to get around
> the problem, I guess. Shouldn't waste that much performance on a big
> machine. Certainly a lot more complicated, though.
>
> On 10/17/2011 03:00 PM, Sergei Tulentsev wrote:
>
>
>
>
>
>
>
> > offtopic: is Redis on windows supported? :)
> > I know many projects that don't care about windows, and also a few
> > that chose specifically not to run on it. :-)
>
>
> One could run a virtual machine with Linux inside Windows to get around
> the problem, I guess. Shouldn't waste that much performance on a big
> machine. Certainly a lot more complicated, though.
>
> On 10/17/2011 03:00 PM, Sergei Tulentsev wrote:
>
>
>
>
>
>
>
> > offtopic: is Redis on windows supported? :)
> > I know many projects that don't care about windows, and also a few
> > that chose specifically not to run on it. :-)
>
> > On Mon, Oct 17, 2011 at 4:43 PM, Karim <medkarim.esska...@gmail.com
> > <mailto:medkarim.esska...@gmail.com>> wrote:
>
> > Thanks for your ideas !
>
> > @Josiah : Pipe commands are not limited on size ? I mean if I have a
> > very big string to set, it will be truncated ? (Especially on
> > Windows). My BCP tool just generate raw data file (formatted to redis
> > syntax).
>
> > @Jak : I'm using Redis especially as a cache to a big database (more
> > than 2 Tb); clients are connected to cache in order to have the
> > optimal response time. The server is running under Windows Server 2008
> > with 32 Gb of memory.
>
> > On 15 oct, 18:19, Jak Sprats <jakspr...@gmail.com
> > <mailto:medkarim.esska...@gmail.com>> wrote:
>
> > Thanks for your ideas !
>
> > @Josiah : Pipe commands are not limited on size ? I mean if I have a
> > very big string to set, it will be truncated ? (Especially on
> > Windows). My BCP tool just generate raw data file (formatted to redis
> > syntax).
>
> > @Jak : I'm using Redis especially as a cache to a big database (more
> > than 2 Tb); clients are connected to cache in order to have the
> > optimal response time. The server is running under Windows Server 2008
> > with 32 Gb of memory.
>
> > On 15 oct, 18:19, Jak Sprats <jakspr...@gmail.com
> > <mailto:jakspr...@gmail.com>> wrote:
> > > Or do half and half ... half w/ the rewrite to AOF and half like
> > > Josiah said.
>
> > > I have to ask, how big is your harddrive that you can fit the data
> > > into RAM but cant fit it on your harddrive
>
> > > On Oct 14, 11:58 am, Josiah Carlson <josiah.carl...@gmail.com
> > > Or do half and half ... half w/ the rewrite to AOF and half like
> > > Josiah said.
>
> > > I have to ask, how big is your harddrive that you can fit the data
> > > into RAM but cant fit it on your harddrive
>
> > > On Oct 14, 11:58 am, Josiah Carlson <josiah.carl...@gmail.com
> > <mailto:josiah.carl...@gmail.com>> wrote:
>
> > > > Pipe your commands into the redis-cli instead of using AOF.
> > Then there
> > > > is no file on disk to grow too large, and you are limited
> > basically to
> > > > the same performance as the AOF reading.
>
> > > > For example...
>
> > > > your command to generate redis commands | redis-cli > /dev/null
>
> > > > If you want to see what is happening, you can have another console
> > > > running the MONITOR command, or whatever else you want.
>
> > > > Regards,
> > > > - Josiah
>
> > > > On Fri, Oct 14, 2011 at 6:27 AM, Karim
> > <medkarim.esska...@gmail.com <mailto:medkarim.esska...@gmail.com>>
>
> > > > Pipe your commands into the redis-cli instead of using AOF.
> > Then there
> > > > is no file on disk to grow too large, and you are limited
> > basically to
> > > > the same performance as the AOF reading.
>
> > > > For example...
>
> > > > your command to generate redis commands | redis-cli > /dev/null
>
> > > > If you want to see what is happening, you can have another console
> > > > running the MONITOR command, or whatever else you want.
>
> > > > Regards,
> > > > - Josiah
>
> > > > On Fri, Oct 14, 2011 at 6:27 AM, Karim
> > wrote:
> > > > > Hello !
>
> > > > > For now, to load data into Redis, I'm following this method :
> > > > > From an existing RDBMS, I managed to create a simple BCP
> > that generate
> > > > > Redis protocol out of the data, then I replace the AOF file
> > and start
> > > > > my Redis instance.
>
> > > > > Is there any best way doing this (Transforming data to Redis
> > > > > protocol) ? As I'm loading millions of rows, the generated
> > AOF file
> > > > > becomes too big !
> > > > > Any option to tell Redis to load data from AOF each 15 min for
> > > > > example ?
>
> > > > > Thanks.
>
> > > > > --
> > > > > You received this message because you are subscribed to the
> > Google Groups "Redis DB" group.
> > > > > To post to this group, send email to
> > redi...@googlegroups.com <mailto:redi...@googlegroups.com>.
> > > > > Hello !
>
> > > > > For now, to load data into Redis, I'm following this method :
> > > > > From an existing RDBMS, I managed to create a simple BCP
> > that generate
> > > > > Redis protocol out of the data, then I replace the AOF file
> > and start
> > > > > my Redis instance.
>
> > > > > Is there any best way doing this (Transforming data to Redis
> > > > > protocol) ? As I'm loading millions of rows, the generated
> > AOF file
> > > > > becomes too big !
> > > > > Any option to tell Redis to load data from AOF each 15 min for
> > > > > example ?
>
> > > > > Thanks.
>
> > > > > --
> > > > > You received this message because you are subscribed to the
> > Google Groups "Redis DB" group.
> > > > > To post to this group, send email to
> > <mailto:redis-db%2Bunsu...@googlegroups.com>.
> > <http://groups.google.com/group/redis-db?hl=en>.
>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com
> > <mailto:redi...@googlegroups.com>.
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com
> > <mailto:redis-db%2Bunsu...@googlegroups.com>.
Josiah Carlson
Oct 17, 2011, 7:19:37 PM10/17/11
to redi...@googlegroups.com
The way that pipes work is that it stalls the sending program if the
receiving program isn't processing it fast enough, modulo some
potentially system-configurable buffer size. I used to do this kind of
thing...
receiving program isn't processing it fast enough, modulo some
potentially system-configurable buffer size. I used to do this kind of
thing...
zcat db_dump.sql.gz | filter_out_big_table_data | psql dbname
It worked quite well.
- Josiah
Karim
Oct 18, 2011, 3:50:40 AM10/18/11
to Redis DB
Very interesting, I will try your idea and compare loading time.
For Linux, it seems that I can use it (I'm waiting for a confirmation
from security guys).
Thanks.
Karim
For Linux, it seems that I can use it (I'm waiting for a confirmation
from security guys).
Thanks.
Karim
Reply all
Reply to author
Forward
0 new messages