port to windows
Jak Sprats
has anyone successfully ported redis to windows (I remember hearing
about some packages, are any stable/tested)?
would using Node.js's libuv in a port to windows make sense (I also
vaguely remember this being discussed)
I need to use redis on a window's box (not overly excited about
this :).
- jak
Marc Gravell
- jak
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Regards,
Marc
Dušan D. Majkić
> about some packages, are any stable/tested)?
Yes, feel free to give it a try.
Redis 2.4.2 binaries can be found at:
https://github.com/dmajkic/redis/downloads
You can also build from source, using any MinGW install. If you install
tcl/tk in MSYS, you can run all tests that came with Redis. Before
submitting binaries, I check that all tests pass for both 32bit and 64bit.
Since Windows lacks fork() all background saves are performed as
foreground, so turn off automatic saves, and call SAVE when and
if you need.
I use it as 64bit memory storage for 32bit app (when run on Win64).
Jak Sprats
thanks Dusan,
Pretty cool that you did this, what is the performance like? how does
redis-benchmark run on it?
Also there is no support for UnixDomainSockets ... they also dont
exist on windows, right?
I did not know that windows did not support fork (I assumed this to be
so fundamental that it would be supported everywhere ... live and
learn).
I am pretty sure I will need snapshotting & I cant have it run in the
foreground ... which means I am kinda screwed, right? This is where an
external process that rewrites the AOF file continually and creates
snapshots would be killer ...
I will check it out, thanks, jak
Josiah Carlson
the cygwin fork is an interesting beast. Basically it copies all known
memory regions from one process to the other via a shared mmap. It's a
bit slow ;)
Also, I heard that you could fork with the posix subsystem, but that
precluded you from using win32/win64, which means that basically
everything else you wanted to do would be a PITA.
Regards,
- Josiah
On Thu, Nov 17, 2011 at 10:19 PM, Jak Sprats <jaks...@gmail.com> wrote:
>
> thanks Dusan,
>
> Pretty cool that you did this, what is the performance like? how does
> redis-benchmark run on it?
>
> Also there is no support for UnixDomainSockets ... they also dont
> exist on windows, right?
>
> I did not know that windows did not support fork (I assumed this to be
> so fundamental that it would be supported everywhere ... live and
> learn).
>
> I am pretty sure I will need snapshotting & I cant have it run in the
> foreground ... which means I am kinda screwed, right? This is where an
> external process that rewrites the AOF file continually and creates
> snapshots would be killer ...
>
> I will check it out, thanks, jak
>
>
> On Nov 17, 2:54 am, Duąan D. Majkić <dmaj...@gmail.com> wrote:
>> > has anyone successfully ported redis to windows (I remember hearing
>> > about some packages, are any stable/tested)?
>>
>> Yes, feel free to give it a try.
>>
>> Redis 2.4.2 binaries can be found at:
>>
>> https://github.com/dmajkic/redis/downloads
>>
>> You can also build from source, using any MinGW install. If you install
>> tcl/tk in MSYS, you can run all tests that came with Redis. Before
>> submitting binaries, I check that all tests pass for both 32bit and 64bit.
>>
>> Since Windows lacks fork() all background saves are performed as
>> foreground, so turn off automatic saves, and call SAVE when and
>> if you need.
>>
>> I use it as 64bit memory storage for 32bit app (when run on Win64).
>
Dušan D. Majkić
> memory regions from one process to the other via a shared mmap. It's a
> bit slow ;)
True, and useless for Redis.
Redis background saving relays on Copy-On-Write optimization that
is provided with modern Unix kernels. That means that fork() will not
copy or allocate memory unless parent process writes to it. There is
no useless allocating, and it should save to disk even if parent dies.
I'm not aware of a way to do documented memory Copy-On-Write
on Windows without kernel drivers or other "magic". I could be
wrong and I would love to be proven wrong.
Cygwin fork() is useless since it will copy all memory from parent
process to child. Since redis is a storage engine, and can have
gigabytes of data, it is not a good thing to do.
I had an idea to start background thread, and implement COW
directly for changed deleted Redis objects to hidden extra database.
While traversing objects from thread, if key exists in hidden db
write
On the other hand, I use it fine without background saving for
what I need:
* Start 64bit Redis and use it as 64bit storage for 32bit app.
* SLAVE OF Unix master
* Develop with local Redis on Windows platform
Regards
Dusan Majkic
Josiah Carlson
>> the cygwin fork is an interesting beast. Basically it copies all known
>> memory regions from one process to the other via a shared mmap. It's a
>> bit slow ;)
>
> True, and useless for Redis.
Yep. I wasn't offering it as a solution, just a "there is this thing,
but it's not what we are used to".
> Redis background saving relays on Copy-On-Write optimization that
> is provided with modern Unix kernels. That means that fork() will not
> copy or allocate memory unless parent process writes to it. There is
> no useless allocating, and it should save to disk even if parent dies.
>
> I'm not aware of a way to do documented memory Copy-On-Write
> on Windows without kernel drivers or other "magic". I could be
> wrong and I would love to be proven wrong.
I see two potential pieces to make it happen on Windows, but it could
get nasty. Basically replace malloc/free with anonymous mmap regions
(whee, a new malloc replacement :P). Windows has a method of passing
file handles between processes, which, I believe, can be used to pass
references to these mmap regions across a pipe (Linux, BSD, etc., all
have this capability, btw, and is how Python's multiprocessing library
can pass file handles, sockets, etc., from one process to another on
Windows, Linux, OSX, BSD, etc.). As for the copy on write, well,
http://support.microsoft.com/kb/103858 . Basically it allows you to
convert shared memory to copy-on-write shared memory.
Whether that can be turned into a viable fork replacement on
Windows... I don't know. It's more moving pieces than I am comfortable
with. I thought I heard mentioned that using this kind of thing is how
Chrome can start new processes so fast, but I may be mistaken.
Regards,
- Josiah
> Cygwin fork() is useless since it will copy all memory from parent
> process to child. Since redis is a storage engine, and can have
> gigabytes of data, it is not a good thing to do.
>
> I had an idea to start background thread, and implement COW
> directly for changed deleted Redis objects to hidden extra database.
> While traversing objects from thread, if key exists in hidden db
> write
>
> On the other hand, I use it fine without background saving for
> what I need:
>
> * Start 64bit Redis and use it as 64bit storage for 32bit app.
> * SLAVE OF Unix master
> * Develop with local Redis on Windows platform
>
> Regards
> Dusan Majkic
>
Jak Sprats
> directly for changed deleted Redis objects to hidden extra database.
> While traversing objects from thread, if key exists in hidden db
> write
W/ this approach, any CRUD operation would write (the whole key-value
tuplet) to the hiddenDB & the 2nd (dumping) thread would first ask the
hiddenDB & if-not-present ask the normalDB.
Makes sense, it still hits the problem, that if a 1million member ZSET
keeps getting changed, it needs to be copied to the hiddenDB every
change. Also context swapping would end up in SEGVs as pointers will
have moved (e.g. due to rehashing)
Perhaps implementing something like key-level-locking (similar to the
blockingness of the MIGRATE command) could be employed to be able to
dump rdb's w/o COW. This means requests accessing this key would have
to be queued during when locked by another thread, and this gets us
into the whole ugly area of deadlocks, etc....
This is a not-new pretty-hard problem. I have always been a fan of a
2nd process that tries to compact the AOF file as best as possible.
Jak Sprats
conceptually, this idea works, I think.
The problem is windows supports COW between processes, but if redis is
dynamically allocating memory, then there is no way that a 2nd process
can access this memory unless it is shared. (this is a pretty subtle
show-stopper)
Tweaking an allocator to use shared-memory is AFAIK a bad idea in
practice ...
Cool brainstorming though :)
- jak
On Nov 18, 9:03 am, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> 2011/11/18 Dušan D. Majkić <dmaj...@gmail.com>:
>
> >> the cygwin fork is an interesting beast. Basically it copies all known
> >> memory regions from one process to the other via a shared mmap. It's a
> >> bit slow ;)
>
> > True, and useless for Redis.
>
> Yep. I wasn't offering it as a solution, just a "there is this thing,
> but it's not what we are used to".
>
> > Redis background saving relays on Copy-On-Write optimization that
> > is provided with modern Unix kernels. That means that fork() will not
> > copy or allocate memory unless parent process writes to it. There is
> > no useless allocating, and it should save to disk even if parent dies.
>
> > I'm not aware of a way to do documented memory Copy-On-Write
> > on Windows without kernel drivers or other "magic". I could be
> > wrong and I would love to be proven wrong.
>
> I see two potential pieces to make it happen on Windows, but it could
> get nasty. Basically replace malloc/free with anonymous mmap regions
> (whee, a new malloc replacement :P). Windows has a method of passing
> file handles between processes, which, I believe, can be used to pass
> references to these mmap regions across a pipe (Linux, BSD, etc., all
> have this capability, btw, and is how Python's multiprocessing library
> can pass file handles, sockets, etc., from one process to another on
> Windows, Linux, OSX, BSD, etc.). As for the copy on write, well,http://support.microsoft.com/kb/103858. Basically it allows you to
Josiah Carlson
> Hi Josiah,
>
> conceptually, this idea works, I think.
>
> The problem is windows supports COW between processes, but if redis is
> dynamically allocating memory, then there is no way that a 2nd process
> can access this memory unless it is shared. (this is a pretty subtle
> show-stopper)
My thought is that since Redis does point-in-time persistence, you
would spawn a process, pass over file handle references to any memory
regions you know about with a size and base address, let the other
side remap those regions (with some lookup tables for proper base
address stuff), then it would go about dumping in the standard way.
When done, the process would end, and that would be that.
> Tweaking an allocator to use shared-memory is AFAIK a bad idea in
> practice ...
It's what TCMalloc and some of the other "faster than malloc"
replacements actually do (use anonymous mmaps).
> Cool brainstorming though :)
Thank you :)
- Josiah
Josiah Carlson
handling this stuff. The only thing that it doesn't work well on, is
hashes and long strings. But... if you have a single bit in a tree
node, you can use some tricks to only copy a tree node once.
I've been meaning to implement a particular kind of tree for inclusion
into Redis with this sort of functionality which offers many other
conveniences:
* low-pointer overhead
* shallow tree depth
* one structure for a mapping (replace hashes), list, set, zset, and
long strings (to solve the issue with long string changes).
I haven't gotten around to doing it because I've been busy, but some
weekend in the next month or two I will feel inspired and hack it
together. It would probably need some integration, but it would make
non-forking point-in-time persistence possible.
Regards,
- Josiah
Dušan D. Majkić
> keeps getting changed, it needs to be copied to the hiddenDB every
> change.
No, only first and original key/value pair is needed, since that is the one
that should go to disk. Writing to rdb would be first from hidden db, and
if there is no value there, than from parent.
Problems that remain are that redis can hold more databases, so keys
should be prefixed. And second issue is that databases are in fact
deprecated feature.
Jak Sprats
> > keeps getting changed, it needs to be copied to the hiddenDB every
> > change.
>
> No, only first and original key/value pair is needed, since that is the one
> that should go to disk. Writing to rdb would be first from hidden db, and
> if there is no value there, than from parent.
OK, I get it, write the original key when it is dirties, what I wrote
was stupid :)
There is still a race condition, think about the interaction of the
following and remember they can happen in any order (I hate race
conditions w/ threads)
1.) key is being modified
2.) original key is being copied to hiddenDB
3.) dumping thread is checking normalDB
There would have to be some locking, also the hash-table can resize
itself during dump ... seems like locks would have to be put in all
over the place.
On another note (and this doesnt help my use case, but it helps
Dusan's maybe), you could have master->slave and BGSAVE could be a
noop on the master and then a in-process SAVE on the slave and
hopefully the SAVE happens quick enough on the slave that the
transactions that happen during the SAVE get buffered in the kernel
(you can make the slave's kernel Q's real big) .....
that way you could get persistence and have it be blocking, but not on
the master.
just a thought
Jak Sprats
> > practice ...
> It's what TCMalloc and some of the other "faster than malloc"
> replacements actually do (use anonymous mmaps).
I get that TCMalloc uses anonymous maps, but these aren't anonymous
maps in shared-memory (AFAIK), they are in per-process memory. If you
know of a standard allocator that is simple to tweak so that it
allocates from shared-memory, please share, because I have had to
write my own more than once, and it sucked.
Josiah Carlson
Not off the top of my head and I've usually used wrappers around all
of the shared memory ugly over the years.
- Josiah
kehai chen
Dušan D. Majkić
> function:zmalloc_get_rss,int the "zmalloc.c".
On Windows port it equals to zmalloc_used_memory()
kehai chen
I am reading source-code of redis,but I don't understand the function:zmalloc_get_rss,int the "zmalloc.c".
Ronny Meeus
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
>
This function just returns the amount of "resident memory" in use by
the current running process.
The information is present in the stat file found in the /proc filesystem.
Resident memory is defined in
http://www.ehow.com/about_5497321_much-linux-memory-used-process.html
as:
"The resident memory size of a process reported by Linux includes only
the amount of physical memory the process and the shared libraries it
references are using at a given time. Segments moved to swap space are
not included. Like with virtual memory size, resident memory size
includes the space used by shared libraries. "
--
Ronny
Jak Sprats
I had an idea on how to fake something like fork on windows ... here
is the setup
1.) master: ignores SAVE & BGSAVE
2.) slave: on SAVE or BGSAVE, enters into saving-mode (starts in
process SAVE)
3.) on slave saving-mode, requests are still accepted (from the
master) but are queued in some buffer.
4.) when the in-process SAVE is done on the slave, it will replay the
contents of the commands in the queue, and once done, leave saving-
mode.
There is a race condition: master running at 100%, slave goes into
saving mode, queue builds so quickly it cant drain ... but in
practice, most of redis' work is actually tcp soft interrupt handling,
so I dont think this race condition will happen in practice ...
I think this makes sense and would work, thoughts?
- jak
On Nov 17, 6:54 am, Dušan D. Majkić <dmaj...@gmail.com> wrote:
> > has anyone successfully ported redis towindows(I remember hearing
> > about some packages, are any stable/tested)?
>
> Yes, feel free to give it a try.
>
> Redis 2.4.2 binaries can be found at:
>
> https://github.com/dmajkic/redis/downloads
>
> You can also build from source, using any MinGW install. If you install
> tcl/tk in MSYS, you can run all tests that came with Redis. Before
> submitting binaries, I check that all tests pass for both 32bit and 64bit.
>
> SinceWindowslacks fork() all background saves are performed as