How to build LevelDB on Windows?
IM46
Berkeley DB but after a full day of trying to get it to compile, I'm
about to give up. The main problem appears to be with port.h. I've set
it up to use the Chromium platform (I do not see a native windows
option). This however introduces a dependency on various sources files
in the /base directory of Chromium. I've attempted to drop the
referenced files into the project but this produces endless errors.
Has anyone successfully compiled LevelDB on Windows? If so, can you
provide some guidance?
baibaichen
chrome
1 get source code http://dev.chromium.org/developers/how-tos/get-the-code
2 building http://dev.chromium.org/developers/how-tos/build-instructions-windows
After #1, you will get the levelDB.sln at thirdparty/levelDB.
I think it is unnecessary to follow up #2 to configure compiler
environment except installing windows 7 sdk
This is my way to build LevelDB on windows
IM46
Chromium build, but I'll give this a try!
On May 23, 2:14 am, baibaichen <baibaic...@gmail.com> wrote:
> To build LevelDB, you should follow the instruction on how to build
> chrome
>
> 1 get source codehttp://dev.chromium.org/developers/how-tos/get-the-code
Edouard A
bit. All LevelDB tests pass. Our engine is using LevelDB as a backend
for a couple of days now and we've got no problem to report so far!
This required writing a specific port_win.cc and a new environment as
the posix environment will not work on Windows.
Two limitations :
1 - port_win.cc uses TBB to implement atomic operations as Visual
Studio 2010 doesn't offer C++ 11 atomics (I could have used #pragma
omp atomic or Interlocked operations, but lazyness overwhelmed me :p)
2 - our environment implementation is based on Boost and therefore
requires boost. There is a number of limits in env_posix that
prevented straightfoward adaptation, mainly the improper handling of
paths containing "\" characters and some other stuff I don't recall
because I'm too tired as I'm typing this.
We also modified the code so that it successfully compiles on FreeBSD
(this required a couple of minor changes on port_posix.h and
env_posix.cc).
By the way I think there is a bug in the way condition variables are
used, when signaling a condition variable the mutex shouldn't be held.
I'd be happy to share our modifications.
-Edouard
Paul Davis
> used, when signaling a condition variable the mutex shouldn't be held.
You have to signal a condition variable while the mutex is locked [1].
If you signal without a lock on the mutex you run into a race
condition called the lost wakeup. There's a note in [2] that I always
remember reading talking about people commonly getting this wrong due
to some old documentation being incorrect. Sadly I can never remember
which way is right so I always have to look it up again.
[1] https://computing.llnl.gov/tutorials/pthreads/#ConVarSignal
[2] http://www.amazon.com/Multithreaded-Programming-PThreads-Bil-Lewis/dp/0136807291
Dave Smith
>
> By the way I think there is a bug in the way condition variables are
> used, when signaling a condition variable the mutex shouldn't be held.
Actually, I think that's not correct -- you must ALWAYS hold a mutex
when signaling a CV or else you risk missing signals.
There is someone wrong on the Internet and I must correct it! ;)
(http://xkcd.com/386/)
D.
Edouard A
The purpose of the mutex is to protect the variable, not the condition variable mechanisms.
On the Microsoft example based on the newest internal windows CV, they release the mutex when signaling the CV: http://msdn.microsoft.com/en-us/library/ms686903(v=vs.85).aspx
Same thing goes for boost : http://www.boost.org/doc/libs/1_46_1/doc/html/thread/synchronization.html#thread.synchronization.condvar_ref
Edouard A
You have to signal a condition variable while the mutex is locked [1].
If you signal without a lock on the mutex you run into a race
Ok then this is mostly specific to the pthread implementation of the condition variable, thanks for the input.
kurak
> I'd be happy to share our modifications.
MKK
Edouard A
kurak
Especially for the env_boost.cc file, Reimplementing this would be
very tedious.
Though on Windows there's no fread_unlocked afaik. I've changed this
to _fread_nolock.
I've also re-implemented Atomic Pointer with Interlocked* operations,
so it works without TBB.
On Jun 20, 1:39 pm, Edouard A <edouard...@gmail.com> wrote:
> I've attached the "port_win" files, one severe limitation is requiring TBB
> for atomic. I think you could use some #pragma for the same effect but in
> our case TBB is fine.
>
> I also included an environment based on Boost. Just making a Windows port
> isn't enough, the port_posix is not 100% posix compatible (doesn't compile
> on FreeBSD for example) and even if it were you'll get a number of problems
> because of the \ in Windows paths.
>
> I didn't remove all the comments and didn't rename everything correctly.
> Hope it doesn't confuse you.
>
> env_boost is tested on Windows, Linux and FreeBSD. Just compile the file
> instead of env_posix and you're good to go.
>
> The source I'm posting here is BSD licensed.
>
> -Edouard
>
>
>
>
>
>
>
>
>
> > > I'd be happy to share our modifications.
>
> > Please do. That'd be immensely helpful.
>
> > MKK
>
>
>
> 5KViewDownload
>
> port_win.h
> 5KViewDownload
>
> env_boost.cc
> 20KViewDownload
Edouard A
#define snprintf _snprintf
#define close _close
#define fread_unlocked _fread_nolock
I moved them around because port.h isn't the right place for them.
I'm curious, how did you use Interlocked operations? Would you care
sharing your implementation?
-Edouard
2011/6/20 kurak <kur...@gmail.com>
Stefan Boberg
> I'm curious, how did you use Interlocked operations? Would you care
> sharing your implementation?
implement AtomicPointer on Windows x86/x64. All aligned pointer
operations are atomic anyway, and you don't need any memory barriers
on x86/x64 due to the memory model.
In fact, that's what the function calls expand to on x86 if you
look at the Chromium implementation - just plain loads and stores. And
that's what I did for my Boost-free Windows port.
-Stefan
--
Stefan Boberg, EA DICE
Technical Director, Frostbite Engine Team
Edouard A
into another?
Although the pointer operation is atomic, you have no guarantee the
compiler will generate only one instruction for your code.
Example:
void * a = ...;
void * b = ...;
if (a != b)
a = b;
will probably generate something along the lines of
mov eax, [a]
cmp [b], eax
jz next
; welcome to race condition world, we hope you will enjoy your stay
mov [b], eax
next:
On the other hand
volatile void * a = ...;
volatile void * b = ... ;
will probably generate the right assembly
mov eax, [b]
lock cmpxchg [a], eax
jz next
mov [b], eax
next:
Am I missing something?
-Edouard
kurak
really sure that this is The Good Way.
But it looks like this:
// Initialize to hold v
AtomicPointer::AtomicPointer(void* v)
{
InterlockedExchangePointer(&rep_, v);
}
// Read and return the stored pointer with the guarantee that no
// later memory access (read or write) by this thread can be
// reordered ahead of this read.
void* AtomicPointer::Acquire_Load() const
{
void* r;
InterlockedExchangePointer(&r, rep_ );
return r;
}
// Set v as the stored pointer with the guarantee that no earlier
// memory access (read or write) by this thread can be reordered
// after this store.
void AtomicPointer::Release_Store(void* v)
{
InterlockedExchangePointer(&rep_, v);
}
// Read the stored pointer with no ordering guarantees.
void* AtomicPointer::NoBarrier_Load() const
{
void* r = reinterpret_cast<void*>(rep_);
return r;
}
// Set va as the stored pointer with no ordering guarantees.
void AtomicPointer::NoBarrier_Store(void* v)
{
rep_ = reinterpret_cast<intptr_t>(v);
}
I guess I could also use MemoryBarrier() macro, but I'm not sure if it
does anything on x86.
Could you guys post your benchmarks from x86 on Windows?
I get rather lousy results with all the optimizations enabled + SSE2.
Using MSVC10.
Is that because of 32-bits? Lack of Snappy?
CPU: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz
OS: Windows 7 32-bit
LevelDB: version 1.2
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
WARNING: Snappy compression is not enabled
------------------------------------------------
fillseq : 5.976 micros/op; 18.5 MB/s
fillsync : 7.501 micros/op; 14.7 MB/s (10000 ops)
fillrandom : 8.018 micros/op; 13.8 MB/s
overwrite : 10.676 micros/op; 10.4 MB/s
readrandom : 17.242 micros/op;
readrandom : 14.986 micros/op;
readseq : 0.413 micros/op; 267.7 MB/s
readreverse : 0.701 micros/op; 157.7 MB/s
Edouard A
LevelDB: version 1.2
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
WARNING: Snappy compression is not enabled
------------------------------------------------
fillseq : 5.018 micros/op; 22.0 MB/s
fillsync : 5.300 micros/op; 20.9 MB/s (10000 ops)
fillrandom : 6.442 micros/op; 17.2 MB/s
overwrite : 6.605 micros/op; 16.7 MB/s
readrandom : 16.933 micros/op;
readrandom : 16.727 micros/op;
readseq : 0.382 micros/op; 289.7 MB/s
readreverse : 0.671 micros/op; 164.8 MB/s
compact : 1172064.000 micros/op;
readrandom : 9.463 micros/op;
readseq : 0.315 micros/op; 350.8 MB/s
readreverse : 0.544 micros/op; 203.4 MB/s
fill100K : 1119.551 micros/op; 85.2 MB/s (1000 ops)
crc32c : 5.547 micros/op; 704.2 MB/s (4K per op)
snappycomp : 0.000 micros/op; (snappy failure)
snappyuncomp : 0.000 micros/op; (snappy failure)
acquireload : 0.840 micros/op; (each op is 1000 loads)
Edouard A
> AtomicPointer::AtomicPointer(void* v)
> {
> InterlockedExchangePointer(&rep_, v);
> }
As far as I can tell this code does the following
void * temp = rep_;
rep_ = v;
return temp;
I don' think that's the intent.
I think you want to do
InterlockedExchangePointer(&v, rep_);
-Edouard
Edouard A
Edouard A
build. The Linux and FreeBSD machines are identical (Q6600). They both
run at comparable speed.
Linux :
LevelDB: version 1.2
Date: Tue Jun 21 13:13:28 2011
CPU: 4 * Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
CPUCache: 4096 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
WARNING: Snappy compression is not enabled
------------------------------------------------
fillseq : 7.847 micros/op; 14.1 MB/s
fillsync : 8.171 micros/op; 13.5 MB/s (10000 ops)
fillrandom : 10.326 micros/op; 10.7 MB/s
overwrite : 14.657 micros/op; 7.5 MB/s
readrandom : 34.661 micros/op;
readrandom : 33.236 micros/op;
readseq : 1.144 micros/op; 96.7 MB/s
readreverse : 3.484 micros/op; 31.8 MB/s
compact : 3091813.000 micros/op;
readrandom : 18.797 micros/op;
readseq : 0.938 micros/op; 117.9 MB/s
readreverse : 3.029 micros/op; 36.5 MB/s
fill100K : 963.255 micros/op; 99.0 MB/s (1000 ops)
crc32c : 4.506 micros/op; 866.9 MB/s (4K per op)
snappycomp : 3.000 micros/op; (snappy failure)
snappyuncomp : 7.000 micros/op; (snappy failure)
acquireload : 5.857 micros/op; (each op is 1000 loads)
FreeBSD:
LevelDB: version 1.2
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
WARNING: Snappy compression is not enabled
------------------------------------------------
fillseq : 7.060 micros/op; 15.7 MB/s
fillsync : 7.085 micros/op; 15.6 MB/s (10000 ops)
fillrandom : 10.303 micros/op; 10.7 MB/s
overwrite : 12.296 micros/op; 9.0 MB/s
readrandom : 50.320 micros/op;
readrandom : 49.764 micros/op;
readseq : 1.036 micros/op; 106.8 MB/s
readreverse : 1.634 micros/op; 67.7 MB/s
compact : 1638005.000 micros/op;
readrandom : 14.949 micros/op;
readseq : 0.514 micros/op; 215.4 MB/s
readreverse : 0.908 micros/op; 121.9 MB/s
fill100K : 3467.195 micros/op; 27.5 MB/s (1000 ops)
crc32c : 4.891 micros/op; 798.6 MB/s (4K per op)
snappycomp : 4.000 micros/op; (snappy failure)
snappyuncomp : 12.000 micros/op; (snappy failure)
acquireload : 6.261 micros/op; (each op is 1000 loads)
kurak
benchmark results on the project site.
Just for the record -- my results with Snappy:
CPU: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz
OS: Windows 7 32-bit
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
fillseq : 6.090 micros/op; 18.2 MB/s
fillsync : 7.401 micros/op; 14.9 MB/s (10000 ops)
fillrandom : 6.822 micros/op; 16.2 MB/s
overwrite : 9.293 micros/op; 11.9 MB/s
readrandom : 20.768 micros/op;
readrandom : 19.895 micros/op;
readseq : 0.501 micros/op; 220.7 MB/s
readreverse : 0.832 micros/op; 132.9 MB/s
compact : 1640164.000 micros/op;
readrandom : 12.346 micros/op;
readseq : 0.421 micros/op; 262.6 MB/s
readreverse : 0.700 micros/op; 158.0 MB/s
fill100K : 1163.116 micros/op; 82.0 MB/s (1000 ops)
crc32c : 4.516 micros/op; 865.0 MB/s (4K per op)
snappycomp : 12.769 micros/op; 305.9 MB/s (output: 55.1%)
snappyuncomp : 1.721 micros/op; 2270.3 MB/s
acquireload : 0.100 micros/op; (each op is 1000 loads)
BTW. I've just noticed, that corruption tests are failing. Does any
one else have this problem?
Edouard A
FreeBSD and Linux).
Which failure(s) do you have?
RenKai
When I upgraded it to version 4.6.0, I pass the corruption tests.
I don't know whether the problem is related to C++0x features.
Edouard A
> BTW. I've just noticed, that corruption tests are failing. Does any
> one else have this problem?
On r33 corruption this test now fails.
-Edouard
Gabor Cselle
Poe
quite the dance trying to get it compiled (and I'm a novice).
Edouard Alligand
> Any chance I can get a copy of your windows binaries? I've been doing
> quite the dance trying to get it compiled (and I'm a novice).
Which binary are you looking for? Our program based on LevelDB (which
is freely available on our website) or the compiled leveldb
libraries/unit tests?
Keep in mind that compiled libraries will require you to use the exact
same compiler and C++ libraries to be useable.
Regards.
-Edouard
