Several druid-processing unit-tests fail on Windows

[Jeff Rosen]

Hi everyone,

I'm trying to build on Windows and I get several unit tests in the druid-processing module that fail with errors like the following.

testPersistEmptyColumn(io.druid.segment.IndexMergerTest)  Time elapsed: 0.067 sec  <<< ERROR!

java.io.FileNotFoundException: C:\Users\jeff\AppData\Local\Temp\1425673793776-0\v8-tmp\index.drd (The requested opera

tion cannot be performed on a file with a user-mapped section open)

        at java.io.FileOutputStream.open(Native Method)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:221)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:171)

        at io.druid.segment.IndexMerger.createIndexDrdFile(IndexMerger.java:912)

        at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:873)

        at io.druid.segment.IndexMerger.merge(IndexMerger.java:319)

        at io.druid.segment.IndexMerger.persist(IndexMerger.java:181)

        at io.druid.segment.IndexMerger.persist(IndexMerger.java:151)

        at io.druid.segment.IndexMerger.persist(IndexMerger.java:134)

        at io.druid.segment.IndexMergerTest.testPersistEmptyColumn(IndexMergerTest.java:154)

I'm guessing that this is might be related Windows stricter file locking semantics?  I'll try and debug but I thought I'd ask here in case building on Windows is known not to work?   

This is more of a git question than druid, but what causes these changes to show up in a freshly cloned repo on master? Is this normal?  I've never seen this before with a fresh clone.

C:\GitHub\druid [master +1 ~0 -5 !]> git status

On branch master

Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:

  (use "git add/rm <file>..." to update what will be committed)

  (use "git checkout -- <file>..." to discard changes in working directory)

        deleted:    docs/content/Tutorial:-A-First-Look-at-Druid.md

        deleted:    docs/content/Tutorial:-All-About-Queries.md

        deleted:    docs/content/Tutorial:-Loading-Batch-Data.md

        deleted:    docs/content/Tutorial:-Loading-Streaming-Data.md

        deleted:    docs/content/Tutorial:-The-Druid-Cluster.md

Untracked files:

  (use "git add <file>..." to include in what will be committed)

        docs/content/Tutorial

no changes added to commit (use "git add" and/or "git commit -a")

-jr

[Eric Tschetter]

Jeff,

I do not know of anyone who has developed on Druid on Windows, so it
would not surprise me if there are things we do that Windows is less
than happy with. Hopefully they are easy fixes...

On your git changes, my guess is that you opened those files in an
editor and it decided that you wanted Windows "\n\r" instead of the
unix "\n".

--Eric

> --
> You received this message because you are subscribed to the Google Groups
> "Druid Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to druid-developm...@googlegroups.com.
> To post to this group, send email to druid-de...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/druid-development/dc8f00a1-000c-4b1d-b45b-59d977eb883b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gian Merlino]

The git confusion is probably because Windows filenames can't have colons in them. I'm not sure if there's a way to work around that, but maybe the internet knows.

[Eric Tschetter]

That also makes sense. Silly Windows ;)

> --
> You received this message because you are subscribed to the Google Groups
> "Druid Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to druid-developm...@googlegroups.com.
> To post to this group, send email to druid-de...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/druid-development/51f2feac-5245-4e1a-a6b8-813f8858dc3b%40googlegroups.com.

[Charles Allen]

IIRC hadoop can't have `:' characters in the path either. We actually have some unit tests and logic to specially handle the : character depending on the filesystem.

[Charles Allen]

The issue of Druid on Windows has come up a few times in the past.

Generally the conclusion is that we try to make druid compatible with any OS, but simply do not have the engineering bandwidth (or tools) to make sure it works on all java platforms.

An example of things that are not currently windows compatible are that the classpath separator in Windows is the ; character, whereas in linux it is the : character. I would expect the middle managers launching tasks to have major issues building the proper classpath arguments. 

On Friday, March 6, 2015 at 1:22:43 PM UTC-8, Jeff Rosen wrote:

[Jeff Rosen]

Thanks to all that replied.  Right now, my immediate need is to evaluate druid.  If things work out, I'd be happy to help workout some of these cross-platform issues, but for now I've just moved to linux to keep my life simple.

-jr

--
You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/8S3K7gSa8hA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.

To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/cbc67e46-73e4-4793-aa5f-2c16ac935691%40googlegroups.com.