Question regarding VoldemortBuildAndPush job

[Andrey Stanovsky]

Hello,

I'm working on a test project that should push avro data from hdfs to a voldemort read-only cluster. I've faced a lot of obstacles and did not succeed so far, I was stopped by the following thing:

 java.lang.IllegalArgumentException: Should not get various nodeId shuffled to us! First nodeId seen: 0, currentNodeId: 2 

How do I need to specify to push only into one node or I misunderstood the logic there? I thought, in theory, job should distribute data evenly by its own logic. Or data about nodeId and partition  should be placed into an avro file ? There is no too much material about that in the internet. I can provide you with all I have configured for voldemort or azkaban build and push job.

[Félix GV]

Hi Andrey,

Can you tell us what version (or commit hash) of Voldemort are you running? Did you modify the code in any way? Also, can you post your cluster.xml, server.properties and your BnP job config?

Thanks!

-F

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Andrey Stanovsky]

I'm running release-1.10.14-cutoff. I've also put deps from voldemort build to the created VoldemorBuildAndPush jobtype.
Attaching files that you requested.

I think I need to tell you a full story, since I don't know if I'm heading the right direction.
I've created hadoop job that spawns test avro file with key and value fields that are both strings.
The entire goal is to push this data to the read-only voldemort cluster.

First of all, I came to this voldemort build and push page.
The link to the tarball of an example has been changed to some hhw-new-bersion.tar.gz. Ok, when I unpacked it and built with ant no problem has occurred, but when I tried to run the job as described in bundled readme file I caught  CNF exception regarding azkaban.jobs.CommandLineJobRunner class. And indeed, I could not find this class in sources...

Ok if this didn't work, I downloaded and installed azkaban solo server 2.5, created VoldemorBuildAndPush type and tried through web ui run this job that I've attached.
You can find that paths to avro input file are simple / path but when I tried to switch to hdfs:// protocol I had exception related to protobuf CNF O_o...

Should have mentioned that I've removed these nodeid, partitionid checks and recompiled voldemort jars, but eventually I got another exception that webhdfs paths are wrong. it tried to access webhfs://:50070/<some_inner_path_to_output> and of course failed.

воскресенье, 24 апреля 2016 г., 3:20:46 UTC+3 пользователь Félix GV написал:

[Arunachalam]

We had a gradle build rewrite on recently. Can you try 1.10.10 ?

[Arunachalam]

1.10.10 had an outstanding bug and try this on branch 1.10.14 and let us know if the error still happens ?

Thanks,

Arun.

[Andrey Stanovsky]

I've tried 1.10.10/14, last sources from repo... I'm getting deps from lib folder and also voldemort + voldemort-contrib jar and put all these things to VOldemortBuildandPush lib folder... and when I run the job, I caught this exception:

Caused by: java.lang.NoClassDefFoundError: com/google/protobuf/MessageOrBuilder 24-04-2016 23:53:01 PDT bap2 ERROR - at org.apache.hadoop.ipc.ProtobufRpcEngine.getProxy(ProtobufRpcEngine.java:90) 24-04-2016 23:53:01 PDT bap2 ERROR - at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:537) 24-04-2016 23:53:01 PDT bap2 ERROR - at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:334) 24-04-2016 23:53:01 PDT bap2 ERROR - at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:241) 24-04-2016 23:53:01 PDT bap2 ERROR - at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:141)

понедельник, 25 апреля 2016 г., 9:30:41 UTC+3 пользователь Arun Thirupathi написал:

[Arunachalam]

Which version of Hadoop you are running this job from ? The error indicates that protobuf is missing in the jar.

Instead of putting voldemort and voldemort-contrib.jar I recommend building the shadowJar

https://github.com/voldemort/voldemort/blob/master/build.gradle#L201

./gradlew shadowJar should build a shadowJar with all the Hadoop dependencies.

Thanks,

Arun.

Thanks,

Arun.

[Andrey Stanovsky]

I've picked up cdh5

Tried shadow jar, put it to the VoldemortBuildandPush/lib instead of deps+vold+vold-contrib, azkaban didn't even start with this exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/security/AccessControlException
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2493

when I returned deps+vold+vold-contrib jars, server started normally.


понедельник, 25 апреля 2016 г., 10:28:00 UTC+3 пользователь Arun Thirupathi написал:

[Arunachalam]

The shadow jar is for hadoop push side. What you are doing for server side is right.

That brings up the question where are you seeing this exception ? Hadoop build and push or in the server ?

[Andrey Stanovsky]

It's from azkaban start logs

понедельник, 25 апреля 2016 г., 17:44:35 UTC+3 пользователь Arun Thirupathi написал:

[Félix GV]

Can you try running BnP without Azkaban, just to rule out the possibility that you have an Azkaban setup issue?

Try this:

./gradlew clean bnpJar
./bin/run-bnp.sh

[Andrey Stanovsky]

That worked now, but partially. The job didn't succeed to place output to the hdfs correctly, it created parts but all of them are empty when I try to check content by avro tool jar. Also the output folder name and the folder that job tries to fetch are different. Actual folder is vold-build-and-push--<some_hash> but it tries to fetch from the folder with the hostname of the voldemort cluster. And it fails. I don't know if it incorrect folder name or how I configured webhdfs. I added only the following property:

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>

btw, Reduce output records=0  - is it normal?

понедельник, 25 апреля 2016 г., 18:42:59 UTC+3 пользователь Félix GV написал:

[Félix GV]

On Mon, Apr 25, 2016 at 1:36 PM, Andrey Stanovsky <sand...@gmail.com> wrote:

Reduce output records=0

That is not normal, no... If there is data in your input directory, there should be reduce output records.

Can you post your BnP job configs please?

-F

--
--
Félix

[Andrey Stanovsky]

Attached bap.job and cluster.xml (once again, since I;ve changed partitions)
172.16.238.13 - is ip of my first voldemort node running on docker.

sample record of my input after running it through avro tools:
{"key":"974","value":"107 1640 1000 924 1703 1010 1127 1134 1228 1236 1366"}


вторник, 26 апреля 2016 г., 0:51:15 UTC+3 пользователь Félix GV написал:

[Félix GV]

Actually, I just remembered (and confirmed in our logs) that reduce output records is always zero. That's normal, because BnP does not use a standard record collector in the reduce tasks (I think).

Your configs look fine... Can you post your job logs in a gist? (Attachments don't always work well on the mailing list)

Thanks!

-F

--

[Andrey Stanovsky]

Link to gist with the logs

Actual output folder /tmp/node0 has never been created ...

вторник, 26 апреля 2016 г., 23:50:13 UTC+3 пользователь Félix GV написал:

[Andrey Stanovsky]

Hmm, I was wrong, I've added my custom logging to check the hdfs in HadoopStoreBuilder.build() method and files are visible at that point... but when job fails, I cannot observe them through the web ui (at least)

среда, 27 апреля 2016 г., 0:05:51 UTC+3 пользователь Andrey Stanovsky написал:

[Félix GV]

Hmm...

You can set build.output.keep=true if you want to keep those files. Otherwise BnP cleans them up by default.

Can you check the logs on one of the servers? It may have more info on what's failing. In particular, maybe it doesn't have permission to read the output directory?

-F

[Arunachalam]

It feels like a permission issue. Can you run the fetch from the command line to see if you are able to fetch the files. This is a shell program we use to see if we have the right permissions.

#!/bin/sh

#

# Wrapper to fetch a file from HDFS.  Sets up all the parameters for

# a fetch from a box in an LVA1 production read-only cluster.

#

# Usage: hdfs-fetcher webhdfs://url

#

url="$1"

# Override the default JVM options to add the magic needed for Kerberos authentication

export VOLD_OPTS="-Xmx2G -server -Dcom.sun.management.jmxremote -Djava.security.krb5.realm=<Domain> -Djava.security.krb5.kdc=<Kdc> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log "

# Attach NetBeans debugger, if you are interested.

# -Xdebug -Xrunjdwp:transport=dt_socket,address=8800,server=y,suspend=y

# Configure your config dirs, the dir in our case contains a keytab file and a directory hadoop with 2 xmls.

vold_config_dir=<Config Dir that contains key tab and hadoop xmls>

voldemort-1.10.14/bin/run-class.sh \

    voldemort.store.readonly.fetcher.HdfsFetcher \

    "$url" \

    $vold_config_dir/<UserKeyTabFile> \

    <UserName> \

    $vold_config_dir/hadoop \

    <Directory>/download-`date +%s`

[Andrey Stanovsky]

[21:56:23,351 voldemort.server.protocol.admin.ReadOnlyStoreFetchOperation] WARN File fetcher class has not instantiated correctly. Assuming local file [voldemort-scheduler-service1-t3; AsyncOp ID 0]
[21:56:23,370 voldemort.server.protocol.admin.AdminServiceRequestHandler] ERROR handleAsyncStatus failed for request(request_id: 0) [voldemort-admin-server-t2]
voldemort.VoldemortException: Fetch url webhdfs://172.16.238.10:50070/tmp/node0 is not readable
        at voldemort.server.protocol.admin.ReadOnlyStoreFetchOperation.operate(ReadOnlyStoreFetchOperation.java:77)
        at voldemort.server.protocol.admin.AsyncOperation.run(AsyncOperation.java:35)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Arun, ok I'll try that one

среда, 27 апреля 2016 г., 0:50:37 UTC+3 пользователь Félix GV написал:

[Félix GV]

On Tue, Apr 26, 2016 at 3:00 PM, Andrey Stanovsky <sand...@gmail.com> wrote:

[21:56:23,351 voldemort.server.protocol.admin.ReadOnlyStoreFetchOperation] WARN File fetcher class has not instantiated correctly. Assuming local file [voldemort-scheduler-service1-t3; AsyncOp ID 0]

That part of the log is interesting. It seems like there is a set up issue...

Can you look in the rest of your server logs to see what's going on with your file fetcher initialization? Are you running Hadoop with Kerberos security? If you are NOT using a secure Hadoop installation, then Voldemort needs to be told explicitly to disable security. This is done in the Hadoop xml configs passed to the Voldemort servers by setting hadoop.security.authentication=simple

-F

--
--
Félix

[Andrey Stanovsky]

Hmm, do I need to provide hadoop configs to voldemort ? And how to specify the path to it? Which variable do I need to export and what files is Voldemort expecting?

среда, 27 апреля 2016 г., 1:00:48 UTC+3 пользователь Andrey Stanovsky написал:

[Félix GV]

Yes, you do,

We should make sure the full list of required configs is properly spelled out on the website, which is not the case currently...

That config is: readonly.hadoop.config.path

-F

--
--
Félix

[Andrey Stanovsky]

Ok
readonly.hadoop.config.path - does it come to the Voldemort's server.properties?

About your previous questions:
No security is enabled, since it's just a testing project and I need simple POC running first =)
ok, so I need to add this guy too:

hadoop.security.authentication=simple

Cool, I like when there is some path to go )
Btw, I've created a gist with full Voldemort node log

среда, 27 апреля 2016 г., 1:22:47 UTC+3 пользователь Félix GV написал:

-F

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

[Félix GV]

readonly.hadoop.config.path belongs in server.properties

hadoop.security.authentication belongs in core-site.xml

[Arunachalam]

Andrey, one thing we would appreciate if you are able to make this work is, if you can help us re-write some of the wiki documentation for read only build and push. 

We run the build and push inside a special config and we have lost track of what are the changes required for making it run from a vanilla shell.

You can modify the docs and send the pull request on gh-page, once you make things work.

https://github.com/voldemort/voldemort/commit/8bba74552f22843886ec0c40dc3b8d3b53222772

Thanks,

Arun.

[Andrey Stanovsky]

Ok, now it throws "invalid keytab file path" exception... there is some progress here)

But it rises a question, since I'm not too awared of the kerberos and keytabs, do I need one, if my security is off ? 

среда, 27 апреля 2016 г., 1:44:30 UTC+3 пользователь Félix GV написал:

-F

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

[Arunachalam]

Can you set the following property to empty string ?

readonly.keytab.path=

If not set to empty string, it assumes it to be config directory + voldemrt.headless.keytab

Thanks,

Arun.

On Tue, Apr 26, 2016 at 3:49 PM, Andrey Stanovsky <sand...@gmail.com> wrote:

Ok, now it throws "invalid keytab" exception... there is some progress here)

But it rises a question, since I'm not to awared of the kerberos and keytabs, do I need one, if my security is off ? 

[Andrey Stanovsky]

You are amazing guys!!! Thanks a lot, it works! =) Now I can go to bed, in my TZ it's 2 AM 

среда, 27 апреля 2016 г., 1:53:00 UTC+3 пользователь Arun Thirupathi написал:

-F

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Arunachalam]

Can you update the document to help future people like you ? Not urgent, but even little fixing of the document will go a long way.

Thanks,

Arun.

[Félix GV]

Hehe congrats!

I'm glad to see Voldemort landed in Russia ;)

Доброй ночи (:

[Andrey Stanovsky]

I'm from Ukraine, but nevertheless =)

Arun, ok I will try to update the doc if ypu provide me more details about where and how )

среда, 27 апреля 2016 г., 2:02:56 UTC+3 пользователь Félix GV написал:

-F

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Arunachalam]

Fork the github repo.

https://github.com/voldemort/voldemort/tree/gh-pages

Modify the file (in your repo)

https://github.com/voldemort/voldemort/blob/gh-pages/build-and-push.html

This is the file, that powers the page.

http://www.project-voldemort.com/voldemort/build-and-push.html

and send us a pull request. If that is too much. Just send us the step by step of all the changes you have to do, common pitfalls and we can update the documentation as well.

Thanks,

Arun.

--
Félix

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--

[Félix GV]

I'm sorry :/ ... I got confused by the Cyrillic-looking script :/ !

I hope to hear more about your experiments with V-RO (:

--
Félix

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--

[Andrey Stanovsky]

Felix, no worries)
Arun, I think I will cope with PR if I coped with BnP :))

[Félix GV]

Haha XD ...

Another handy trick with the gh-pages branch is that you can push changes to the branch in your clone of the repo, and view what it looks like online with the styling and everything. For example, this is my copy of the gh-pages: http://felixgv.github.io/voldemort/build-and-push.html (just replace the subdomain with your GitHub username)

One caveat when doing this is that it takes a minute or so for GitHub to regenerate your pages after you push, so the changes are not immediately visible.

-F

On Tue, Apr 26, 2016 at 11:12 PM, Andrey Stanovsky <sand...@gmail.com> wrote:

Felix, no worries)
Arun, I think I will cope with PR if I coped with BnP :))

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

[Arunachalam]

Andrey,

     Just a short summary will do as well, I will work on making the edits, if you are busy. It is sad to see every one partner has to jump these hoops which we don't have an handle of.

Thanks,

Arun.

[bholbi...@gmail.com]

Hi Arun,

Can i know how to build a shadowJar.Which platform is needed for this one.Is this a user defined jar or predefined jar.I just want to implement in my project.

The problem in my project like when i am pushing the data from hdfs to Voldemort,still the output is in HDFS.

Please help me.

Regards,

Binaya

On Saturday, April 23, 2016 at 2:11:43 PM UTC-7, Andrey Stanovsky wrote:

Hello,

I'm working on a test project that should push avro data from hdfs to a voldemort read-only cluster. I've faced a lot of obstacles and did not succeed so far, I was stopped by the following thing:

 java.lang.IllegalArgumentException: Should not get various nodeId shuffled to us! First nodeId seen: 0, currentNodeId: 2 

How do I need to specify to push only into one node or I misunderstood the logic there? I thought, in theory, job should distribute data evenly by its own logic. Or data about nodeId and partition  should be placed into an avro file ? There is no too much material about that in the internet. I can provide you with all I have configured for voldemort or azkaban build and push job.

[Arunachalam]

./gradlew bnpJar

after this command you will find the bnp jar here.

./build/libs/voldemort-1.10.16-bnp.jar

Thanks,

Arun.

--

[Arunachalam]

Andrey,

     One last ping, or has too much time passed by now ?

Thanks,

Arun.

[Andrey Stanovsky]

Arun,

  I'm sorry for not replying in time, too much work and PhD consumed all my time and focus.

If it's still relevant I could try to write small guide of starting BnP from the vanilla shell.

четверг, 26 мая 2016 г., 21:39:31 UTC+3 пользователь Arun Thirupathi написал:

Andrey,

     One last ping, or has too much time passed by now ?

Thanks,

Arun.

On Mon, May 9, 2016 at 1:57 PM, Arunachalam <arunac...@gmail.com> wrote:

Andrey,

     Just a short summary will do as well, I will work on making the edits, if you are busy. It is sad to see every one partner has to jump these hoops which we don't have an handle of.

Thanks,

Arun.

On Wed, Apr 27, 2016 at 9:12 AM, Félix GV <fel...@gmail.com> wrote:

Haha XD ...

Another handy trick with the gh-pages branch is that you can push changes to the branch in your clone of the repo, and view what it looks like online with the styling and everything. For example, this is my copy of the gh-pages: http://felixgv.github.io/voldemort/build-and-push.html (just replace the subdomain with your GitHub username)

One caveat when doing this is that it takes a minute or so for GitHub to regenerate your pages after you push, so the changes are not immediately visible.

-F

On Tue, Apr 26, 2016 at 11:12 PM, Andrey Stanovsky <sand...@gmail.com> wrote:

Felix, no worries)
Arun, I think I will cope with PR if I coped with BnP :))

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Arunachalam]

That will be awesome, if you can write it.

Thanks,

Arun.

On Sat, Jul 23, 2016 at 12:34 AM, Andrey Stanovsky <sand...@gmail.com> wrote:

Arun,

  I'm sorry for not replying in time, too much work and PhD consumed all my time and focus.

If it's still relevant I could try to write small guide of starting BnP from the vanilla shell.

четверг, 26 мая 2016 г., 21:39:31 UTC+3 пользователь Arun Thirupathi написал:

Andrey,

     One last ping, or has too much time passed by now ?

Thanks,

Arun.

On Mon, May 9, 2016 at 1:57 PM, Arunachalam <arunac...@gmail.com> wrote:

Andrey,

     Just a short summary will do as well, I will work on making the edits, if you are busy. It is sad to see every one partner has to jump these hoops which we don't have an handle of.

Thanks,

Arun.

On Wed, Apr 27, 2016 at 9:12 AM, Félix GV <fel...@gmail.com> wrote:

Haha XD ...

Another handy trick with the gh-pages branch is that you can push changes to the branch in your clone of the repo, and view what it looks like online with the styling and everything. For example, this is my copy of the gh-pages: http://felixgv.github.io/voldemort/build-and-push.html (just replace the subdomain with your GitHub username)

One caveat when doing this is that it takes a minute or so for GitHub to regenerate your pages after you push, so the changes are not immediately visible.

-F

On Tue, Apr 26, 2016 at 11:12 PM, Andrey Stanovsky <sand...@gmail.com> wrote:

Felix, no worries)
Arun, I think I will cope with PR if I coped with BnP :))

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Andrey Stanovsky]

Hi again,

I've made a draft of the guide, please take a look

LINK

вторник, 26 июля 2016 г., 8:05:14 UTC+3 пользователь Arun Thirupathi написал:

Haha XD ...

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--
--
Félix

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.