TREC KBA --- corpus released, queries about to be released

[John R. Frank]

KBAers,

The KBA 2012 corpus has been posted.  We recommend joining the streamcorpus google group for info about the corpus, which will be used by multiple TREC tracks.

      https://groups.google.com/d/msg/streamcorpus/7SYc3_pDcXY/ZfeTHDYPu9QJ

We plan to post query entities and target slots tomorrow.

v0_2_0 of the corpus is 6.45TB after XZ compression.  One recommended approach to working with the corpus is to use the KBA target entities to filter as you download instead of saving all 6.45TB locally.  While such a filter can impact recall, we have observed that essentially all highly relevant documents directly mention some name of a target entity.  This significantly reduces the total document set required for KBA processing.  We look forward to hearing about your approaches to dealing with this data.

More soon!

The KBA Organizers

[Dai Zhang]

Hi,John

We now have some difficulties in downloading the corpus, for the network environment here (Beijing,China) is quite pooooor. In the past two days, we have only downloaded less than 20G corpus files, which means that we have to take nearly a whole year to get the new data in 2013 corpus. It makes us anxious and frustrated .

Do you have any solutions for this? We hope to hear from you soon.

Thanks a lot.

在 2013年4月7日星期日UTC+8下午2时08分54秒，John R. Frank写道：

[John R. Frank]

> In the past two days, we have only downloaded less than 20G corpus
> files, which means that we have to take nearly a whole year to get the
> new data in 2013 corpus.

Two steps toward solution:

1) We are creating a smaller version of the corpus that has only
English-and-unknown-language documents and has StreamItem.body.raw removed
if StreamItem.body.clean_visible is present. This will be considerably
smaller. This should be ready for download from S3 in the next ~48hrs.

2) If network transfer from S3 is problematic, then a good alternative is
AWS Export, which allows you to buy a physical disk, ship it to Amazon
staff with a barcode that instructs them to copy a particular part of S3
on to the disk, and ship it back to you:

http://aws.amazon.com/importexport/

I have used AWS Export several times, and it is straightforward. Just
follow the instructions carefully. If you run into difficulties with
this, please contact me offlist.


Regards,
John

[John R. Frank]

>> In the past two days, we have only downloaded less than 20G corpus
>> files, which means that we have to take nearly a whole year to get the
>> new data in 2013 corpus.
>
> Two steps toward solution:
>
> 1) We are creating a smaller version of the corpus that has only
> English-and-unknown-language documents and has StreamItem.body.raw
> removed if StreamItem.body.clean_visible is present. This will be
> considerably smaller. This should be ready for download from S3 in the
> next ~48hrs.

I forgot to mention the BEST way to cope with the large corpus: use EC2!

For $0.24 per day, you can run a t1.micro instance in EC2 to test out your
tools for filtering. Then, to process the whole corpus and generate a
high-recall excerpt, you can either make a couple bigger instances for a
few days or use EMR.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html

http://aws.amazon.com/elasticmapreduce/



We'll let you know when the clean_visible/English excerpt is done.


jrf

[Jingang Wang]

Hi John,

You mean, in kba 2013, you will provide a clean_visible/English excerpt corpus for the participants?

--
You received this message because you are subscribed to the Google Groups "TREC-KBA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Wang Jingang(王金刚)

Ph.D. Candidate at  

Lab of High Volume Language Information Processing & Cloud Computing

School of Computer Science

Beijing Institute of Technology

Beijing 100081

P.R China

[John R. Frank]

> You mean, in kba 2013, you will provide a clean_visible/English excerpt
> corpus for the participants?

Yes. Job is loading now. Should finish in a day or two.

jrf

[Jingang Wang]

Great news! It would save a lot of hardware resource for us.

Thanks all the organizers' contribution.

Would you mind post it in the google group after the work is done? If yes, we could begin to download the new data.

jrf

--
You received this message because you are subscribed to the Google Groups "TREC-KBA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dai Zhang]

It's very kind of you for helping us so much, we will discuss about it and make a decision .Thank you .

在 2013年4月9日星期二UTC+8下午8时27分35秒，John R. Frank写道：

[Praveen Kumar]

John, this is from last year's KBA. Can you add dir-names.txt for this years corpus after the clean english corpus is uploaded, so that it is easier to download the corpus like last time.

• You can also retrieve the corpus using wget, like this:

  ## Fetch the list of directory names -- date-hour strings
  wget http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-stream-corpus-2012/dir-names.txt
  		    
  ## Use GNU parallel to make multiple wget requests in parallel.  
  ## The --continue flag makes this restartable.
  cat dir-names.txt | parallel -j 10 --eta 'wget --recursive --continue --no-host-directories --no-parent --reject "index.html*" http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-stream-corpus-2012/{}/index.html'
  	      

  
  

[John R. Frank]

This file is now uploaded:

http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0/dir-names.txt


I will ask NIST to update the web page info below.

Thanks for prompting me to do this.


jrf

> John, this is from last year's KBA. Can you add dir-names.txt for this years corpus after the clean english corpus is uploaded, so that it is easier to download the corpus like last time.

> * You can also retrieve the corpus using wget, like this:

>
> ## Fetch the list of directory names -- date-hour strings
> wget http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-stream-corpus-2012/dir-names.txt
>
> ## Use GNU parallel to make multiple wget requests in parallel.
> ## The --continue flag makes this restartable.
> cat dir-names.txt | parallel -j 10 --eta 'wget --recursive --continue --no-host-directories --no-parent --reject "index.html*" http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-stream-corpus-20
> 12/{}/index.html'
>
>

> --
> You received this message because you are subscribed to the Google Groups "TREC-KBA" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+u...@googlegroups.com.

> �
> �
>
>

[Praveen Kumar]

Thanking you, Jon!

Can you kindly give an rough estimate of how big the cleansed corpus is?

[Praveen Kumar]

Hi,

I have couple of questions.

1) Has the cleansed corpus been uploaded? If yes, where can find the link to download it?

2) Is this link http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0/dir-names.txt of directory names for the cleansed or non-cleansed corpus? Is it is for non-cleansed, would it be possible to upload a similar directory file be uploaded for the cleansed corpus as well?

Praveen

[John R. Frank]

> 1) Has the cleansed corpus been uploaded? If yes, where can find the
> link to download it?

Not yet. We lost two zookeeper instances last week, so had to restart.
It is about 75% done now, so probably another day or so. Will post when
it is done and verified.

> 2) Is this
> link�http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0/dir-names.txt�of
> directory names for the cleansed or non-cleansed corpus? Is it is for
> non-cleansed, would it be possible to upload a similar directory file be
> uploaded for the cleansed corpus as well?

The directory names are the same, but the prefix path changes:

http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0-english-and-unknown-language/

You could start partial fetching now, however it has not been verified.


jrf

[zhangchu...@gmail.com]

王博士：

      你好。我是北邮模式识别实验室Web搜索方向的博士，我们参加了今年的KBA评测，在google group中看到您的留言，知道您也参加今年的KBA评测，所以有个关于数据集的下载的问题请教您一下。我们这边的下载速度比较慢，想请问一下你们的数据集是如何获得的，下载速度怎么样？

      盼回复。

      祝好。

                                                       张春云
在 2013年4月9日星期二UTC+8下午8时31分05秒，Jingang Wang写道：

Hi John,

You mean, in kba 2013, you will provide a clean_visible/English excerpt corpus for the participants?

On Tue, Apr 9, 2013 at 8:27 PM, John R. Frank <j...@mit.edu> wrote:

In the past two days, we have only downloaded less than 20G corpus files, which means that we have to take nearly a whole year to get the new data in 2013 corpus.

Two steps toward solution:

1)  We are creating a smaller version of the corpus that has only English-and-unknown-language documents and has StreamItem.body.raw removed if StreamItem.body.clean_visible is present.  This will be considerably smaller. This should be ready for download from S3 in the next ~48hrs.

I forgot to mention the BEST way to cope with the large corpus:  use EC2!

For $0.24 per day, you can run a t1.micro instance in EC2 to test out your tools for filtering.  Then, to process the whole corpus and generate a high-recall excerpt, you can either make a couple bigger instances for a few days or use EMR.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html

http://aws.amazon.com/elasticmapreduce/



We'll let you know when the clean_visible/English excerpt is done.


jrf

--
You received this message because you are subscribed to the Google Groups "TREC-KBA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Amy]

Hi,

    Has anybody ever download the new cleansed corpus? If you did, please tell me how much is it.

Thanks

在 2013年4月7日星期日UTC+8下午2时08分54秒，John R. Frank写道：

KBAers,

[John R. Frank]

>     Has anybody ever download the new cleansed corpus? If you did,
> please tell me how much is it.

Approximately 4.6TB

See this post for how to work with it

https://groups.google.com/d/msg/streamcorpus/fi8Y8yseF8o/viJjiFNVLNsJ



jrf

[Jingang Wang]

Hi John,

I can't open the link for cleansed version of the stream corpus.

http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0-english-and-unknown-language//dir-names.txt

Have the cleansed corpus uploaded done yet？

Thanks.

--
You received this message because you are subscribed to the Google Groups "TREC-KBA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[John R. Frank]

> Have the cleansed corpus uploaded done yet？

Try these commands:

wget http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0/dir-names.txt

cat dir-names.txt | parallel -j 10 --eta 'wget --recursive --continue --no-host-directories --no-parent --reject "index.html*" http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0-english-and-unknown-language/{}/index.html' >& log &


I just generated those index files. There are about 4000 chunk files
missing, and will be finished in the next day or so. I will post when
they are done.

This is about 4.6TB and a bit over 500M files.

We could drop another TB or maybe two off the size by more aggressive
pruning of parts of the data. We'll look at this --- give us feedback on
how downloading and initial filtering is working for you. Is anyone using
AWS?


jrf

[hn]

Hi John,

 

I can successfully decrypt and unzip the file. The last step is deserializing. After deserialize, I can read the data, but the only problem I have is that I see a lot of \n and \t in the file.

Here the deserialize code, can you give advice.

 

def deserialize_path(filename):

    chunk = Chunk(path=filename)

    mesgs = list(chunk)

    itemcount = len(mesgs)

 

    outfilename = filename + ".txt"

    file = open(outfilename, 'a')

    for i in range(0, itemcount):

        file.write(str(mesgs[i])

    file.close()

 

 

Thanks,

Hung

[John R. Frank]

> I can successfully decrypt and unzip the file. The last step is
> deserializing. After deserialize, I can read the data, but the only
> problem I have is that I see a lot of \n and \t in the file.

> Here the deserialize code, can you give advice.
> �
> def deserialize_path(filename):
> ��� chunk = Chunk(path=filename)
> ��� mesgs = list(chunk)

This step is doing the equivalent of repr'ing all the StreamItem messages,
which is not what you want:

mesgs = [repr(si) for si in chunk]


Thrift messages are in-memory objects with properties. For the
streamcorpus thrift messages, there are many properties.

Here is an example of accessing the many properties in the streamcorpus
format and the KBA 2013 data:

https://github.com/trec-kba/streamcorpus/blob/master/examples/py/iterating-over-tokens.py

You can read all of the properties available in a streamcorpus.StreamItem
by reading this thrift interface definition file (read from the bottom
up):

https://github.com/trec-kba/streamcorpus/blob/master/if/streamcorpus.thrift


John

[Jingang Wang]

Hi Praveen,

You could find the list of all directory names here http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0-english-and-unknown-language/index.html

On Thu, May 2, 2013 at 11:22 AM, Praveen Kumar <gpravee...@gmail.com> wrote:

When I use the following command to get the list of directories, 

wget http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0-english-and-unknown-language/dir-names.txt

I get the error,

--2013-05-01 23:18:41--  http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0-english-and-unknown-language/dir-names.txt

Resolving s3.amazonaws.com... 72.21.194.31

Connecting to s3.amazonaws.com|72.21.194.31|:80... connected.

HTTP request sent, awaiting response... 404 Not Found

2013-05-01 23:18:42 ERROR 404: Not Found.

But, this works fine

wget http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0/dir-names.txt

Can you please check.

Thanks,

Praveen

--
You received this message because you are subscribed to the Google Groups "TREC-KBA" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trec-kba+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[John R. Frank]

You can also use this dir-names.txt file; it is the same.

wget http://s3.amazonaws.com/aws-publicdatasets/trec/kba/kba-streamcorpus-2013-v0_2_0/dir-names.txt


jrf

[Praveen Kumar]

Thanks, Wang and John!

On Sunday, 7 April 2013 02:08:54 UTC-4, John R. Frank wrote: