goko logs

[Max Gibiansky]

Hey guys. So I think I'd like to join in councilroom development and help implement goko log parsing. A thread on f.ds (http://forum.dominionstrategy.com/index.php?topic=4185.0 ) pointed me to this group and suggested I say hi. 

Is anyone working on that already? Just checking so I don't end up stepping on anyone's toes/duplicating work. 

[Michael McCallister]

Hey Max,

The user named "philosophyguy" on f.ds was working on some goko log parsing. And based on what GitHub tells me, it looks like there is some activity on the "parse-goko" branch of https://github.com/michaeljb/dominionstats. I don't know if that's the same person or not.

GitHub is the preferred way to collaborate in code, and it has the issue tracker. This group is probably the best way to reach the largest number of people who have some development interest in the site.

I've been a bit tuned out of this lately, but I'd be happy to get back into it and help. Let me know what you need.

Mike

On Wed, May 1, 2013 at 4:54 AM, Max Gibiansky <maxsi...@gmail.com> wrote:

Hey guys. So I think I'd like to join in councilroom development and help implement goko log parsing. A thread on f.ds (http://forum.dominionstrategy.com/index.php?topic=4185.0 ) pointed me to this group and suggested I say hi. 

Is anyone working on that already? Just checking so I don't end up stepping on anyone's toes/duplicating work. 

--
You received this message because you are subscribed to the Google Groups "Councilroom.com development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to councilroom-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Max Gibiansky]

Great!  I've forked the project on github and gotten in touch with both michaeljb and philosophyguy. Thanks!

[Max Gibiansky]

Oh, oops, I did have a question but I mis-sent it and it didn't go to councilroom-dev. Resending.

So I see two separate ways that cr code scrapes logs - one is through scrape.py, called from update-loop.py, and the second is through update.py which sets a background task with an IsotropicScraper . What's the deal with that? Which one is used in production? Or are they both used? I didn't get how they related to each other. 

The one that deals with S3 buckets is one that I won't be able to rewrite and test on my own, since I don't have access to the S3 instance... well, I'll figure out how to deal with that later I suppose. Writing the parsing code for now.

On Wed, May 1, 2013 at 9:17 AM, Michael McCallister <mi...@mccllstr.com> wrote:

[Michael McCallister]

Hey Max,

Sorry for the slow reply.

The production site has been using the update_runner.sh script. It, in turn, runs the update.py script once and then queues itself up to be run again on a daily basis. The update.py script was a re-implementation of update_loop.py as I worked to make it more parallelizable. You can get a sense of its history by looking at the git logs. I've tried to be verbose in them.

In terms of the S3 buckets, I think your concern is how to insert the scrapped Gogo logs into them? If so, I can get you a set of credentials to use for development

Mike

[Max Gibiansky]

Thanks!

Yep, with the s3 buckets my concern is to make sure that I don't break anything there when editing it a bit to put in goko logs. The part that deals with the s3 buckets should remain unchanged but, well, I can't really be sure I've got everything working right unless I try it. Development credentials would help but there's no hurry, there's a lot to be done with the parsing, I'm still working on that and am working with local stuff at the moment.

[Max Gibiansky]

Just an update, I finally finished goko-log parsing. Took a while because some of the things that iso displayed in obvious ways are not explicitly stated on goko and needed to be inferred with card-specific logic (like, Band of Misfits as a self-trasher, or whether cities are activated to level 3, etc.) Has some things that iso didn't,  it'll keep track of Masq passing properly and such.

Next up is getting the scraping working in bulk with update_runner.sh/update.py. I think I'll just make myself an s3 account so I can play around with it freely until I get  everything working right, so I don't need to worry about messing up currently-running things. (Or I suppose appropriately safeguarded credentials on your current setup, depending on how easy that is for you to set up).

[Andrew Iannaccone]

Hehe.  I think there are at least five people producing Goko log parsing code.  I suppose that's because it's a prerequisite to pretty much everything else.  Hopefully things will widen up a bit once the basics are in place.

I've got a reasonable bulk downloader working using tornado for asynchronous requests... keeps up with new logs goko adds in real time and reasonably zippy for downloading existing logs too.  Unfortunately, I have no idea where the current version of this project is supposed to be living.  Is there a single fork of councilroom on github that everyone is working with?

I'm pretty new to collaboration on github.  I'd be grateful if someone could help me get started or point me towards a useful guide.

Thanks,

Andrew

[Max Gibiansky]

Are there really that many people? When I checked here a month or so there wasn't anyone actively working on it. Well, maybe not for councilroom, maybe elsewhere. Anyway. Current production councilroom code is at https://github.com/mikemccllstr/dominionstats , I forked it at https://github.com/ftlftw/dominionstats and have been working on updating it to work with goko, I'll see about reintegrating it back in when it's working. I don't think anyone else is actively working on councilroom code right now? As far as I know?

I'd love to take a look at the downloader code you have, I did it in I guess a naive way and it's glacially slow.

I don't remember how I got started with github, I just googled like "how to use git" and "how to use github". It's pretty simple.

[Andrew Iannaccone]

Well, seeing as you're ftl, that's at least one fewer!  Also serakfalcon and michaeljb, I believe.  And me, but mine is crap... just the game results and supply card piles.

Serakfalcon is working in PHP.  He has his code on github too: https://github.com/serakfalcon/DominionCoding/

I'll see if I can figure out how to upload my code on github.  It takes down something like 30 logs per second, IIRC.

--Andrew

[Andrew Iannaccone]

Correction... my downloader/parser only gets about 10 logs per second.  I can do it at 30 if I first download the whole directory using nutki's xargs script (~50 logs/s).

The limiting factor is Goko.  I can send them 30-100 requests simultaneously, but they just won't give me back more than 10 logs/s.  Nutki's script asks them to send the logs gzipped, so maybe that's the difference.

--Andrew

--
Andrew Iannaccone
andrew.i...@gmail.com