Use of DataStream with different data subsets for different epochs

[hoshi...@gmail.com]

Hey everyone,

For one of my projects, I have a validation dataset that contains about 1 million entries. It would take way too much time to validate on that entire dataset for each validation, so I've been wondering: Is there a way to make the DataStream return a subset of that dataset for each epoch?

For example: For each validation, I want to use a subset of 1000 validation entries. For my first validation, the first epoch that the DataStream returns should contain the entries [0, ..., 999], the second one for the second validation should contain the entries [1000, ..., 1999], and so on. If possible, the batch size should be freely configurable as well.

I've been studying the API and experimenting with different IterationSchemes and DataStreams, but I've not been able to find a solution.

Apologies if it's something really simple that I've overseen, and thanks in advance!

[David Warde-Farley]

I'd actually thought about this problem before, and how it'd be nice to have, and so I went ahead and implemented it. It's pretty straightforward. (Python 3.x only, shouldn't be hard to port to Python 2 as I've noted in the comments.)

Basically, construct your stream however you want, with whatever batch sizes etc., then wrap it in this thing, telling it how many batches (or examples, if the wrapped stream produces single examples) you want to constitute an "epoch".

https://gist.github.com/dwf/e3a811a9463c802c5b5e4248806de1d1

--
You received this message because you are subscribed to the Google Groups "fuel-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fuel-users+unsubscribe@googlegroups.com.
To post to this group, send email to fuel-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fuel-users/01308b64-08ee-4118-91f8-219a053b718c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[hoshi...@gmail.com]

I'll have a look at it - thanks!