Missing Reports

[Barbie]

Hi folks,

I forgot to include the number of missing reports in my summary I posted
yesterday. For September alone this was 496784. That's just under half a
million reports. That's how bad Amazon SimpleDB has got.

There was a bit of a hiccup with 10th October as result sets requested
between 14:00 and 14:15, were all returning reports posted for around
04:00. I've now skipped these and will be going back to them later. Now
catching up from the last few weeks.

We should be back on track by next week.

Cheers,
Barbie.
--
Birmingham Perl Mongers <http://birmingham.pm.org>
Memoirs Of A Roadie <http://barbie.missbarbell.co.uk>
CPAN Testers Blog <http://blog.cpantesters.org>
YAPC Conference Surveys <http://yapc-surveys.org>
Ark Appreciation Pages <http://ark.eology.org>

[Andreas Koenig]

Barbie <bar...@missbarbell.co.uk> writes:

> Hi folks,
>
> I forgot to include the number of missing reports in my summary I posted
> yesterday. For September alone this was 496784. That's just under half a
> million reports. That's how bad Amazon SimpleDB has got.
>
> There was a bit of a hiccup with 10th October as result sets requested
> between 14:00 and 14:15, were all returning reports posted for around
> 04:00. I've now skipped these and will be going back to them later. Now
> catching up from the last few weeks.
>
> We should be back on track by next week.

Wonderful, thank you! I'm so much looking forward to the day.

Funny, I mentioned to Slaven today that your process was hung on Oct 10
and he said I should tell you because maybe you do not know. And when I
came home you had already blogged about it.

But maybe I should tell you about another interesting observation I have
not yet talked about: some cpantesters have wrong clocks. One tester's
watch is 3 hours and three minutes off, another one's about one hour. I
have seen several others only some minutes or some seconds wrong.
Possibly somebody in the whole setup is relying on the times contained
within the reports while others are relying on real time?

Oh my god, and on Oct 10th one tester had even different clocks on two
of his server. Four hours between them.

According to log.txt, tests arrived in this order:

[2012-10-10T07:46:48Z]
[2012-10-10T06:26:56Z]
[2012-10-10T07:47:12Z]
[2012-10-10T11:45:48Z]
[2012-10-10T11:46:20Z]
[2012-10-10T07:47:28Z]
[2012-10-10T06:27:17Z]
[2012-10-10T07:47:18Z]
[2012-10-10T07:47:32Z]
[2012-10-10T11:47:52Z]
[2012-10-10T06:27:19Z]
[2012-10-10T14:08:46Z]
[2012-10-10T06:26:37Z]
[2012-10-10T11:48:20Z]
[2012-10-10T11:49:29Z]
[2012-10-10T07:47:43Z]
[2012-10-10T11:49:49Z]

Is this interesting?

--
andreas

[David Golden]

On Wed, Nov 7, 2012 at 3:44 PM, Andreas Koenig
<andreas.koe...@franz.ak.mind.de> wrote:
>
> But maybe I should tell you about another interesting observation I have
> not yet talked about: some cpantesters have wrong clocks. One tester's
> watch is 3 hours and three minutes off, another one's about one hour. I
> have seen several others only some minutes or some seconds wrong.
> Possibly somebody in the whole setup is relying on the times contained
> within the reports while others are relying on real time?

Because some people run offline and then submit, the log could have
times in any order. Of course, people might just have clocks that are
off, too.

We pretty much just accept whatever time submitters claim -- just like
we accept whatever *ELSE* submitters claim.

David


--
David Golden <x...@xdg.me>
Take back your inbox! → http://www.bunchmail.com/
Twitter/IRC: @xdg

[Andreas Koenig]

David Golden <x...@xdg.me> writes:

> On Wed, Nov 7, 2012 at 3:44 PM, Andreas Koenig
> <andreas.koe...@franz.ak.mind.de> wrote:
>>
>> But maybe I should tell you about another interesting observation I have
>> not yet talked about: some cpantesters have wrong clocks. One tester's
>> watch is 3 hours and three minutes off, another one's about one hour. I
>> have seen several others only some minutes or some seconds wrong.
>> Possibly somebody in the whole setup is relying on the times contained
>> within the reports while others are relying on real time?
>
> Because some people run offline and then submit, the log could have
> times in any order. Of course, people might just have clocks that are
> off, too.
>
> We pretty much just accept whatever time submitters claim -- just like
> we accept whatever *ELSE* submitters claim.

What about adding a date field for the timestamp when the report was
received by metabase? Would it not provide a chance to sanitize the
chain? Barbie could query for a field with guaranteed monotonously
increasing values, and those reports with timestamps in the future could
be silently corrected.

--
andreas

[David Golden]

On Tue, Dec 4, 2012 at 1:57 AM, Andreas Koenig
<andreas.koe...@franz.ak.mind.de> wrote:
> What about adding a date field for the timestamp when the report was
> received by metabase? Would it not provide a chance to sanitize the
> chain? Barbie could query for a field with guaranteed monotonously
> increasing values, and those reports with timestamps in the future could
> be silently corrected.

The number one thing to do is get off AWS. That might solve enough of
the problems.

We've finally been offered some servers and things are in progress to
deliver them, but it has been and continues to be a slow process.
Maybe January unless there are more delays.

After that, we can look into ways of improving the replication process
along those lines. Since the current plan is to migrate to MongoDB
(mst be damned), their object IDs are already monotonic, so we might
get indexed insertion date ordering for free.