__scatter__ property
276 views
Skip to first unread message
Stephen Johnson
Jan 11, 2011, 12:32:38 PM1/11/11
to Google App Engine
Checking my datastore statistics today I noticed that a new property
has magically appeared named __scatter__ of type ShortBlob. I don't
have any property defined with this name and it has the App Engine
format of __xyz__ naming so I'm assuming it's something Google has
introduced. However, this is a little disconcerting that it is showing
up in datastore statistics and no mention of this has been provided by
google. Also, it doesn't show up in the datastore viewere. Is anyone
else seeing this magic property? Should I be concerned? Anyone know
it's purpose?
has magically appeared named __scatter__ of type ShortBlob. I don't
have any property defined with this name and it has the App Engine
format of __xyz__ naming so I'm assuming it's something Google has
introduced. However, this is a little disconcerting that it is showing
up in datastore statistics and no mention of this has been provided by
google. Also, it doesn't show up in the datastore viewere. Is anyone
else seeing this magic property? Should I be concerned? Anyone know
it's purpose?
Ikai Lan (Google)
Jan 11, 2011, 1:42:24 PM1/11/11
to Google App Engine
It's a property used by the Mapper API:
http://www.google.com/codesearch?q=__scatter__+package:http://appengine-mapreduce\.googlecode\.com&origq=__scatter__&btnG=Search+Trunk
--
--
Ikai Lan
Developer Programs Engineer, Google App Engine
Developer Programs Engineer, Google App Engine
Blogger: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Stephen Johnson
Jan 11, 2011, 2:06:32 PM1/11/11
to google-a...@googlegroups.com
Thanks Ikai,
I've been using mapreduce for a long time and have never seen properties added to our own entities like that. I'll take a look and see what it's being used for so I understand it better and if this side-effect is documented anywhere since it is strange and a little disconcerting to see unknown properties popping up on your own entities.
Thanks,
Steve
Ikai Lan (Google)
Jan 11, 2011, 3:29:58 PM1/11/11
to Google App Engine
The scatter property is being used to better distribute work across different worker shards. Prior to the scatter property, there were several incidents where work was unevenly distributed across tasks.
--
--
Ikai Lan
Developer Programs Engineer, Google App Engine
Developer Programs Engineer, Google App Engine
Blogger: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
frew
Jan 11, 2011, 3:42:15 PM1/11/11
to Google App Engine
As Ikai said, this was added in support of making mapper more
efficient. It's supposed to be completely invisible, so its visibility
is a bug. Can you let me know where you're seeing it (admin console?
API?) and your appid? Also, in case you're worried, Google is picking
up the bill for this property, so it shouldn't affect your storage
costs any.
Thanks
-Fred
On Jan 11, 12:29 pm, "Ikai Lan (Google)" <ikai.l+gro...@google.com>
wrote:
> >>> .
> >> .
> > .
efficient. It's supposed to be completely invisible, so its visibility
is a bug. Can you let me know where you're seeing it (admin console?
API?) and your appid? Also, in case you're worried, Google is picking
up the bill for this property, so it shouldn't affect your storage
costs any.
Thanks
-Fred
On Jan 11, 12:29 pm, "Ikai Lan (Google)" <ikai.l+gro...@google.com>
wrote:
> The scatter property is being used to better distribute work across
> different worker shards. Prior to the scatter property, there were several
> incidents where work was unevenly distributed across tasks.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blogger:http://googleappengine.blogspot.com
> Reddit:http://www.reddit.com/r/appengine
> Twitter:http://twitter.com/app_engine
>
> different worker shards. Prior to the scatter property, there were several
> incidents where work was unevenly distributed across tasks.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blogger:http://googleappengine.blogspot.com
> Reddit:http://www.reddit.com/r/appengine
> Twitter:http://twitter.com/app_engine
>
> On Tue, Jan 11, 2011 at 11:06 AM, Stephen Johnson <onepagewo...@gmail.com>wrote:
>
>
>
>
>
>
>
> > Thanks Ikai,
> > I've been using mapreduce for a long time and have never seen properties
> > added to our own entities like that. I'll take a look and see what it's
> > being used for so I understand it better and if this side-effect is
> > documented anywhere since it is strange and a little disconcerting to see
> > unknown properties popping up on your own entities.
> > Thanks,
> > Steve
>
> > On Tue, Jan 11, 2011 at 11:42 AM, Ikai Lan (Google) <
> > ikai.l+gro...@google.com <ikai.l%2Bgro...@google.com>> wrote:
>
> >> It's a property used by the Mapper API:
>
> >>http://www.google.com/codesearch?q=__scatter__+package:http://appengi....googlecode\.com&origq=__scatter__&btnG=Search+Trunk<http://www.google.com/codesearch?q=__scatter__+package:http://appengi...>
>
>
>
>
>
>
>
> > Thanks Ikai,
> > I've been using mapreduce for a long time and have never seen properties
> > added to our own entities like that. I'll take a look and see what it's
> > being used for so I understand it better and if this side-effect is
> > documented anywhere since it is strange and a little disconcerting to see
> > unknown properties popping up on your own entities.
> > Thanks,
> > Steve
>
> > On Tue, Jan 11, 2011 at 11:42 AM, Ikai Lan (Google) <
> > ikai.l+gro...@google.com <ikai.l%2Bgro...@google.com>> wrote:
>
> >> It's a property used by the Mapper API:
>
>
> >> --
> >> Ikai Lan
> >> Developer Programs Engineer, Google App Engine
> >> Blogger:http://googleappengine.blogspot.com
> >> Reddit:http://www.reddit.com/r/appengine
> >> Twitter:http://twitter.com/app_engine
>
> >> --
> >> Ikai Lan
> >> Developer Programs Engineer, Google App Engine
> >> Blogger:http://googleappengine.blogspot.com
> >> Reddit:http://www.reddit.com/r/appengine
> >> Twitter:http://twitter.com/app_engine
>
> >> On Tue, Jan 11, 2011 at 9:32 AM, Stephen Johnson <onepagewo...@gmail.com>wrote:
>
> >>> Checking my datastore statistics today I noticed that a new property
> >>> has magically appeared named __scatter__ of type ShortBlob. I don't
> >>> have any property defined with this name and it has the App Engine
> >>> format of __xyz__ naming so I'm assuming it's something Google has
> >>> introduced. However, this is a little disconcerting that it is showing
> >>> up in datastore statistics and no mention of this has been provided by
> >>> google. Also, it doesn't show up in the datastore viewere. Is anyone
> >>> else seeing this magic property? Should I be concerned? Anyone know
> >>> it's purpose?
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Google App Engine" group.
> >>> To post to this group, send email to google-a...@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
>
> >>> Checking my datastore statistics today I noticed that a new property
> >>> has magically appeared named __scatter__ of type ShortBlob. I don't
> >>> have any property defined with this name and it has the App Engine
> >>> format of __xyz__ naming so I'm assuming it's something Google has
> >>> introduced. However, this is a little disconcerting that it is showing
> >>> up in datastore statistics and no mention of this has been provided by
> >>> google. Also, it doesn't show up in the datastore viewere. Is anyone
> >>> else seeing this magic property? Should I be concerned? Anyone know
> >>> it's purpose?
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Google App Engine" group.
> >>> To post to this group, send email to google-a...@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> .
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/google-appengine?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-a...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> >>>http://groups.google.com/group/google-appengine?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-a...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > .
Stephen Johnson
Jan 11, 2011, 4:22:55 PM1/11/11
to google-a...@googlegroups.com
Hi Fred,
I can see it when I go to Datastore Statistics page and choose an individual entity. On about half of my entities I can see the property and half I don't see the property. The first four of my entities alphabetically have it and the fifth doesn't. appid is my-lms-test
Steve
I can see it when I go to Datastore Statistics page and choose an individual entity. On about half of my entities I can see the property and half I don't see the property. The first four of my entities alphabetically have it and the fifth doesn't. appid is my-lms-test
Steve
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
frew
Jan 11, 2011, 4:58:41 PM1/11/11
to Google App Engine
Oh yeah, I see it now. I'll get it fixed.
On Jan 11, 1:22 pm, Stephen Johnson <onepagewo...@gmail.com> wrote:
> Hi Fred,
> I can see it when I go to Datastore Statistics page and choose an individual
> entity. On about half of my entities I can see the property and half I don't
> see the property. The first four of my entities alphabetically have it and
> the fifth doesn't. appid is my-lms-test
> Steve
>
>
>
>
>
>
>
> On Tue, Jan 11, 2011 at 1:42 PM, frew <f...@google.com> wrote:
> > As Ikai said, this was added in support of making mapper more
> > efficient. It's supposed to be completely invisible, so its visibility
> > is a bug. Can you let me know where you're seeing it (admin console?
> > API?) and your appid? Also, in case you're worried, Google is picking
> > up the bill for this property, so it shouldn't affect your storage
> > costs any.
>
> > Thanks
> > -Fred
>
> > On Jan 11, 12:29 pm, "Ikai Lan (Google)" <ikai.l+gro...@google.com<ikai.l%2Bgro...@google.com>
> > > >>> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib
On Jan 11, 1:22 pm, Stephen Johnson <onepagewo...@gmail.com> wrote:
> Hi Fred,
> I can see it when I go to Datastore Statistics page and choose an individual
> entity. On about half of my entities I can see the property and half I don't
> see the property. The first four of my entities alphabetically have it and
> the fifth doesn't. appid is my-lms-test
> Steve
>
>
>
>
>
>
>
> On Tue, Jan 11, 2011 at 1:42 PM, frew <f...@google.com> wrote:
> > As Ikai said, this was added in support of making mapper more
> > efficient. It's supposed to be completely invisible, so its visibility
> > is a bug. Can you let me know where you're seeing it (admin console?
> > API?) and your appid? Also, in case you're worried, Google is picking
> > up the bill for this property, so it shouldn't affect your storage
> > costs any.
>
> > Thanks
> > -Fred
>
> > ikai.l%2Bgro...@google.com <ikai.l%252Bgro...@google.com>>> wrote:
>
> > > >> It's a property used by the Mapper API:
>
> >http://www.google.com/codesearch?q=__scatter__+package:http://appengi....com&origq=__scatter__&btnG=Search+Trunk<http://www.google.com/codesearch?q=__scatter__+package:http://appengi...>
>
> > > >> It's a property used by the Mapper API:
>
> > e...@googlegroups.com>
> > > >>> .
> > > >>> For more options, visit this group at
> > > >>>http://groups.google.com/group/google-appengine?hl=en.
>
> > > >> --
> > > >> You received this message because you are subscribed to the Google
> > Groups
> > > >> "Google App Engine" group.
> > > >> To post to this group, send email to
> > google-a...@googlegroups.com.
> > > >> To unsubscribe from this group, send email to
> > > >> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib
> > > >>> .
> > > >>> For more options, visit this group at
> > > >>>http://groups.google.com/group/google-appengine?hl=en.
>
> > > >> --
> > > >> You received this message because you are subscribed to the Google
> > Groups
> > > >> "Google App Engine" group.
> > > >> To post to this group, send email to
> > google-a...@googlegroups.com.
> > > >> To unsubscribe from this group, send email to
> > e...@googlegroups.com>
> > > >> .
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/google-appengine?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to google-a...@googlegroups.com
> > .
> > > > To unsubscribe from this group, send email to
> > > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib
> > > >> .
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/google-appengine?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to google-a...@googlegroups.com
> > .
> > > > To unsubscribe from this group, send email to
Stephen Johnson
Jan 11, 2011, 5:02:23 PM1/11/11
to google-a...@googlegroups.com
Cool. And thx for the quick responses!
César de Tassis Filho
Jan 12, 2011, 5:22:26 AM1/12/11
to google-a...@googlegroups.com
Hi.
I can see __scatter__ property when I go to Datastore Statistics too in most of my applications. In addition, the same day I started seeing that property in stats GAE started showing Datastore Statistics in apps that had, in the past, data stored but doesn't have anymore (like "ctassisf" AppID). Is this a bug too?
Thanks!
Jayesh
Jan 16, 2011, 11:00:47 PM1/16/11
to Google App Engine
__scatter__ also shows up in the autogenerated bulkloader.yaml file
When I downloaded the data, in the output xml the __scatter__ field
was left empty and when I tried to import it into my dev server it
gave the error
BadPropertyError: __scatter__ is a reserved property name.
--
Jayesh
On Jan 12, 3:22 pm, "C?sar de Tassis Filho" <ctass...@gmail.com>
wrote:
When I downloaded the data, in the output xml the __scatter__ field
was left empty and when I tried to import it into my dev server it
gave the error
BadPropertyError: __scatter__ is a reserved property name.
--
Jayesh
On Jan 12, 3:22 pm, "C?sar de Tassis Filho" <ctass...@gmail.com>
wrote:
Reply all
Reply to author
Forward
0 new messages