Indexed fields misrepresented in the new console for nested structs?

[Rohan Chandiramani]

I have the following struct in my Go code:

Channel struct {

Name               string                                      `datastore:"-" json:"name,omitempty" goon:"id"`

UserId              int                                          `datastore:",noindex" json:"userid,omitempty"`

Subscription        Subscription                         `datastore:",noindex" json:"subscription,omitempty"`

PersonalInformation []PersonalInformation        `datastore:",noindex" json:"personalInformation,omitempty"`

}

PersonalInformation struct {

FirstName    string `datastore:",noindex" json:"firstName,omitempty"`

LastName     string `datastore:",noindex" json:"lastName,omitempty"`

StreetAdress string `datastore:",noindex" json:"streetAdress,omitempty"`

Province     string `datastore:",noindex" json:"province,omitempty"`

PostalCode   string `datastore:",noindex" json:"postalCode,omitempty"`

CountryCode  string `datastore:",noindex" json:"countryCode,omitempty"`

}

When I look at the new console, all values from PersonalInformation are marked indexed and greyed out.

It seems to only happen with Arrays, because the nested type Subscription isn't an array and is correctly marked as non indexed.

https://cloud.google.com/appengine/docs/go/datastore/reference?hl=en

If an outer struct is tagged "noindex" then all of its implicit flattened fields are effectively "noindex".

So is the console incorrectly marking these nested array values as indexed.

or 

Is the datastore itself marking these as indexed correctly and I can't find this specific behaviour in the documentation?

-Not related to this problem-

Thank you for adding the 'Refresh' button!

Thank you for showing what entities derive from if they have an ancestor!

[Nick (Cloud Platform Support)]

It definitely seems, from a quick run-through of the doc and your schema, that the inner entities' flattened fields should be "noindex" here. Are you able to run queries which would hit the index, and do you see results?

[Rohan Chandiramani]

Sorry for the late reply.

I ran a few queries and got nothing back on those fields so either I'm using the wrong queries or (probably) the datastore is handling the query correctly, so that's a good thing.

Query:

var channels []models.Channel
_, err = gn.GetAll(datastore.NewQuery(gn.Kind(&models.Channel{})).Filter("PersonalInformation.FirstName =", searchQuery), &channels)
channels == null

Normally I don't look at this screen but because of the index issue I noticed the size of the indexes seemed a bit high:

Highlight struct {

Seconds         int64          `datastore:"-" json:"seconds,omitempty" goon:"id"`

Stream          *datastore.Key `datastore:"-" json:"-" goon:"parent"`

HighlightScore  int            `datastore:",noindex" json:"highlightScore,omitempty"`

ScorePercentile int            `datastore:",noindex" json:"scorePercentile,omitempty"`

Processed       []string       `datastore:",noindex" json:"processed,omitempty"`

Links           []string       `datastore:",noindex" json:"links,omitempty"`

WordStats       []WordStat     `datastore:",noindex" json:"wordStats,omitempty"`

}

I have no indexed fields as you can see yet the overview reports out of 3 mb indexed fields out of 13mb data, that seems a little high?

If that number seems to high it's probably linked to the same problem as the datastore viewer is showing.

3mb indexes of course isn't that much but it does add up when you have larger datasets and I assume users will be billed for that.

[Rohan Chandiramani]

For some reason the image didn't get embedded in the post this time, please look at the attachment in my previous post.

On Wednesday, December 30, 2015 at 1:51:55 AM UTC+1, Nick (Cloud Platform Support) wrote:

[Nick (Cloud Platform Support)]

Hey Rohan,

Thanks for the update. It appears that this is definitely an issue worth investigating on our end, and not an issue of your own code. Could you open a Public Issue Tracker (PIT) issue linking to this thread, or reproducing the information from it, and there provide your project ID so that we can look into your Datastore and attempt to find a root cause? Feel free to link the issue thread here once created. We monitor the PIT closely and it should see action quickly.

As an outside chance, is it possible you worked with your entities before adding "noindex"? Or was that there from the beginning?

[Rohan Chandiramani]

Hey Nick, 

Here we go, 

https://code.google.com/p/googleappengine/issues/detail?id=12648

I have also created a small test project demonstrating the problem in the issue tracker.

As an outside chance, is it possible you worked with your entities before adding "noindex"? Or was that there from the beginning?

I'm very sure it has been there from the beginning :)

Cheers,

Rohan

[Nick (Cloud Platform Support)]

Hey Rohan,

Alright, best of luck in proceeding to get this issue looked-at. The reproduction is excellent form for issue-reporting, and I'm sure you'll see a quick reply on that basis.