Chat Time transcript: May 6th, 2009
223 views
Skip to first unread message
Jason (Google)
May 7, 2009, 1:53:31 PM5/7/09
to Google App Engine
Last evening (PDT), the App Engine team held its most recent "Chat
Time" IRC office hour session. The next session will take place May
20th from 9:00-10:00 a.m. in the #appengine channel on
irc.freenode.net. A transcript of yesterday's session and a summary of
the topics covered is provided below.
- A very small percentage of datastore operations will fail, and your
applications have to catch these occasional exceptions; Google helps
mitigate this by retrying individual puts and deletes as well as
transactions [7:02]
- Java will support unique user identities soon, and it's reasonable
to expect feature parity between the Python and Java runtimes [7:05]
- You will be able to update to the new Java SDK (when it's available)
directly in Eclipse if you installed it via the Google Plugin for
Eclipse [7:10]
- A concurrency-related error when committing a transaction will
result in a ConcurrentModificationException being thrown; if the
datastore is in read-only mode, a general DatastoreFailureException is
thrown [7:13, 7:15]
- Applications cannot share datastores currently; if an application
utilizes multiple projects for modularity/implementing a common API
for multiple projects, contact us so we can work through any TOS
issues [7:22]
- The next Python SDK update will include, among other things,
"unindexed" properties, ie disabling indexing on a per property basis,
and keys-only queries, both of which are faster and much cheaper than
the existing options [7:26]
- To clear your local datastore using the SDK for Java, delete WEB-INF/
local_db.bin [7:28]
- Fetching a group of entities in the low-level Java datastore API
does NOT throw an exception if one of the entities doesn't exist,
similar to how batch gets work in Python [7:33]
- http://code.google.com/appengine/docs/java/datastore/dataclasses.html#Core_Value_Types
lists the core data types that can be stored in App Engine's
datastore, similar to the types allowed in Python, and these can be
used with the low-level Java datastore API [7:37]
- If you won't need to query a text-based field, it makes sense to use
the Text type instead of String since writes will be both faster and
cheaper since Text properties aren't indexed [7:43]
- A more scalable offline processing solution than cron is in the
works and "we're looking forward to opening it up soon" [7:49]
- SSL can't be supported for third-party domains until browser
security policies are updated; see
http://googleappengine.blogspot.com/2008/10/announcing-https-support-for-appspotcom.html
[7:50, 7:53]
- Versioning in the datastore likely won't be coming soon but there
are third party object versioning libraries for both Python and Java
that should be easy to plug in [7:56]
- App Engine DOES automatically gzip dynamic requests in both Java and
Python if the client indicates that they can accept gzipped responses
[7:59]
- Datastore viewer for Java is coming [8:00]
[7:02pm] Jason_Google: Hi Everyone. Welcome to Chat Time! I and a few
other Googlers will be in the channel for the next hour or so to
answer your App Engine questions.
[7:02pm] DocSavage: I had a question on the expected timeout/error
rate for datastore queries. Should we expect some low level % of ops
to fail?
[7:03pm] DocSavage: Something like 0.1%?
[7:03pm] maxr_google: I can't give you an exact percentage, but it's
reasonable to expect a small percentage to fail.
[7:04pm] ryan_google: yes, 0.1% is about right
[7:04pm] ryan_google: we're continually working on lowering that
[7:04pm] DocSavage: Are you looking into a way to handle retries on
the server side?
[7:05pm] ryan_google: we do retry puts and deletes outside of txes
server side
[7:05pm] ryan_google: and we retry txes client side
[7:05pm] ryan_google: we have some similar features to increase read
availability eventually
[7:05pm] Kardax: Python got permanent unique user identities
recently... is there any chance Java is getting this valuable feature?
[7:06pm] maxr_google: Yes
[7:06pm] DocSavage: Ryan, so right now if we get a timeout on a put/
delete, it's actually gone through some retries already?
[7:06pm] ryan_google: correct
[7:06pm] maxr_google: Ideally we will release new features to both
runtimes at the same time.
[7:06pm] maxr_google: But that won't always happen.
[7:07pm] Kardax: I imagine getting Java out there in the first place
was quite an undertaking
[7:07pm] Jack2009: I have a question about user access. I have a user
who has a @post.harvard.edu email address, but when he signs in, he
gets signed in under a gmail.com address but he wants email to be sent
(via my app) as @post.harvard.edu. But if I force my app to send it as
@post.harvard.edu, I get an IllegalSenderException
[7:07pm] maxr_google: But unless there's something particularly
language specific about a feature it's reasonable to expect that there
will be eventual parity.
[7:07pm] maxr_google:
[7:07pm] Jason_Google: Kardax: An update to the Java SDK will be
coming shortly that will address a number of issues and bring it more
in line with the recent update to the Python SDK.
[7:08pm] Kardax: I'm looking forward to it, Jason
[7:08pm] ryan_google: re: timeouts, i've weighed in on a few threads
and the main issue, and i'll look for more to weigh in on
[7:08pm] ryan_google: we're definitely aware of them, and we know
they're no fun. they'll never go away entirely, but we're actively
working on ways to mitigate them
[7:08pm] Kardax: Will that Java update be visible using Eclipse's
built-in update functions?
[7:08pm] robokopped: for my app, i'm pulling in a very large rss feed
(over 1MB in size) which is beyond the limit of urlfetch. i'm using
yahoo pipes to pare down the feed using an item range, and then
pulling in a chunk at a time. even with this, i get a significant
number of DownloadErrors due to timeout (i believe it is 5 seconds for
urlfetch). is there a recommendation or best practice to handle
this? and are there any plans to increase thi
[7:08pm] Jason_Google: Jack2009: Email must be sent from an
administrator email. Has he tried registering his email as a Google
account?
[7:09pm] maxr_google: Kardax: I should know the answer to this but
unfortunately I don't.
[7:09pm] Jason_Google: robokopped: I believe this limit will be
increased, although I'm not sure when.
[7:09pm] maxr_google: I believe someone from the eclipse plugin dev
team is going to be signing on. They'll have a better answer.
[7:10pm] Kardax: It's not critical, I was mostly curious
[7:10pm] dan_google: If you installed the SDK via the Eclipse plugin,
you'll be able to update the SDK via Software Update.
[7:11pm] dan_google: IIRC, there's even a UI notification about the
update.
[7:11pm] Kardax: Ok, next question: In the Java low-level API, what
happens when a transaction.commit() has a concurrency error? The docs
aren't clear on this...
[7:11pm] DocSavage: re: timeouts. I'm currently using a two step
approach: (1) generous use of memcache to avoid more timeout-prone
datastore ops, (2) small # of retries. When you take memcache down
during maintenance, though, will you raise an error as opposed to just
having any memcache API call return None?
[7:12pm] maxr_google: Kardax: What sort of details are you interested
in?
[7:12pm] jasonparekh_Goog: Kardax: yes, you'll be able to update to
the latest SDK using the Eclipse plug-in's update mechanism
[7:12pm] maxr_google: Let me bring up the javadoc so I can see exactly
what we've described.
[7:12pm] Kardax: maxr: I'd like to know that it was specifically a
concurrency error, rather than a timeout or other datastore issue.
[7:13pm] maxr_google: Ah yes, and the javadocs don't state that.
[7:13pm] Jason_Google: Kardax: I believe you'll get an exception. App
Engine uses optimistic concurrency. According the Python docs: "While
one app instance is applying changes to entities in an entity group,
all other attempts to update any entity in the group fail instantly.
The app can try the transaction again to apply it to the updated
data."
[7:13pm] maxr_google: You should get a ConcurrentModificationException
[7:13pm] ryan_google1: DocSavage: we expect to take memcache all the
way down rarely, if ever. i'm not sure of the behavior details when
that happens, though. anyone else?
[7:13pm] Kardax: Yeah, I'm converting a Python app to Java and the
Java docs are, um... lacking...
[7:14pm] Jason_Google: Work in progress.
[7:14pm] Jason_Google: There's a lot to document.
[7:14pm] Kardax: Related question: sometimes you make the datastore
read-only. What's the exception for that?
[7:14pm] fkedupmonkey: Open Q: How do people here serialize their data
before storing in JCache?
[7:15pm] ribrdb: I think python behaves as if there was a cache miss
when memcache is down. And I think you can either get an exception or
a cache miss in java, depending on how you configure it.
[7:15pm] maxr_google: I believe it just propagates as a
DatastoreFailureException.
[7:16pm] maxr_google: A more specific exception would be useful, yeah?
[7:16pm] DocSavage: Kardax/maxr_google: Isn't it a
CapabilityDisabledError on put?
[7:16pm] Kardax: A specific exception would be nice so I can provide
an appropriate error to my users.
[7:16pm] maxr_google: Perhaps in python, but in Java we don't have
that one.
[7:16pm] Kardax: I think if it inherits DatastoreFailureException, it
won't break existing apps...
[7:16pm] maxr_google: Yeah, that sounds like a very reasonable feature
request.
[7:17pm] jack2009b: I got disconnected after I posted by last response
[7:18pm] Jason_Google: jack2009b: I don't know if we got your last
response.
[7:18pm] oktopus: I have an app in mind that shows animated cars
passing the user's car and allows the user to adjust mirrors to
eliminate the blind spot. Can such graphics be done on GAE and if so
can you give me some examples or reading, please?
[7:18pm] DocSavage: Any chance that different apps can be set to use
the same datastore? I think it would really help given the way
administration is currently set up.
[7:18pm] jack2009b: Jason_Google: my app sends email as the signed in
user not admin. Yes, he tried to re-register his email as a Google
account, but it gives him an error that's it's already registered. If
he tries to change the password, it automatically signs him in under
his @gmail account!
[7:19pm] Jason_Google: oktopus: Would you be using Flash?
[7:19pm] maxr_google_: DocSavage: Not currently, no.
[7:19pm] oktopus: I use opengl now, but I guess that is not relevant.
[7:19pm] maxr_google_: Could the app with access to the datastore
instead expose an api that other apps could use?
[7:20pm] Jason_Google: jack2009b: Has he tried to clear his browser's
cookies before this?
[7:20pm] Kardax: Next on my list: In Java, when doing a multiple-
entity get (like a list of entities), if any of the entities are not
found, you get an exception and no results at all. Python gives you
"null" for the missing ones. Is there any way to get Python's
behavior in Java?
[7:20pm] fkedupmonkey: @oktopus you could use NativeClient
[7:20pm] DocSavage: Not currently, I was asking more about if this was
looked at for future. re: API use -- This could be a violation of TOS
if the app in question was providing a key function?
[7:21pm] jack2009b: Jason_Google: yes, he completely signed out and
cleared his cookies
[7:21pm] oktopus: I am not familiar with NativeClient. is it
something I can add .com to and learn about?
[7:21pm] Jason_Google: oktopus: You will have to use something that
renders client-side. Flash is one option, O3D is another.
[7:22pm] maxr_google_: DocSavage: Letting multiple apps access the
same datastore is something we've talked about, but not something we
have any specific plans to implement.
[7:22pm] oktopus: Are there example apps that demo either?
[7:22pm] fkedupmonkey: @oktopus - O3D is what I meant ny NativeClient
- a browser plug-in for 3d graphics - very OpenGLy, I think. You can
Google for it.
[7:22pm] maxr_google_: I'm not sure about the TOS issue.
[7:22pm] Jason_Google: jack2009b: If he's cleared his cookies, I don't
see how it's signing him in with his Gmail?
[7:22pm] ryan_google1: we tend to be pretty understanding with TOS
questions like that. if it's for modularity, e.g. an API, that's
obviously different from straight up sharding
[7:23pm] ryan_google1: we usually look at those questions on a case by
case basis. an API for modularity is very different from sharding, for
example.
[7:24pm] Kardax: (I have to be more careful about dropping new
questions in the middle of other conversations
[7:24pm] jack2009b: jason_Google: It seems that he @post.harvard.edu
email address is linked to his @gmail account so when he signs in as
@post.harvard.edu, he is acually signed in as @gmail
[7:24pm] oktopus: Ok, I found a google link http://code.google.com/apis/o3d/
Thanks. Can I use python or will I need to learn java?
[7:25pm] robokopped: this is probably a long shot, but can you tip us
off to when the next GAE python release will be out, and what updates
we can expect?
[7:25pm] Jason_Google: oktopus: It's a JavaScript API.
[7:25pm] ryan_google1: robokopped: soon! within a week or two.
[7:25pm] robokopped: thanks!
[7:25pm] DocSavage: ryan: One concern I had would be automated TOS
checkers that might shut down your app if it receives a lot of calls
from a main app. Would App Engine TOS checkers be manual and contact
us personally before anything like that happens? (This is based on
some other well-publicized Google shutdowns.)
[7:26pm] fkedupmonkey: i'm having trouble serializing a JDO class for
use in JCache - getting error with serializing User - "Datastore value
is of type com.google.appengine.api.users.User (must be Blob)."
[7:26pm] Jason_Google: jack2009b: You'll need to bring this up with
Gmail or Google Accounts support. I know that Gmail supports aliasing
but I'm under the impression that this is only for other Gmail
addresses.
[7:26pm] ryan_google1: on the datastore side, it will include
"unindexed" properties, ie disabling indexing on a per property basis,
and keys-only queries, both of which are faster and much cheaper than
the existing options
[7:26pm] Jason_Google: jack2009b: Here's one link you can use:
http://www.google.com/support/accounts/?hl=en
[7:27pm] oktopus: Javascript, That sounds good. Thanks to all.
[7:27pm] maxr_google_: fkedupmonkey: That sounds like maybe you saved
some data to the datastore and then changed the type of the field to
be Serialized.
[7:27pm] maxr_google_: Is that possible?
[7:27pm] ryan_google1: DocSavage: it depends on the problem. we'll
have much less tolerance for a malicious app, for example, or one
that's causing system-wide instability, than one that's lightly
sharded or something similarly innocuous
[7:27pm] jack2009b: Jason_Google: thanks for the suggestion. Can you
point me the officlal/best way to submit my issue to jack2009b:
You'll need to bring this up with Gmail or Google Accounts support
[7:27pm] ryan_google1: but generally, yes, we'll contact the
developer, and depending on the problem, we'll often wait a bit for an
acknowledgement
[7:28pm] jack2009b: Jason_Google: thanks for teh link
[7:28pm] fkedupmonkey: @maxr_google ah good call, i do caching as a
later step (this is for code walkthrough). is there a way to get the
local server to clear the database from eclipse?
[7:28pm] maxr_google_: Easiest thing to do is just delete the file on
disk and then bounce your server.
[7:28pm] maxr_google_: the file lives in WEB-INF/local_db.bin
[7:28pm] Jason_Google: jack2009b: Sure. I hope you're able to get that
worked out.
[7:29pm] fkedupmonkey: @maxr_google kk will give that a go
[7:30pm] robokopped: when i get an ApplicationError in python, what
does the value following the error mean? for example,
ApplicationError: 2 or ApplicationError: 5
[7:30pm] Kardax: In Java, when doing a multiple-entity get (like a
list of entities), if any of the entities are not found, you get an
exception and no results at all. Python gives you "null" for the
missing ones. Is there any way to get Python's behavior in Java?
[7:31pm] maxr_google_: Kardax: That's suprising to me.
[7:31pm] maxr_google_: This is with the low level api?
[7:31pm] Kardax: Yes.
[7:31pm] maxr_google_: Doesn't sound right, I'll investigate.
[7:31pm] jack2009b: Jason_Google: the link you gave me was for regular
end-user support. Is there a developer support channel for Google
accounts?
[7:31pm] pranny: folks, I would suggest logging #appengine. It would
be quite useful
[7:31pm] ryan_google1: robokopped: depends. usually we try to
translate those into human-readable error messages and exception
types. there might be some code paths that miss translation, though
[7:31pm] ryan_google1: please report it on the issue tracker!
[7:31pm] Jason_Google: jack2009b: This isn't a developer issue.
[7:32pm] ryan_google1: pranny: we post transcripts to the group
[7:32pm] fkedupmonkey: @maxr_google getting there-- (it's in app-
engine-generated by the way) -- but now get error when trying to store
an empty List - which I thought was serializable?: Cannot use as
value: '[]'
[7:32pm] Kardax: Actually... I may be wrong.
[7:32pm] Jason_Google: jack2009b: We have a group set up for Google
Accounts developer issues, but that's for stuff like AuthSub,
ClientLogin, or OAuth questions. This is a general issue with Google
Accounts.
[7:32pm] jack2009b: Jason_Google: But there's no way on that page to
even contact support
[7:32pm] Kardax: I thought I read that in the docs, but I'm not
finding it now...
[7:33pm] Kardax: Nevermind
[7:33pm] maxr_google_: Kardax: I just ran a quick test - it doesn't
throw an exception for me.
[7:33pm] maxr_google_: If you can repro the exception please post it.
[7:34pm] Kardax: Yeah, I msut have been crossing my eyes or soemthing
and read the wrong "get" exception list
[7:34pm] Jason_Google: jack2009b: You can use a general form like
http://www.google.com/support/accounts/bin/answer.py?answer=48598 and
someone will get back to you, I believe.
[7:34pm] warreninaustintx: Question on App Engine cron jobs. They run
with current_user == None, is that correct (Python SDK)?
[7:34pm] maxr_google_: fkedupmonkey: Is it a ClassCastException?
[7:34pm] maxr_google_: We have a known issue related to that.
[7:34pm] ryan_google1: warreninaustintx: i believe so, yes
[7:34pm] fkedupmonkey: @maxr_google:
java.lang.IllegalArgumentException: Cannot use as value: '[]' at
com.google.appengine.api.memcache.MemcacheServiceImpl.put
(MemcacheServiceImpl.java:306)
[7:35pm] maxr_google_: fkedupmonkey: ah, that's something else and out
of my realm of expertise I'm afraid.
[7:35pm] jack2009b: Jason_Google: thanks for trying but I won't be
able to submit the issue becuase I'm not the user and the user is not
technical enough. But thanks anyway!
[7:36pm] Kardax: In Java's low-level API, is there a list of the data
types that are allowed to go in and how they are returned later?
[7:36pm] Jason_Google: jack2009b: So the user signs into the App
Engine app with his harvard.edu address, but can't send email from
that address. But he can send email through the app if the sender is
set to the Gmail?
[7:37pm] maxr_google_: kardax: Yes, and I believe that's documented.
Let me find it for you.
[7:37pm] Jason_Google: Kardax: Yes, similar to Python.
[7:37pm] fkedupmonkey: @maxr_google okay, perhaps i'll just cache the
rendered HTML output for now. thnx for the tip on DB clearing
[7:37pm] maxr_google_: kardax:
http://code.google.com/appengine/docs/java/datastore/dataclasses.html#Core_Value_Types
[7:38pm] maxr_google_: fkedupmonkey: np
[7:38pm] Kardax: maxr: So this works for the low-level API, too?
[7:38pm] jack2009b: Jason_Google: no, when he saigns into app engine
as x...@post.harvard.edu, it actually signs him in under his
x...@gmail.com account. He can send email from my app as x...@gmail.com
but he wants the email to come from x...@post.harvard.edu.
[7:38pm] Jason_Google: Kardax: Yes.
[7:38pm] maxr_google_: Kardax: Yes, but the exceptions are a little
different.
[7:39pm] Kardax: Ok, thanks much
[7:39pm] maxr_google_: For example a String that's longer than 500
bytes will yield an IllegalArgumentException
[7:39pm] Jason_Google: jack2009b: Yeah, that's strange. Unfortunately,
I don't have any answers for you on that.
[7:39pm] jack2009b:
[7:39pm] jack2009b:
[7:39pm] jack2009b:
[7:40pm] jack2009b: (3rd try's a charm)
[7:40pm] Kardax: maxr: That makes sense.
[7:41pm] ryan_google1: jack2009b: are both the gmail and the
post.harvard addresses attached to the google account? and gmail is
the "primary" email address?
[7:41pm] ryan_google1: that might be it
[7:41pm] ryan_google1: (i don't actually know much about this, though,
so no promises on followup q's :P)
[7:42pm] Kardax: General datastore question: Does it make sense to use
Text exclusively (in place of String) to avoid the disk overhead of a
String's "simple index"?
[7:42pm] ryan_google1: kardax: if you're sure you'll *never* need to
query on those fields, then sure!
[7:42pm] ryan_google1: writes to that kind will be a little faster,
maybe, and a lot cheaper
[7:42pm] oktopus: I have an app which enables users to create their
own suggestions/comments list web page. Each user's web page is just a
datastore record linked to any comments which are also related
datastore records. Is there anyway that the user's page can get seen
by google spiders?
[7:43pm] ryan_google1: in the upcoming release we'll let you disable
indexing for any property, without resorting to hacks like Text
instead of String
[7:43pm] Kardax: Expanding on that, is there a way to remove the
simple index on other types that will never be queries?
[7:43pm] Kardax: Ahh, beat me to it
[7:43pm] ryan_google1: oktopus: we'll crawl it as long as you don't
have to log in to see those pages
[7:43pm] Kardax: I'm using a data structure that uses "gets"
exclusively, and I want to avoid all the overhead for queries that
will never run
[7:44pm] oktopus: My application is like a roll-your-own facebook
page.
[7:44pm] Jason_Google: oktopus: You should be able to use robots.txt
to control this, I believe.
[7:45pm] Kardax: Are there any plans to make "cron jobs" more
scalable?
[7:45pm] robokopped: oktopus, look into sitemaps.xml
[7:45pm] robokopped: * sitemap.xml
[7:46pm] oktopus: The users pages have links like
http://simplifyconnections.appspot.com/?place=Playground (notice the
question mark). So they are just virtual links, sort of.
[7:46pm] ryan_google1: kardax: very good call on avoiding indexing.
you should notice a drop in cpu when you switch to Text or use our new
feature
[7:46pm] ryan_google1: kardax: could you elaborate?
[7:46pm] Kardax: Elaborate on indexing or cron jobs?
[7:47pm] oktopus: So, does robot.txt still apply? I am not familiar
with robot.txt.
[7:47pm] Kardax: I am definitely looking forward to the CPU drop. In
my original python version (being converted to Java now), the CPU
usage was alarmingly high.
[7:47pm] maxr_google_: Kardax: elaborate on cron jobs
[7:47pm] Jason_Google: oktopus: http://en.wikipedia.org/wiki/Robots_exclusion_standard
[7:48pm] _robot: whoa
[7:48pm] Kardax: Ok, basically as a site grows and users increase
their content, the amount of work needed to update each user's
revelvant info increases.
[7:48pm] _robot: we have google people here?
[7:48pm] Jason_Google: Yes, it still applies. sitemap.xml that
robokopped brought up also applies.
[7:48pm] Kardax: It's not feasable to do it all in a single HTTP
request for everyone.
[7:48pm] Kardax: It could be done in multiple requests for batches of
suers.
[7:48pm] ryan_google1: kardax: ah, right
[7:48pm] ryan_google1: the short answer is, cron jobs are not a
general purpose, scalable offline processing solution
[7:49pm] Kardax: But the current cron system is very in flexible...
you can only run it so often and once at a time.
[7:49pm] Jason_Google: _robot:
http://groups.google.com/group/google-appengine-java/browse_thread/thread/465d1d1c3c75c40b
[7:49pm] ryan_google1: we have one of those in the works, and we're
looking forward to opening it up soon
[7:49pm] ryan_google1: but cron isn't it
[7:49pm] Kardax: That'll be helpful
[7:49pm] Kardax: My get-rich-quick pland doesn't need it now but it
could be useful later
[7:49pm] ryan_google1: glad to hear it!
[7:49pm] _robot: when will ssl be supported for non appspot sites?
[7:49pm] ryan_google1: we're looking forward to having people try it
out
[7:50pm] Jason_Google: _robot: I believe the short answer is when
browser security policies are modernized.
[7:50pm] Kardax: Regarding this future scalable offline processing
solution, do you envision it being useful for non HTTP applications?
(Scientific computer, for example)?
[7:50pm] ryan_google1: +1. details in
http://googleappengine.blogspot.com/2008/10/announcing-https-support-for-appspotcom.html
[7:51pm] ryan_google1: kardax: sure, maybe. app engine's focus is on
webapps, so those use cases will be better supported than general-
purpose distributed computing
[7:51pm] ryan_google1: but it will definitely be possible
[7:51pm] oktopus: So, I am asking how google spider would find a web
page that has "/?=..." as part of the address.
[7:51pm] _robot: i thought most modern browsers had support already?
[7:51pm] Kardax: Hmm... I think it might be interesting to make an app
engine raytracer that hands off a pixel to each node
[7:52pm] ryan_google1: interesting! as long as you don't need
realtime, at least
[7:52pm] Kardax: No, more like a complex pre-rendered scene, like a
frame from Toy Story
[7:53pm] Kardax: I bet amatuer 3d artists would be very interested in
scaling their work beyond the limitations of their own machines
[7:53pm] Kardax: Gonna rack up some hefty CPU bills, though
[7:53pm] Jason_Google: _robot: This isn't my area of expertise, but I
believe certificates can only be associated with a single domain,
which prevents us from offering SSL on third-party domains also. I
know there are proposals in the works to address this and other
limitations. I probably can't give you a better explanation than that,
unfortunately.
[7:55pm] Jason_Google: We'll be here for another 5 or so minutes, so
get your questions in.
[7:55pm] DocSavage: I noticed the Google Code team implemented
mercurial on top of bigtable. Has the App Engine team looked at their
implementation? A "mercurial" service for entities would be truly
awesome for my projects...
[7:56pm] Jason_Google: Or feel free to share any new applications
you've built.
[7:56pm] ryan_google1: versioning for the datastore is definitely a
common request
[7:56pm] Kardax: I'm though my original list of questions... trying to
think of more while there's still tiem
[7:56pm] fkedupmonkey: here's my app (under construction!)
http://best-website-ever.appspot.com/
[7:56pm] ryan_google1: it's unlikely that we'll do it, though, at
least in the foreseeable future
[7:56pm] Jason_Google: Ah fkedupmonkey, not very humble, are you?
[7:57pm] fkedupmonkey: @Jason_Google it's an artifact of the company i
work for
[7:57pm] robokopped: lol 1997 all over again
[7:57pm] ryan_google1: there are third party object versioning
libraries (both python and java) that should be easy to plug in though
[7:57pm] ryan_google1: that's what we usually recommend
[7:57pm] fkedupmonkey: but yet the most scalable 1997 ever
[7:57pm] DocSavage: ryan, what are your recommended python ones?
[7:57pm] Jason_Google: fkedupmonkey: Goodness gracious. The only thing
I'm missing is the blink tag.
[7:57pm] Kardax: I like the Netscape ad, fkedupmonkey
[7:58pm] fkedupmonkey: @Jason_Google view it in Firefox- WebKit
deprecated the blink tag, for reals
[7:58pm] ryan_google1: docsavage: don't know off the top of my head,
sorry
[7:58pm] Jason_Google: Oh, there it is.
[7:58pm] Jason_Google: Horray for WebKit.
[7:58pm] robokopped: LOL
[7:58pm] robokopped: <FONT SIZE=\"6\" FACE=\"courier\" COLOR=blue>
[7:58pm] robokopped: <MARQUEE WIDTH=100% BEHAVIOR=ALTERNATE
BGColor=yellow>
[7:58pm] robokopped: WELCOME TO MY WEB PAGE!!!
[7:58pm] robokopped: </MARQUEE></FONT>
[7:59pm] Kardax: OK, a question. I should probably know this, but oh
well. Do you automatically GZIP dynamic requestsin Java?
[7:59pm] oktopus: http://simplifyconnections.appspot.com/?place=I20W-Abilene
is an example
[7:59pm] fkedupmonkey: to whooever posted about the hit counter, i'll
be implementing that based on sharded counter
http://code.google.com/p/googleappengine/source/browse/#svn/trunk/java/demos/shardedcounter
[7:59pm] Jason_Google: Kardax: It's in here: http://code.google.com/appengine/kb/general.html
[8:00pm] Jason_Google: More specifically:
http://code.google.com/appengine/kb/general.html#compression
[8:00pm] Jason_Google: Short answer if you don't want to click: yes.
[8:00pm] Kardax: I see it I knew it happened for static stuff...
wasn't sure about dynamic.
[8:00pm] Kardax: Another: any chance for a datastore viewer in Java?
[8:00pm] maxr_google_: fkedupmonkey: Could you please file an issue
for the memcache exception you're getting?
[8:01pm] maxr_google_: Kardax: Yes, there will be a datastore viewer.
[8:01pm] Jason_Google: One is in the works. It probably won't make the
next SDK update, but watch out for it.
[8:01pm] fkedupmonkey: @maxr_google sure, will do. i figured it was a
user error. i'm not the best java user.
[8:01pm] DocSavage: When is the next SDK update?
[8:01pm] Kardax: Cool. I got good use out of the Python one
[8:01pm] Jason_Google: DocSavage: In the next week or so.
[8:01pm] DocSavage: Awesome.
[8:02pm] Jason_Google: OK, this marks the end of another awesome Chat
Time.
[8:02pm] oktopus: G`day
[8:02pm] DocSavage: I figured mine would be the last comment
[8:02pm] Kardax: Thanks for the time, it was incredibly helpful
[8:02pm] Jason_Google: We'll be back here on the 20th, 9-10 a.m. PDT.
Until then, join us in the discussion groups.
[8:02pm] DocSavage: Thanks guys. And thanks for the last SDK update
with unique user iD, etc.
[8:03pm] Jason_Google: Oh, I'll be posting a transcript of this
session to the groups as well, before the end of tomorrow.
[8:03pm] Jason_Google: So long!
[8:03pm] maxr_google_: adios!
Time" IRC office hour session. The next session will take place May
20th from 9:00-10:00 a.m. in the #appengine channel on
irc.freenode.net. A transcript of yesterday's session and a summary of
the topics covered is provided below.
- A very small percentage of datastore operations will fail, and your
applications have to catch these occasional exceptions; Google helps
mitigate this by retrying individual puts and deletes as well as
transactions [7:02]
- Java will support unique user identities soon, and it's reasonable
to expect feature parity between the Python and Java runtimes [7:05]
- You will be able to update to the new Java SDK (when it's available)
directly in Eclipse if you installed it via the Google Plugin for
Eclipse [7:10]
- A concurrency-related error when committing a transaction will
result in a ConcurrentModificationException being thrown; if the
datastore is in read-only mode, a general DatastoreFailureException is
thrown [7:13, 7:15]
- Applications cannot share datastores currently; if an application
utilizes multiple projects for modularity/implementing a common API
for multiple projects, contact us so we can work through any TOS
issues [7:22]
- The next Python SDK update will include, among other things,
"unindexed" properties, ie disabling indexing on a per property basis,
and keys-only queries, both of which are faster and much cheaper than
the existing options [7:26]
- To clear your local datastore using the SDK for Java, delete WEB-INF/
local_db.bin [7:28]
- Fetching a group of entities in the low-level Java datastore API
does NOT throw an exception if one of the entities doesn't exist,
similar to how batch gets work in Python [7:33]
- http://code.google.com/appengine/docs/java/datastore/dataclasses.html#Core_Value_Types
lists the core data types that can be stored in App Engine's
datastore, similar to the types allowed in Python, and these can be
used with the low-level Java datastore API [7:37]
- If you won't need to query a text-based field, it makes sense to use
the Text type instead of String since writes will be both faster and
cheaper since Text properties aren't indexed [7:43]
- A more scalable offline processing solution than cron is in the
works and "we're looking forward to opening it up soon" [7:49]
- SSL can't be supported for third-party domains until browser
security policies are updated; see
http://googleappengine.blogspot.com/2008/10/announcing-https-support-for-appspotcom.html
[7:50, 7:53]
- Versioning in the datastore likely won't be coming soon but there
are third party object versioning libraries for both Python and Java
that should be easy to plug in [7:56]
- App Engine DOES automatically gzip dynamic requests in both Java and
Python if the client indicates that they can accept gzipped responses
[7:59]
- Datastore viewer for Java is coming [8:00]
[7:02pm] Jason_Google: Hi Everyone. Welcome to Chat Time! I and a few
other Googlers will be in the channel for the next hour or so to
answer your App Engine questions.
[7:02pm] DocSavage: I had a question on the expected timeout/error
rate for datastore queries. Should we expect some low level % of ops
to fail?
[7:03pm] DocSavage: Something like 0.1%?
[7:03pm] maxr_google: I can't give you an exact percentage, but it's
reasonable to expect a small percentage to fail.
[7:04pm] ryan_google: yes, 0.1% is about right
[7:04pm] ryan_google: we're continually working on lowering that
[7:04pm] DocSavage: Are you looking into a way to handle retries on
the server side?
[7:05pm] ryan_google: we do retry puts and deletes outside of txes
server side
[7:05pm] ryan_google: and we retry txes client side
[7:05pm] ryan_google: we have some similar features to increase read
availability eventually
[7:05pm] Kardax: Python got permanent unique user identities
recently... is there any chance Java is getting this valuable feature?
[7:06pm] maxr_google: Yes
[7:06pm] DocSavage: Ryan, so right now if we get a timeout on a put/
delete, it's actually gone through some retries already?
[7:06pm] ryan_google: correct
[7:06pm] maxr_google: Ideally we will release new features to both
runtimes at the same time.
[7:06pm] maxr_google: But that won't always happen.
[7:07pm] Kardax: I imagine getting Java out there in the first place
was quite an undertaking
[7:07pm] Jack2009: I have a question about user access. I have a user
who has a @post.harvard.edu email address, but when he signs in, he
gets signed in under a gmail.com address but he wants email to be sent
(via my app) as @post.harvard.edu. But if I force my app to send it as
@post.harvard.edu, I get an IllegalSenderException
[7:07pm] maxr_google: But unless there's something particularly
language specific about a feature it's reasonable to expect that there
will be eventual parity.
[7:07pm] maxr_google:
[7:07pm] Jason_Google: Kardax: An update to the Java SDK will be
coming shortly that will address a number of issues and bring it more
in line with the recent update to the Python SDK.
[7:08pm] Kardax: I'm looking forward to it, Jason
[7:08pm] ryan_google: re: timeouts, i've weighed in on a few threads
and the main issue, and i'll look for more to weigh in on
[7:08pm] ryan_google: we're definitely aware of them, and we know
they're no fun. they'll never go away entirely, but we're actively
working on ways to mitigate them
[7:08pm] Kardax: Will that Java update be visible using Eclipse's
built-in update functions?
[7:08pm] robokopped: for my app, i'm pulling in a very large rss feed
(over 1MB in size) which is beyond the limit of urlfetch. i'm using
yahoo pipes to pare down the feed using an item range, and then
pulling in a chunk at a time. even with this, i get a significant
number of DownloadErrors due to timeout (i believe it is 5 seconds for
urlfetch). is there a recommendation or best practice to handle
this? and are there any plans to increase thi
[7:08pm] Jason_Google: Jack2009: Email must be sent from an
administrator email. Has he tried registering his email as a Google
account?
[7:09pm] maxr_google: Kardax: I should know the answer to this but
unfortunately I don't.
[7:09pm] Jason_Google: robokopped: I believe this limit will be
increased, although I'm not sure when.
[7:09pm] maxr_google: I believe someone from the eclipse plugin dev
team is going to be signing on. They'll have a better answer.
[7:10pm] Kardax: It's not critical, I was mostly curious
[7:10pm] dan_google: If you installed the SDK via the Eclipse plugin,
you'll be able to update the SDK via Software Update.
[7:11pm] dan_google: IIRC, there's even a UI notification about the
update.
[7:11pm] Kardax: Ok, next question: In the Java low-level API, what
happens when a transaction.commit() has a concurrency error? The docs
aren't clear on this...
[7:11pm] DocSavage: re: timeouts. I'm currently using a two step
approach: (1) generous use of memcache to avoid more timeout-prone
datastore ops, (2) small # of retries. When you take memcache down
during maintenance, though, will you raise an error as opposed to just
having any memcache API call return None?
[7:12pm] maxr_google: Kardax: What sort of details are you interested
in?
[7:12pm] jasonparekh_Goog: Kardax: yes, you'll be able to update to
the latest SDK using the Eclipse plug-in's update mechanism
[7:12pm] maxr_google: Let me bring up the javadoc so I can see exactly
what we've described.
[7:12pm] Kardax: maxr: I'd like to know that it was specifically a
concurrency error, rather than a timeout or other datastore issue.
[7:13pm] maxr_google: Ah yes, and the javadocs don't state that.
[7:13pm] Jason_Google: Kardax: I believe you'll get an exception. App
Engine uses optimistic concurrency. According the Python docs: "While
one app instance is applying changes to entities in an entity group,
all other attempts to update any entity in the group fail instantly.
The app can try the transaction again to apply it to the updated
data."
[7:13pm] maxr_google: You should get a ConcurrentModificationException
[7:13pm] ryan_google1: DocSavage: we expect to take memcache all the
way down rarely, if ever. i'm not sure of the behavior details when
that happens, though. anyone else?
[7:13pm] Kardax: Yeah, I'm converting a Python app to Java and the
Java docs are, um... lacking...
[7:14pm] Jason_Google: Work in progress.
[7:14pm] Jason_Google: There's a lot to document.
[7:14pm] Kardax: Related question: sometimes you make the datastore
read-only. What's the exception for that?
[7:14pm] fkedupmonkey: Open Q: How do people here serialize their data
before storing in JCache?
[7:15pm] ribrdb: I think python behaves as if there was a cache miss
when memcache is down. And I think you can either get an exception or
a cache miss in java, depending on how you configure it.
[7:15pm] maxr_google: I believe it just propagates as a
DatastoreFailureException.
[7:16pm] maxr_google: A more specific exception would be useful, yeah?
[7:16pm] DocSavage: Kardax/maxr_google: Isn't it a
CapabilityDisabledError on put?
[7:16pm] Kardax: A specific exception would be nice so I can provide
an appropriate error to my users.
[7:16pm] maxr_google: Perhaps in python, but in Java we don't have
that one.
[7:16pm] Kardax: I think if it inherits DatastoreFailureException, it
won't break existing apps...
[7:16pm] maxr_google: Yeah, that sounds like a very reasonable feature
request.
[7:17pm] jack2009b: I got disconnected after I posted by last response
[7:18pm] Jason_Google: jack2009b: I don't know if we got your last
response.
[7:18pm] oktopus: I have an app in mind that shows animated cars
passing the user's car and allows the user to adjust mirrors to
eliminate the blind spot. Can such graphics be done on GAE and if so
can you give me some examples or reading, please?
[7:18pm] DocSavage: Any chance that different apps can be set to use
the same datastore? I think it would really help given the way
administration is currently set up.
[7:18pm] jack2009b: Jason_Google: my app sends email as the signed in
user not admin. Yes, he tried to re-register his email as a Google
account, but it gives him an error that's it's already registered. If
he tries to change the password, it automatically signs him in under
his @gmail account!
[7:19pm] Jason_Google: oktopus: Would you be using Flash?
[7:19pm] maxr_google_: DocSavage: Not currently, no.
[7:19pm] oktopus: I use opengl now, but I guess that is not relevant.
[7:19pm] maxr_google_: Could the app with access to the datastore
instead expose an api that other apps could use?
[7:20pm] Jason_Google: jack2009b: Has he tried to clear his browser's
cookies before this?
[7:20pm] Kardax: Next on my list: In Java, when doing a multiple-
entity get (like a list of entities), if any of the entities are not
found, you get an exception and no results at all. Python gives you
"null" for the missing ones. Is there any way to get Python's
behavior in Java?
[7:20pm] fkedupmonkey: @oktopus you could use NativeClient
[7:20pm] DocSavage: Not currently, I was asking more about if this was
looked at for future. re: API use -- This could be a violation of TOS
if the app in question was providing a key function?
[7:21pm] jack2009b: Jason_Google: yes, he completely signed out and
cleared his cookies
[7:21pm] oktopus: I am not familiar with NativeClient. is it
something I can add .com to and learn about?
[7:21pm] Jason_Google: oktopus: You will have to use something that
renders client-side. Flash is one option, O3D is another.
[7:22pm] maxr_google_: DocSavage: Letting multiple apps access the
same datastore is something we've talked about, but not something we
have any specific plans to implement.
[7:22pm] oktopus: Are there example apps that demo either?
[7:22pm] fkedupmonkey: @oktopus - O3D is what I meant ny NativeClient
- a browser plug-in for 3d graphics - very OpenGLy, I think. You can
Google for it.
[7:22pm] maxr_google_: I'm not sure about the TOS issue.
[7:22pm] Jason_Google: jack2009b: If he's cleared his cookies, I don't
see how it's signing him in with his Gmail?
[7:22pm] ryan_google1: we tend to be pretty understanding with TOS
questions like that. if it's for modularity, e.g. an API, that's
obviously different from straight up sharding
[7:23pm] ryan_google1: we usually look at those questions on a case by
case basis. an API for modularity is very different from sharding, for
example.
[7:24pm] Kardax: (I have to be more careful about dropping new
questions in the middle of other conversations
[7:24pm] jack2009b: jason_Google: It seems that he @post.harvard.edu
email address is linked to his @gmail account so when he signs in as
@post.harvard.edu, he is acually signed in as @gmail
[7:24pm] oktopus: Ok, I found a google link http://code.google.com/apis/o3d/
Thanks. Can I use python or will I need to learn java?
[7:25pm] robokopped: this is probably a long shot, but can you tip us
off to when the next GAE python release will be out, and what updates
we can expect?
[7:25pm] Jason_Google: oktopus: It's a JavaScript API.
[7:25pm] ryan_google1: robokopped: soon! within a week or two.
[7:25pm] robokopped: thanks!
[7:25pm] DocSavage: ryan: One concern I had would be automated TOS
checkers that might shut down your app if it receives a lot of calls
from a main app. Would App Engine TOS checkers be manual and contact
us personally before anything like that happens? (This is based on
some other well-publicized Google shutdowns.)
[7:26pm] fkedupmonkey: i'm having trouble serializing a JDO class for
use in JCache - getting error with serializing User - "Datastore value
is of type com.google.appengine.api.users.User (must be Blob)."
[7:26pm] Jason_Google: jack2009b: You'll need to bring this up with
Gmail or Google Accounts support. I know that Gmail supports aliasing
but I'm under the impression that this is only for other Gmail
addresses.
[7:26pm] ryan_google1: on the datastore side, it will include
"unindexed" properties, ie disabling indexing on a per property basis,
and keys-only queries, both of which are faster and much cheaper than
the existing options
[7:26pm] Jason_Google: jack2009b: Here's one link you can use:
http://www.google.com/support/accounts/?hl=en
[7:27pm] oktopus: Javascript, That sounds good. Thanks to all.
[7:27pm] maxr_google_: fkedupmonkey: That sounds like maybe you saved
some data to the datastore and then changed the type of the field to
be Serialized.
[7:27pm] maxr_google_: Is that possible?
[7:27pm] ryan_google1: DocSavage: it depends on the problem. we'll
have much less tolerance for a malicious app, for example, or one
that's causing system-wide instability, than one that's lightly
sharded or something similarly innocuous
[7:27pm] jack2009b: Jason_Google: thanks for the suggestion. Can you
point me the officlal/best way to submit my issue to jack2009b:
You'll need to bring this up with Gmail or Google Accounts support
[7:27pm] ryan_google1: but generally, yes, we'll contact the
developer, and depending on the problem, we'll often wait a bit for an
acknowledgement
[7:28pm] jack2009b: Jason_Google: thanks for teh link
[7:28pm] fkedupmonkey: @maxr_google ah good call, i do caching as a
later step (this is for code walkthrough). is there a way to get the
local server to clear the database from eclipse?
[7:28pm] maxr_google_: Easiest thing to do is just delete the file on
disk and then bounce your server.
[7:28pm] maxr_google_: the file lives in WEB-INF/local_db.bin
[7:28pm] Jason_Google: jack2009b: Sure. I hope you're able to get that
worked out.
[7:29pm] fkedupmonkey: @maxr_google kk will give that a go
[7:30pm] robokopped: when i get an ApplicationError in python, what
does the value following the error mean? for example,
ApplicationError: 2 or ApplicationError: 5
[7:30pm] Kardax: In Java, when doing a multiple-entity get (like a
list of entities), if any of the entities are not found, you get an
exception and no results at all. Python gives you "null" for the
missing ones. Is there any way to get Python's behavior in Java?
[7:31pm] maxr_google_: Kardax: That's suprising to me.
[7:31pm] maxr_google_: This is with the low level api?
[7:31pm] Kardax: Yes.
[7:31pm] maxr_google_: Doesn't sound right, I'll investigate.
[7:31pm] jack2009b: Jason_Google: the link you gave me was for regular
end-user support. Is there a developer support channel for Google
accounts?
[7:31pm] pranny: folks, I would suggest logging #appengine. It would
be quite useful
[7:31pm] ryan_google1: robokopped: depends. usually we try to
translate those into human-readable error messages and exception
types. there might be some code paths that miss translation, though
[7:31pm] ryan_google1: please report it on the issue tracker!
[7:31pm] Jason_Google: jack2009b: This isn't a developer issue.
[7:32pm] ryan_google1: pranny: we post transcripts to the group
[7:32pm] fkedupmonkey: @maxr_google getting there-- (it's in app-
engine-generated by the way) -- but now get error when trying to store
an empty List - which I thought was serializable?: Cannot use as
value: '[]'
[7:32pm] Kardax: Actually... I may be wrong.
[7:32pm] Jason_Google: jack2009b: We have a group set up for Google
Accounts developer issues, but that's for stuff like AuthSub,
ClientLogin, or OAuth questions. This is a general issue with Google
Accounts.
[7:32pm] jack2009b: Jason_Google: But there's no way on that page to
even contact support
[7:32pm] Kardax: I thought I read that in the docs, but I'm not
finding it now...
[7:33pm] Kardax: Nevermind
[7:33pm] maxr_google_: Kardax: I just ran a quick test - it doesn't
throw an exception for me.
[7:33pm] maxr_google_: If you can repro the exception please post it.
[7:34pm] Kardax: Yeah, I msut have been crossing my eyes or soemthing
and read the wrong "get" exception list
[7:34pm] Jason_Google: jack2009b: You can use a general form like
http://www.google.com/support/accounts/bin/answer.py?answer=48598 and
someone will get back to you, I believe.
[7:34pm] warreninaustintx: Question on App Engine cron jobs. They run
with current_user == None, is that correct (Python SDK)?
[7:34pm] maxr_google_: fkedupmonkey: Is it a ClassCastException?
[7:34pm] maxr_google_: We have a known issue related to that.
[7:34pm] ryan_google1: warreninaustintx: i believe so, yes
[7:34pm] fkedupmonkey: @maxr_google:
java.lang.IllegalArgumentException: Cannot use as value: '[]' at
com.google.appengine.api.memcache.MemcacheServiceImpl.put
(MemcacheServiceImpl.java:306)
[7:35pm] maxr_google_: fkedupmonkey: ah, that's something else and out
of my realm of expertise I'm afraid.
[7:35pm] jack2009b: Jason_Google: thanks for trying but I won't be
able to submit the issue becuase I'm not the user and the user is not
technical enough. But thanks anyway!
[7:36pm] Kardax: In Java's low-level API, is there a list of the data
types that are allowed to go in and how they are returned later?
[7:36pm] Jason_Google: jack2009b: So the user signs into the App
Engine app with his harvard.edu address, but can't send email from
that address. But he can send email through the app if the sender is
set to the Gmail?
[7:37pm] maxr_google_: kardax: Yes, and I believe that's documented.
Let me find it for you.
[7:37pm] Jason_Google: Kardax: Yes, similar to Python.
[7:37pm] fkedupmonkey: @maxr_google okay, perhaps i'll just cache the
rendered HTML output for now. thnx for the tip on DB clearing
[7:37pm] maxr_google_: kardax:
http://code.google.com/appengine/docs/java/datastore/dataclasses.html#Core_Value_Types
[7:38pm] maxr_google_: fkedupmonkey: np
[7:38pm] Kardax: maxr: So this works for the low-level API, too?
[7:38pm] jack2009b: Jason_Google: no, when he saigns into app engine
as x...@post.harvard.edu, it actually signs him in under his
x...@gmail.com account. He can send email from my app as x...@gmail.com
but he wants the email to come from x...@post.harvard.edu.
[7:38pm] Jason_Google: Kardax: Yes.
[7:38pm] maxr_google_: Kardax: Yes, but the exceptions are a little
different.
[7:39pm] Kardax: Ok, thanks much
[7:39pm] maxr_google_: For example a String that's longer than 500
bytes will yield an IllegalArgumentException
[7:39pm] Jason_Google: jack2009b: Yeah, that's strange. Unfortunately,
I don't have any answers for you on that.
[7:39pm] jack2009b:
[7:39pm] jack2009b:
[7:39pm] jack2009b:
[7:40pm] jack2009b: (3rd try's a charm)
[7:40pm] Kardax: maxr: That makes sense.
[7:41pm] ryan_google1: jack2009b: are both the gmail and the
post.harvard addresses attached to the google account? and gmail is
the "primary" email address?
[7:41pm] ryan_google1: that might be it
[7:41pm] ryan_google1: (i don't actually know much about this, though,
so no promises on followup q's :P)
[7:42pm] Kardax: General datastore question: Does it make sense to use
Text exclusively (in place of String) to avoid the disk overhead of a
String's "simple index"?
[7:42pm] ryan_google1: kardax: if you're sure you'll *never* need to
query on those fields, then sure!
[7:42pm] ryan_google1: writes to that kind will be a little faster,
maybe, and a lot cheaper
[7:42pm] oktopus: I have an app which enables users to create their
own suggestions/comments list web page. Each user's web page is just a
datastore record linked to any comments which are also related
datastore records. Is there anyway that the user's page can get seen
by google spiders?
[7:43pm] ryan_google1: in the upcoming release we'll let you disable
indexing for any property, without resorting to hacks like Text
instead of String
[7:43pm] Kardax: Expanding on that, is there a way to remove the
simple index on other types that will never be queries?
[7:43pm] Kardax: Ahh, beat me to it
[7:43pm] ryan_google1: oktopus: we'll crawl it as long as you don't
have to log in to see those pages
[7:43pm] Kardax: I'm using a data structure that uses "gets"
exclusively, and I want to avoid all the overhead for queries that
will never run
[7:44pm] oktopus: My application is like a roll-your-own facebook
page.
[7:44pm] Jason_Google: oktopus: You should be able to use robots.txt
to control this, I believe.
[7:45pm] Kardax: Are there any plans to make "cron jobs" more
scalable?
[7:45pm] robokopped: oktopus, look into sitemaps.xml
[7:45pm] robokopped: * sitemap.xml
[7:46pm] oktopus: The users pages have links like
http://simplifyconnections.appspot.com/?place=Playground (notice the
question mark). So they are just virtual links, sort of.
[7:46pm] ryan_google1: kardax: very good call on avoiding indexing.
you should notice a drop in cpu when you switch to Text or use our new
feature
[7:46pm] ryan_google1: kardax: could you elaborate?
[7:46pm] Kardax: Elaborate on indexing or cron jobs?
[7:47pm] oktopus: So, does robot.txt still apply? I am not familiar
with robot.txt.
[7:47pm] Kardax: I am definitely looking forward to the CPU drop. In
my original python version (being converted to Java now), the CPU
usage was alarmingly high.
[7:47pm] maxr_google_: Kardax: elaborate on cron jobs
[7:47pm] Jason_Google: oktopus: http://en.wikipedia.org/wiki/Robots_exclusion_standard
[7:48pm] _robot: whoa
[7:48pm] Kardax: Ok, basically as a site grows and users increase
their content, the amount of work needed to update each user's
revelvant info increases.
[7:48pm] _robot: we have google people here?
[7:48pm] Jason_Google: Yes, it still applies. sitemap.xml that
robokopped brought up also applies.
[7:48pm] Kardax: It's not feasable to do it all in a single HTTP
request for everyone.
[7:48pm] Kardax: It could be done in multiple requests for batches of
suers.
[7:48pm] ryan_google1: kardax: ah, right
[7:48pm] ryan_google1: the short answer is, cron jobs are not a
general purpose, scalable offline processing solution
[7:49pm] Kardax: But the current cron system is very in flexible...
you can only run it so often and once at a time.
[7:49pm] Jason_Google: _robot:
http://groups.google.com/group/google-appengine-java/browse_thread/thread/465d1d1c3c75c40b
[7:49pm] ryan_google1: we have one of those in the works, and we're
looking forward to opening it up soon
[7:49pm] ryan_google1: but cron isn't it
[7:49pm] Kardax: That'll be helpful
[7:49pm] Kardax: My get-rich-quick pland doesn't need it now but it
could be useful later
[7:49pm] ryan_google1: glad to hear it!
[7:49pm] _robot: when will ssl be supported for non appspot sites?
[7:49pm] ryan_google1: we're looking forward to having people try it
out
[7:50pm] Jason_Google: _robot: I believe the short answer is when
browser security policies are modernized.
[7:50pm] Kardax: Regarding this future scalable offline processing
solution, do you envision it being useful for non HTTP applications?
(Scientific computer, for example)?
[7:50pm] ryan_google1: +1. details in
http://googleappengine.blogspot.com/2008/10/announcing-https-support-for-appspotcom.html
[7:51pm] ryan_google1: kardax: sure, maybe. app engine's focus is on
webapps, so those use cases will be better supported than general-
purpose distributed computing
[7:51pm] ryan_google1: but it will definitely be possible
[7:51pm] oktopus: So, I am asking how google spider would find a web
page that has "/?=..." as part of the address.
[7:51pm] _robot: i thought most modern browsers had support already?
[7:51pm] Kardax: Hmm... I think it might be interesting to make an app
engine raytracer that hands off a pixel to each node
[7:52pm] ryan_google1: interesting! as long as you don't need
realtime, at least
[7:52pm] Kardax: No, more like a complex pre-rendered scene, like a
frame from Toy Story
[7:53pm] Kardax: I bet amatuer 3d artists would be very interested in
scaling their work beyond the limitations of their own machines
[7:53pm] Kardax: Gonna rack up some hefty CPU bills, though
[7:53pm] Jason_Google: _robot: This isn't my area of expertise, but I
believe certificates can only be associated with a single domain,
which prevents us from offering SSL on third-party domains also. I
know there are proposals in the works to address this and other
limitations. I probably can't give you a better explanation than that,
unfortunately.
[7:55pm] Jason_Google: We'll be here for another 5 or so minutes, so
get your questions in.
[7:55pm] DocSavage: I noticed the Google Code team implemented
mercurial on top of bigtable. Has the App Engine team looked at their
implementation? A "mercurial" service for entities would be truly
awesome for my projects...
[7:56pm] Jason_Google: Or feel free to share any new applications
you've built.
[7:56pm] ryan_google1: versioning for the datastore is definitely a
common request
[7:56pm] Kardax: I'm though my original list of questions... trying to
think of more while there's still tiem
[7:56pm] fkedupmonkey: here's my app (under construction!)
http://best-website-ever.appspot.com/
[7:56pm] ryan_google1: it's unlikely that we'll do it, though, at
least in the foreseeable future
[7:56pm] Jason_Google: Ah fkedupmonkey, not very humble, are you?
[7:57pm] fkedupmonkey: @Jason_Google it's an artifact of the company i
work for
[7:57pm] robokopped: lol 1997 all over again
[7:57pm] ryan_google1: there are third party object versioning
libraries (both python and java) that should be easy to plug in though
[7:57pm] ryan_google1: that's what we usually recommend
[7:57pm] fkedupmonkey: but yet the most scalable 1997 ever
[7:57pm] DocSavage: ryan, what are your recommended python ones?
[7:57pm] Jason_Google: fkedupmonkey: Goodness gracious. The only thing
I'm missing is the blink tag.
[7:57pm] Kardax: I like the Netscape ad, fkedupmonkey
[7:58pm] fkedupmonkey: @Jason_Google view it in Firefox- WebKit
deprecated the blink tag, for reals
[7:58pm] ryan_google1: docsavage: don't know off the top of my head,
sorry
[7:58pm] Jason_Google: Oh, there it is.
[7:58pm] Jason_Google: Horray for WebKit.
[7:58pm] robokopped: LOL
[7:58pm] robokopped: <FONT SIZE=\"6\" FACE=\"courier\" COLOR=blue>
[7:58pm] robokopped: <MARQUEE WIDTH=100% BEHAVIOR=ALTERNATE
BGColor=yellow>
[7:58pm] robokopped: WELCOME TO MY WEB PAGE!!!
[7:58pm] robokopped: </MARQUEE></FONT>
[7:59pm] Kardax: OK, a question. I should probably know this, but oh
well. Do you automatically GZIP dynamic requestsin Java?
[7:59pm] oktopus: http://simplifyconnections.appspot.com/?place=I20W-Abilene
is an example
[7:59pm] fkedupmonkey: to whooever posted about the hit counter, i'll
be implementing that based on sharded counter
http://code.google.com/p/googleappengine/source/browse/#svn/trunk/java/demos/shardedcounter
[7:59pm] Jason_Google: Kardax: It's in here: http://code.google.com/appengine/kb/general.html
[8:00pm] Jason_Google: More specifically:
http://code.google.com/appengine/kb/general.html#compression
[8:00pm] Jason_Google: Short answer if you don't want to click: yes.
[8:00pm] Kardax: I see it I knew it happened for static stuff...
wasn't sure about dynamic.
[8:00pm] Kardax: Another: any chance for a datastore viewer in Java?
[8:00pm] maxr_google_: fkedupmonkey: Could you please file an issue
for the memcache exception you're getting?
[8:01pm] maxr_google_: Kardax: Yes, there will be a datastore viewer.
[8:01pm] Jason_Google: One is in the works. It probably won't make the
next SDK update, but watch out for it.
[8:01pm] fkedupmonkey: @maxr_google sure, will do. i figured it was a
user error. i'm not the best java user.
[8:01pm] DocSavage: When is the next SDK update?
[8:01pm] Kardax: Cool. I got good use out of the Python one
[8:01pm] Jason_Google: DocSavage: In the next week or so.
[8:01pm] DocSavage: Awesome.
[8:02pm] Jason_Google: OK, this marks the end of another awesome Chat
Time.
[8:02pm] oktopus: G`day
[8:02pm] DocSavage: I figured mine would be the last comment
[8:02pm] Kardax: Thanks for the time, it was incredibly helpful
[8:02pm] Jason_Google: We'll be back here on the 20th, 9-10 a.m. PDT.
Until then, join us in the discussion groups.
[8:02pm] DocSavage: Thanks guys. And thanks for the last SDK update
with unique user iD, etc.
[8:03pm] Jason_Google: Oh, I'll be posting a transcript of this
session to the groups as well, before the end of tomorrow.
[8:03pm] Jason_Google: So long!
[8:03pm] maxr_google_: adios!
Reply all
Reply to author
Forward
0 new messages