#appengine IRC Office Hours chat transcript - Jul 20, 2011
Johan Euphrosine
Enjoy yesterday App Engine IRC Office Hours transcript below, next
session should happen Wednesday, Aug 3, 2011 7PM PST on
irc://irc.freenode.net/#appengine.
(Thanks a lot to dunker and his jsonbot for logging it)
--------------------------------------------------------------------------------------------
2011-07-20 18:01:26 CEST App Engine IRC Office Hours starts
--------------------------------------------------------------------------------------------
<dunker> Greg said there will be sample billings for the new billing
system in 1.5.2 but cant find it in the test version .. is that still
going to be in 1.5.2 ?
<moraes> dude what were those real time transcripts from io talks.
science fiction those were!
<dunker> moraes: would be a nice read
filipncs (~fili...@1385165222.dhcp.dbnet.dk) has joined #appengine
<marzia_google> @dunker, we will have sample bills very soon - but not
exactly when 1.5.2 is released
kytibe (~kyt...@212.174.109.55) has joined #appengine
<robertk> yeah surely there is a nice speak to text engine in g+
hangouts, right?
<dunker> ah ok cool
<proppy> robertk: not yet
<robertk> any of you googlers here familiar with the scheduler changes
that have been going on?
schuppe (d8ef2d04@gateway/web/freenode/ip.216.239.45.4) has joined #appengine
<rwilhelm23> Question on the scheduler - response time is the main
variable when it comes to determing whether to spin up more instances,
as opposed to cpi_ms and api_ms
<dunker> second question is is there already being tinkered with the
scheduler ? some ppl report different behaviour of their apps ..
quicker shutdown and start of instances
<dunker> ah rwilhelm23 is already on track
<proppy> robertk: which change have you been experiencing ?
<mbw> rwilhelm23: I would hope it would be request latency, using
cpu/api ms would not make much sense
<marzia_google> response time is a big factor - but also start up time
for your app
<itsPhilip> Already anyone from billing here?
<marzia_google> the question is: would we be better having a request
in a pending queue or starting a new instance
<dunker> interesting
kevlar_mac (~kevlar@nat/google/x-psdjhwmjktkgopat) has joined #appengine
<dunker> oi kevlar_google
kevlar_google (~kevlar@unaffiliated/eko) has joined #appengine
<proppy> dunker: you scared him
<dunker> hehe yeah ;]
<kevlar_google> Hi.
hacked (~vinces@pdpc/supporter/active/vinces) has joined #appengine
schuppe_ (d8ef2d04@gateway/web/freenode/ip.216.239.45.4) has joined #appengine
<robertk> @googlers ( proppy, marzia_google, tmatsuo_google) see
http://groups.google.com/group/google-appengine/browse_thread/thread/baf439a6e073f6da#
<dunker> well just out of curiosity how many parameters does the
scheduler take into account
<robertk> well, ,my post in that thread at least
<dunker> and do you have simulators to test it ?
<robertk> basically i keep seeing batches of instances get killed then restarted
<robertk> it is odd
<robertk> seeing on python apps. hr and ms
<robertk> also seeing stuff get killed quick
<tmatsuo_google> itsPhilip: do you have any question about billing?
I'm not from the billing team though
<mbw> the last post also mentions that always on instances are not
used at all, Ive seen that also
<robertk> yeah seen that too
<robertk> on an hr app
<itsPhilip> I am interested if Google will react to the recent traffic
price changes of amazon and msft.
<proppy> robertk: gathering some info about the instance behaviour changes
<proppy> robertk: I will get back here in a few minutes
<marzia_google> it's difficult to diagnose specific scheduler
questions in realtime on chat, so I'll forward the thread you
referenced - and also if you can provide app ids offline, that will be
helpful, and any relevant logging issues
<marzia_google> so in summary - need more information, will investigate
<robertk> you da man proppy
<itsPhilip> [msft & amazon made incoming traffic free and amazon also
decreased costs for outgoing traffic]
<robertk> marzia_google: yeah not sure i'm seeing specific logging
issues. just strange patterns with my instances
<proppy> robertk: idle-at-eviction for python had dropped, and we
have since added more capacity to adapt
<proppy> robertk: let us now if you see similar behaviour in the future
<marzia_google> our billing plans have not currently changed from
those announced at I/O
<nubbius> Hi, i've got a question
<robertk> we are all unfortunately aware.... :/
<robertk> nubbius: just ask it
<nubbius> Anyone know when the limit of URLFetch expand to more than 1mb?
<marzia_google> the request size for URLFetches will be 1mb
indefinitely - the response size was lifted to 32mb
<proppy> mbw: yes, dynamic instances will get priority over always on
instance now
<marzia_google> we have nothing new to announce there
vinces (~vinces@pdpc/supporter/active/vinces) has joined #appengine
<itsPhilip> marzia_google: Don't you expect google to get into the
pricing competition?
<robertk> proppy: funny thing is that i'm seeing instances actively
serving go away and get replaced. low mem usage
<uriel> question: when will billing be enabled for Go?
<Wooble> nubbius: 2 december 2010
<tmatsuo_google> itsPhilip: As far as I know, I don't think so,
especially in a short-term, but in a long run, there is a posibility
for us to change in the future because we want to offer competitive
price, but we can not compare IaaS and PaaS directly though.
<uriel> will that have to wait for the new billing system?
<robertk> proppy: ah, so is that change because always on is going away?
<nubbius> Marzia we use GAE to upload documents in Google Docs and use
this method, so we can't upload from our app docs bigger than 1 Mbyte
<itsPhilip> tmatsuo_google: Isn't Azure PaaS? ;-)
<proppy> robertk: scheduler will now send request to always on
instances only if there are no dynamic instance, or all dynamic
instances are busy
<marzia_google> @nubbius understood. it would definitely be nice to have
<dunker> is there any clue to be shared on how the transition with the
scheduler is going to take place ? as peoples billings depend on it is
there going to be a "big switch" day or something ?
<robertk> proppy: interesting
<proppy> nubbius: maybe you could bug the Docs API team to provide
resumable upload ?
<nubbius> Also we have an application that treat personal data
(nubbius.com), and we would like to know when SSL will be avalaible
for GAE custom domain apps
<proppy> nubbius: aa_ is a Googler working on Docs API
<marzia_google> we always look to optimize the scheduler,
independently from any billing changes
<proppy> aa_: he might be able to help when back online
PSIlits (~PSI...@165.170.194.130) has joined #appengine
<marzia_google> with the new billing we will provide more options for
you to trade off performance and instances
<dunker> thing is changing the scheduler changes the bill
<marzia_google> basically giving you the option of choosing not to use
aspects of the scheduler
<ikai_google> I'm back. I HATE TIME ZONES, real meeting is at 1
<dunker> re ikai_google
<tmatsuo_google> itsPhilip: ah sorry, but we're still confident our
current pricing model is competitive :)
<proppy> ikai_google: ahah
<dunker> ah thats nice marzia_google
<kevlar_google> ikai_google: gcal ftw :)
<proppy> ikai_google: http://www.tmzns.com/
<ikai_google> I'm USING gcal
<marzia_google> eg for 1.5.2 as in the release notes, we will allow
you to modify some scheduling parameters
<kevlar_google> lol
<ikai_google> eastern time and bogota are not the same thing
<ikai_google> one accounts fot DST
<ikai_google> sigh
<ikai_google> facepalm
<marzia_google> but you can always opt to use the scheduler as designed
<dunker> yeah like that
<itsPhilip> marzia_google: Is it still Google's current plan to roll
out the new pricing before multi threaded python instances will be
available?
<kevlar_google> ikai_google: mine asks me to change my timezone
whenever my laptop's timezone changes, ymmv
<dunker> greg said there were going to be knobs but this makes it more
clear thnx
<ikai_google> kevlar_google: dude that's a total YMMV, it didn't ask
me to change mine
<marzia_google> yes, see greg's FAQ re: half price instances for python
<ikai_google> kevlar_google: so I manually set it .. you would think
all GMT -5s are equal. they are not
<dunker> owja any ETA on python threads ?
<dunker> hehe
<ikai_google> dunker: with 2.7
<dunker> ok cool
<robertk> so with the new scheduler will we get penalized for high
latency requests?
<nubbius> @aa_. Could you help us with limit upload in Google Docs
from GAE? It's reported, viewed in user groups but there is not a
solution to avoid 1 MB limit
PSIlits (~PSI...@165.170.194.130) has joined #appengine
<robertk> ie will we be able to adjust it so that we'll still get more
instances?
<uriel> marzia_google: will Go instances be one-fourth price? Go has
much, much, much lower memory usage than either python or java ;)
<marzia_google> in the future we will allow you to pay to keep more
instances around, yes
<dunker> the knobs are a interesting concept as users are able the
tune themselves
<marzia_google> we don't have the details for go pricing at this time
<itsPhilip> Hypothetical Question: What average price increase does
Google expect internally?
<dunker> any details of the "open source" variant ?
<robertk> marzia_google: was that a response to my question?
<uriel> marzia_google: I see, so no billing for Go until new billing
system then? it would be nice if pricing for Go took into account its
much lower resource consumption ;)
<proppy> itsPhilip: I think those details will come along the pricing calculator
<marzia_google> @robertk - unless you don't think so, yes :)
<ikai_google> uriel: a lot of things would be nice
<robertk> marzia_google: actually i don't.... ;)
<mbw> marzia_google: so, if requests are still over 1 second, it will
still punish you by not spinning up instances? even though the
pending queue latency will go up and it should spin up more instances?
<proppy> robertk: please rephrase then :)
<marzia_google> haha ok, so to be more clear
<marzia_google> we will allow you to adjust the scheduler to be more
or less agressive in spinning up instances
<uriel> ikai_google: I understand :) I just think it would be unfair
to charge the same for Go instances when they use a small fraction of
the memory/cpu
<marzia_google> right now the knobs we are releasing soon will allow
you to introduce more latency and have fewer instances, and take
instances down more aggressively
<Wooble> uriel: presumably each instance will be able to handle
multiple requests at some point.
<marzia_google> in the future we will introduce knobs that do the opposite
<robertk> marzia_google: so right now requests with > 1000ms latency
will get fewer resources, they'll get throttle_code=X in the logs, and
get into a very bad feedback loop for a while. if we're abel to
adjust the scheduler, will we still face issues with high latency
applications?
<marzia_google> allow you to more aggressively keep instances around
<mbw> marzia_google: robertk and I understand that, we are
specifically asking if the 1s punishment barrier is still in effect
<uriel> Wooble: that would be nice too, obviously :)
<marzia_google> so if you are using the auto-scheduler, yes
<marzia_google> if you choose to adjust the scheduler to be more
aggressive at spinning up instances
<marzia_google> that is another story
<ikai_google> uriel: we try to make pricing as fair as possible.
HOWEVER: https://groups.google.com/d/msg/google-appengine/ob-kMuDAAqc/JGvSGLMkvP0J
<itsPhilip> There was a heavy hr outage recently. I looked like you
moved all hr apps to another datacenter. As a consequence the average
latency decreased by a factor of 2-3. How can we rely on the new
pricing structure if the latency can vary that much?
schuppe (d8ef2d04@gateway/web/freenode/ip.216.239.45.4) has joined #appengine
<marzia_google> @itsPhilip to what HR outage are you referring?
Google_Wesley (d8ef2d04@gateway/web/freenode/ip.216.239.45.4) has
joined #appengine
<uriel> ikai_google: well, to be able to enable billing for Go at all
would be an improvement from now :))
<marzia_google> we had an outage on thursday that primarily affected M/S apps
<itsPhilip> I don't know anymore when it was @mbw can you remeber it?
You contacted google because of it.
<mbw> marzia_google: we saw a few odd things on HR, I wouldnt call it
an outage. We just say memcache flushed (which is fine) and a few
more timeouts than normal on datastore
<robertk> itsPhilip: i think HR was up, but memcache was flushed
<marzia_google> HR apps are served out of multiple datacenters by design
<marzia_google> yes, we had an issue with more aggresive than usual
memcache flushing
<ikai_google> uriel: The Go stuff still needs a lot before I'd say the
API is close to complete
<marzia_google> but that's not related to HR
<mbw> marzia_google: thats misleading... they are only served out of
ONE datacenter at a time
<ikai_google> mbw: not completely true. the datastore instances are in
multiple data centers
<mbw> yes, but the app != datastore
<uriel> ikai_google: I see, and there wont be billing until it is
API-complete? (that seems reasonable, just wondering)
<ikai_google> mbw: app > datastore and serving
<ikai_google> it's a superset.
<ikai_google> if the front end web serving goes down, we failover to a
second data center
<kevlar_google> "oh oh, it's magic! You know..."
<uriel> ikai_google: also, while I agree that things aren't always
fair, to charge the same for Go instances would mean it is always
unfair ;) I'm quite certain Go always uses considerably less memory
than Java or Python, and has much lower startup time
<robertk> yeah but you're memcache doesn't follow. it will be reset
too in that case
<Google_Wesley> kevlar> people are too young here to remember that
song (except for the occasional commercials)
<marzia_google> Keep in mind that you can't just charge for memory,
apps also use CPU
<peper> this was probably asked before, but any plans on streamlinging
the M/S -> HR migration?
<ikai_google> peper: yes
<Google_Wesley> peper> yes, we are working on it
<marzia_google> hence the move to instance billing
<peper> any ETA?
<marzia_google> vs just CPU or memory
<ikai_google> peper: no
<nubbius> we have an application that treat personal data
(nubbius.com), and we would like to know when SSL will be avalaible
for GAE custom domain apps
<peper> ok
<ikai_google> peper: soon
<peper> thanks for info :)
<ikai_google> nubbius: soon, ideally
<ikai_google> as a general rule we rarely give ETAs
<itsPhilip> So what are the consequences if the latency increases by a
factor of 3 on all apps? Do we have to pay that or will this be
covered by the SLA?
<ikai_google> the exception is if a prerelease goes out
<peper> yeah, i noticed
<ikai_google> we can tell you that it'lll probably be a week-a month,
and we can also tell you if we find too many bugs with a feature and
it probably *won't* go out
<Google_Wesley> "we are working on it." in general, anything on our
public roadmap means there is at least one engineer actively working
on it.
<uriel> marzia_google: cpu-based billing worked great :(
<robertk> hi Google_Wesley. didn't notice you lurking in here ;)
<marzia_google> but you didn't pay for memory :)
<proppy> guido is having an hangout on G+
<proppy> feel free to pm me your email if you want to join
<ikai_google> itsPhilip: latency is often the result of multiple
factors. the SLA currently only covers uptime. We are not planning on
per API SLAs
<dunker> proppy: get guido over here would be better ;]
<uriel> marzia_google: I wouldnt mind paying for memory too, if I'm
going to be using Go I will use very little memory ;P (but I
understand that it is tricky to account for everything)
<itsPhilip> marzia_google: Fun question: Where does google buys its
memory? I can rent a server with 6core amd, 32gig ram and 20TB gbit
traffic for 130/month? Maybe you should outsource your hosting ;-)
<mbw> marzia_google: We are lobbying for more ram on front ends... we need it
<marzia_google> i'm fairly certain we get it from Best Buy, right?
<marzia_google> (that was clearly a joke, to be explicit)
<itsPhilip> (my offer is from a huge hoster with 4 datacenters; they
will expand to the usa soon)
<proppy> dunker: he seems to really prefer the hangout format
<dunker> yeah just kidding
<peper> marzia_google: otoh allowing one instance to serve multiple
requests in parallel would lower the memory usage :)
<proppy> let me know if you want to join, so I can invite you
<Wooble> itsPhilip: it seems unlikely many hosting companies would
meet google's needs :P
<ikai_google> itsPhilip: those are good prices. You should do what's
best for you. Me? I've spent enough time in my life driving to data
centers kicking servers back to life
<dunker> thats why IRC needs to be integrated in Google+ ;]
<dunker> proppy: i dont have a cam so i cant speak in hangouts only type chat
<Google_Wesley> proppy> can you IM us?
jfmontesdeoca (480ee551@gateway/web/freenode/ip.72.14.229.81) has
joined #appengine
<itsPhilip> ikai_google: Yeah thats the problem ;-)
<ikai_google> and yes, sometimes, you physically need to kick a server
for it to start working again (my guess is weird fan stuff)
<marzia_google> yes we are working on multiple requests/instance for
languages, which java always have
<dunker> which sucks cause i dont have a mic as well
<marzia_google> random comment: you should definitely turn on
threadsafe flag for java
<proppy> Google_Wesley: give me you google+ email can I can invite you
<peper> Can i just join to listen? no webcam
<Wooble> ikai_google: reminds me of the floppy drives on my commodore 64 :)
<Google_Wesley> geoff> i loved my C-64! :)
<itsPhilip> A few months ago I asked Gregory D.. if prospective search
will be free of charge, he wanted to check it but I've missed the
answer: Will it be free of charge?
<proppy> if you have question about App Engine billing, Appstats,
domain console feel free to join guido hangout
<marzia_google> @itsPhilip we are currently not charging for PS, since
it's experimental. But in the future we will, yes.
<ikai_google> lol billing?
<ikai_google> you mean the billing console
<ikai_google> guido can't answer any questions about billing *plans*
<proppy> ikai_google: yep sorry :)
<Google_Wesley> yes
<proppy> thanks for watching my words out
<ikai_google> =)
<ikai_google> I would feel bad if he got a bunch of "what if this"
"what if that" questions
<ikai_google> the man isn't an insurance salesman.
<Google_Wesley> he would get mad
<Google_Wesley> don't make him mad... you woudln't like him when he's angry
<dunker> hehe
<dunker> would be fun to see Guido go mad ;]
<Google_Wesley> although he isn't "cloud hulk"
<Google_Wesley> someone else is... or at least acts like it
<Wooble> can we ask Guido about Python's uncertain future? ;)
<ikai_google> is it uncertain?
<Wooble> I read on the mailing list that it is, so it must be.
<uriel> if you read it on the internets, it is a fact
jfmontesdeoca (480ee551@gateway/web/freenode/ip.72.14.229.81) has
joined #appengine
<Google_Wesley> geoff> what is uncertain?
<Google_Wesley> oh, i need to find that... is that in the Python forum?
<itsPhilip> There was a discussion in the mailing list regarding the
complex TOS. Do you plan to simplify it? The Azure TOS are a good
example of easy to understand TOS.
<Wooble> I don't know; apparently having the BDFL work on a 2.7
runtime isn't an indication Google's going to stick with python.
<Google_Wesley> itsPhilip> we also have a PP paired with the ToS
<ikai_google> itsPhilip: yes, I believe we are
<dunker> real questions is when will 2.7 be here .. but such questions
cannot be answered ;]
<Google_Wesley> Wooble> people don't get it
<ikai_google> itsPhilip: changing legalese though is non trivial. have
you ever heard of "sh*t's easy syndrome"?
http://steve-yegge.blogspot.com/2009/04/have-you-ever-legalized-marijuana.html
<marzia_google> @dunker - does the answer "we're actively working on
it" make you happy
<marzia_google> :)
<dunker> ikai_google: i liked that article
<itsPhilip> PP
<ikai_google> so did I. I'm throwing that term out a lot nowadays.
<dunker> marzia_google: havent heared that one before ;]
<itsPhilip> Yes, the azure pp is: "Don't host anything that is illegal
on azure". I've talked to one of their managers they won't even judge
by theirself what has to be considered illegal. Thats the way to go
IMHO.
<dunker> ikai_google: its indeed all about complexity besides the weed thing
<Wooble> so can I host a website about tiananmen square on Azure?
<itsPhilip> ikai_google: Wow that article is too long to read ;-D
<dunker> itsPhilip: its a nice read though
<Google_Wesley> wooble> yes, just choose to host it in asia... china
if possible.
<dunker> 18:41 < jsonbot> [jsb-debian] - jsonbot 0.7.1.2-1 MIGRATED to
testing (Britney) - http://tinyurl.com/44zntcv
<dunker> yess ;]
<dunker> bit offtopic sorry bout that
<itsPhilip> Google_Wesley: Thats a mean comment ;-)
rsaccon (~rsa...@a85-138-104-86.cpe.netcabo.pt) has joined #appengine
<itsPhilip> Sooo anything new about nextgen queries? =)
<rwilhelm23> Question about the differences in time between the
appstats and the dashboard: appstats will show a consistent "real"
time for a bunch of requests, but the dashboard will show some took
twice as long as others - what are the sources of that "extra" time?
<Google_Wesley> itsPhilip> more "in-progress" now than in the past year...
<ikai_google> itsPhilip: you are getting no exploding indexes required
<ikai_google> that's part of it.
<itsPhilip> Ah nice
<ikai_google> it's in the next release
<Wooble> you should just call that "nextgen", then add "third
generation queries" to the roadmap. :)
<ikai_google> see prerelease notes
<proppy> rwilhelm23: Google_Wesley is asking guido in the hangouts
<Google_Wesley> Guido says> if they're loading requests, Appstats
doesn't start its clock until after the app has initialized, i.e., it
doesn't count imports, etc.
<Google_Wesley> Guido also says> Appstats takes measurements from
point A to point B, but after that, the request isn't quite sent back
to the Appserver because Appstats is pure userspace and just wraps
low-level calls
<Google_Wesley> that's all he can think of at this time
itsPhilip (~Philip@unaffiliated/itsphilip) has joined #appengine
<rwilhelm23> thanks
Leemp (~l...@c-67-183-124-170.hsd1.wa.comcast.net) has joined #appengine
<proppy> robertk> asking on the hangouts asking about C/extension on Python 2.7
<itsPhilip> Why doesn't Guido join the chat?
<ikai_google> I think he said something about preferring the hangouts format
<Google_Wesley> itsPhilip> he and IRC don't get along very well
<proppy> guido> said he couldn't comment publicly yet, but that it
will be open for testing internally first
<dunker> IRC is too nerdy for guido ;]
<proppy> and then to truster tester
<Wooble> having visited #python, I don't blame him :)
<dunker> so it *is* going to be there
<Google_Wesley> can you imagine if you were him, get onto IM, and see
"OHHHHH, it's GUIDO... HIIIIIIII GUIDO!!! I LOVE MY PYTHONZ"
<dunker> didnt know that
<proppy> guido> no plan for allowing people to upload their own C
extension in the first phase of the project
<dunker> Google_Wesley: its tough being a hero ;]
<proppy> guido> but plan to provide access to popular C extension
<rwilhelm23> Cpickle would be nice, but I'm not holding my breath
<Google_Wesley> basically, it needs a LOT of testing to make sure
things don't break before 2.7 support goes out
<mbw> proppy: how about providing lxml? cpickle? etc etc out of the box?
<dunker> do want to say that to Guido someday
<Wooble> sandboxing user-contributed C extensions seems... fairly difficult.
<itsPhilip> If he also has hot groupies? lol
<mbw> Wooble: thus NACL
<proppy> mbw: that I think that's the provide access to popular C extension part
<mbw> I am still amazed that the community hasn't been a lot more
vocal about library support and fast serialization support... it's a
constant pain in my ass
<moraes> i've met guido editting a google doc. i said "OHHHHH, it's
GUIDO... HIIIIIIII GUIDO!!! I LOVE MY PYTHONZ" but he ignored me.
<dunker> moraes: ;]
<dunker> you need to think of a clever question
<proppy> mbw: guido> cpickle is on the list, dunno about lxml he will ask
<dunker> like when is the GIL going to be removed
<dunker> hehe
soyapi (~soy...@41.78.59.2) has joined #appengine
<rwilhelm23> Did you see Brett's post on that?
<tmatsuo_google> if I remember correctly, lxml is on the list
<dunker> proppy: simplejson with accel ?
<proppy> mbw: guido> lxml would probably be on the list too
<marzia_google> ok, i'm off
<moraes> or "why is max a built in function?"
<marzia_google> thanks for coming
<dunker> marzia_google: thanks !
<dunker> ow
<dunker> thats quick
<dunker> learned a lot today though it was usefull
<moraes> damn lxml would be awesome
<Google_Wesley> cPickle is very likely
---------------------------------------------------------------------------------------------
2011-07-20 19:00:25 CEST App Engine IRC Office Hours closes
---------------------------------------------------------------------------------------------