google indexing of weblocks sites
Jan Rychter
how do you deal with Google indexing your site as
sitename.com/someplace?weblocks-g1293=18%3A7DC7227A04E439269B4F01EF46BF8CFA
?
--J.
Daniel Janus
how about adding the following
to your robots.txt:
User-agent: *
Disallow: /*?
?
Best regards,
Daniel
Jan Rychter
Won't this mean that nothing will get indexed, as googlebot won't be
able to establish a session (which requires a redirect through a "?"
link)?
--J.
Jan Rychter
So, while we're pondering this problem, why does weblocks require a
redirect when opening a session?
--J.
Leslie P. Polzer
Jan Rychter wrote:
>> Won't this mean that nothing will get indexed, as googlebot won't be
>> able to establish a session (which requires a redirect through a "?"
>> link)?
>
> So, while we're pondering this problem, why does weblocks require a
> redirect when opening a session?
No clue. Send a patch to remove it if you like.
Jan Rychter
I will, because this makes all weblocks sites invisible to Google.
I'm trying to understand the rationale, though -- the description in
request-handler.lisp seems to indicate this was done in order to support
clients without cookie support.
Is supporting browsers without cookies something we care about?
--J.
Leslie P. Polzer
> > No clue. Send a patch to remove it if you like.
>
> I will, because this makes all weblocks sites invisible to Google.
> I'm trying to understand the rationale, though -- the description in
> request-handler.lisp seems to indicate this was done in order to support
> clients without cookie support.
>
> Is supporting browsers without cookies something we care about?
place for this.
Jan Rychter
>> request-handler.lisp seems to indicate this was done in order to support
>> clients without cookie support.
>>
>> Is supporting browsers without cookies something we care about?
>
> Yes, but I don't understand why we would need the redirect in that
> place for this.
Do we really want to support cookie-less sessions? Why?
Django does not support sessions without cookies:
The Django sessions framework is entirely, and solely, cookie-based. It
does not fall back to putting session IDs in URLs as a last resort, as
PHP does. This is an intentional design decision. Not only does that
behavior make URLs ugly, it makes your site vulnerable to session-ID
theft via the "Referer" header.
(this is from http://docs.djangoproject.com/en/dev/topics/http/sessions/)
I killed the initial redirect in my application, which did not solve the
problem entirely. I don't fully understand why, but only
(setf hunchentoot:*rewrite-for-session-urls* nil)
got rid of sessions in URL parameters entirely. Otherwise they would pop
up in strange places all of a sudden.
More fundamentally, though, I believe weblocks session handling needs to
be reworked, at least for my needs. I don't want to establish sessions
for every request, that's wasteful. Any externally-accessible URL can be
served without a session (possibly from a pre-rendered cache). It is
only when the user accesses actions that we really need a session.
This might have the added advantage of significantly speeding up
weblocks: you would not have to build the whole widget tree on every
request. My lazy-navigation does something similar to a certain extent,
but we could take it much farther.
--J.
Leslie P. Polzer
Jan Rychter wrote:
> Do we really want to support cookie-less sessions? Why?
It's a negotiable feature. Your proposal to make Weblocks
applications accessible without starting a session helps. :)
> More fundamentally, though, I believe weblocks session handling needs to
> be reworked, at least for my needs. I don't want to establish sessions
> for every request, that's wasteful.
Although probably not that serious.
> This might have the added advantage of significantly speeding up
> weblocks: you would not have to build the whole widget tree on every
> request. My lazy-navigation does something similar to a certain extent,
> but we could take it much farther.
IMO lazy-navigation misses an important feature. MAKE-LAZY-NAVIGATION
should be a macro that creates closures of its widget arguments to
build widgets as necessary.
More on-topic again then how about this two-step plan:
1. Disable session URI rewrite and prune annoying redirects.
2. Introduce session-less widget trees.
Jan Rychter
>> be reworked, at least for my needs. I don't want to establish sessions
>> for every request, that's wasteful.
>
> Although probably not that serious.
It would need to be measured. I have suspicions that building widget
trees on every request is expensive, especially as one usually makes
database calls when building widgets. And in case of a bounce request or
a bot request all that stuff sits around in memory until the session
gets expired.
>> This might have the added advantage of significantly speeding up
>> weblocks: you would not have to build the whole widget tree on every
>> request. My lazy-navigation does something similar to a certain extent,
>> but we could take it much farther.
>
> IMO lazy-navigation misses an important feature. MAKE-LAZY-NAVIGATION
> should be a macro that creates closures of its widget arguments to
> build widgets as necessary.
lazy-navigation was just a quick hack, something I noticed I could do
without much effort -- so I did it. It can definitely be improved upon.
> More on-topic again then how about this two-step plan:
>
> 1. Disable session URI rewrite and prune annoying redirects.
> 2. Introduce session-less widget trees.
Sounds good. My problem with (1) is that I don't understand why there
are three calls to redirect in handle-client-request and what purpose
they serve.
--J.
Leslie P. Polzer
Jan Rychter wrote:
> It would need to be measured. I have suspicions that building widget
> trees on every request is expensive, especially as one usually makes
> database calls when building widgets. And in case of a bounce request or
> a bot request all that stuff sits around in memory until the session
> gets expired.
You're right of course, but what I said was meant to only apply to
low-level session init, not the generation and storage of the whole
widget tree.
So I think we should separate the issue of session generation from
the issue of eager widget tree creation (which a proper lazy navigation
macro would solve).
> Sounds good. My problem with (1) is that I don't understand why there
> are three calls to redirect in handle-client-request and what purpose
> they serve.
My analysis of those three redirects, in order of occurence:
(1) seems to be unnecessary.
(2) seems slightly off too -- why would we need to prune the
session id from the URI if a cookie came in? Might warrant
some quick testing of what's going on and what removal
would change.
(3) is the only one that makes sense to me: when we're talking
to a non-AJAX client (or in some other way get passed the
action via normal GET) we want to remove it from the URI
to avoid it being called again on refresh.
In my opinion we should just remove the first two.
(3) is more of an issue; we should ensure that actions will
get called only once (or find some other solution for this)
before removing it.
Jan Rychter
>> It would need to be measured. I have suspicions that building widget
>> trees on every request is expensive, especially as one usually makes
>> database calls when building widgets. And in case of a bounce request or
>> a bot request all that stuff sits around in memory until the session
>> gets expired.
>
> You're right of course, but what I said was meant to only apply to
> low-level session init, not the generation and storage of the whole
> widget tree.
>
> So I think we should separate the issue of session generation from
> the issue of eager widget tree creation
I agree with this...
> (which a proper lazy navigation macro would solve).
...but not with this. No amount of lazy navigation macros will prevent
you from generating the front page, which in many cases will be
expensive to generate. I'd like to have a pre-generated cached version
for clients without a session (they all get the same content anyway) and
actually generate a new widget tree only when an action request comes
in. You don't need to generate a per-session widget tree for pages
accessible via navigation widgets, for example.
>> Sounds good. My problem with (1) is that I don't understand why there
>> are three calls to redirect in handle-client-request and what purpose
>> they serve.
>
> My analysis of those three redirects, in order of occurence:
>
> (1) seems to be unnecessary.
>
> (2) seems slightly off too -- why would we need to prune the
> session id from the URI if a cookie came in? Might warrant
> some quick testing of what's going on and what removal
> would change.
>
> (3) is the only one that makes sense to me: when we're talking
> to a non-AJAX client (or in some other way get passed the
> action via normal GET) we want to remove it from the URI
> to avoid it being called again on refresh.
>
> In my opinion we should just remove the first two.
Thanks, this makes a lot of sense. On my site I removed the first one a
couple of days ago and set hunchentoot:*rewrite-for-session-urls* to
nil. This got rid of the problem. So far things look good, although if
you turn off cookies in your browser, the site will not work beyond the
basic navigation (which is expected). I will kill (2) as well.
I suspect (2) was a way of dealing with hunchentoot's URL rewriting,
which gets in the way unless you turn it off.
--J.
Leslie P. Polzer
> you from generating the front page, which in many cases will be
> expensive to generate. I'd like to have a pre-generated cached version
> for clients without a session (they all get the same content anyway) and
> actually generate a new widget tree only when an action request comes
> in. You don't need to generate a per-session widget tree for pages
> accessible via navigation widgets, for example.
implement
those things.
Why does your frontpage rely on database access or other expenesive
things if you want it to be cached anyway? I'm not sure I'm getting
the
entire point here.
> I suspect (2) was a way of dealing with hunchentoot's URL rewriting,
> which gets in the way unless you turn it off.
Jan Rychter
[...]
>> I suspect (2) was a way of dealing with hunchentoot's URL rewriting,
>> which gets in the way unless you turn it off.
>
> But it's not supposed to jump in when cookies are supported...
That was my understanding as well, but it definitely does kick in, and
in unexpected ways that I do not understand. Try disabling the redirects
in request-handler.lisp and see for yourself.
--J.
Leslie P. Polzer
> That was my understanding as well, but it definitely does kick in, and
> in unexpected ways that I do not understand. Try disabling the redirects
> in request-handler.lisp and see for yourself.
urls* is
set to NIL?
Leslie P. Polzer
> How about just disabling the redirects if tbnl:*rewrite-for-session-
> urls* is set to NIL?