RDFa and robots.txt

[Kurt J]

Hello,

I've realized that MusicBrainz is not very robot friendly - the
robots.txt looks like:

User-agent: *
Disallow: /album/
Disallow: /artist/
Disallow: .....

I think the MB developers aren't too keen to change this. Does this
make RDFa useless? I'm second guessing myself now - should we return
to Ian's idea of a robot-friend subdomain? Any ideas are most
welcome.

-Kurt J

[Bob Ferris]

What's the difference with a sub domain? If a search engine should crawl
the RDFa enriched documents, then MB should allow it somewhere. Whether
it is then on a sub domain or on a top domain, makes no real difference
for me at the moment (however I'm not an expert, or I'm not getting the
whole idea here). They will have the traffic from the search engines or
any other client, which like to crawl these sites.

Cheers,

Bob

[Yves Raimond]

Hello!

I knew about that, but I am not sure it renders the RDFa useless, far
from it. For now, I would say leave the robots.txt as it is. Later on,
if there is a possibility to open it up a little, we could rate-limit
and point robots to another server, but it should (imho) be entirely
transparent (I don't know if djce is on this list, but I am sure he
would know whether that would be feasible or not): no
data.musicbrainz.org please :-/

Right now, the most important thing is to have dereferencable
identifiers for musicbrainz resources, so I think RDFa is definitely
still worth it!

Cheers,
y

>
> -Kurt J
>
> --
> You received this message because you are subscribed to the Google Groups "Music Ontology Specification Group" group.
> To post to this group, send email to music-ontology-sp...@googlegroups.com.
> To unsubscribe from this group, send email to music-ontology-specific...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/music-ontology-specification-group?hl=en.
>
>

[Nicholas J Humfrey]

On 3 Sep 2010, at 10:05, Yves Raimond <yves.r...@gmail.com> wrote:

> Hello!
>
>>
>> I've realized that MusicBrainz is not very robot friendly - the
>> robots.txt looks like:
>>
>> User-agent: *
>> Disallow: /album/
>> Disallow: /artist/
>> Disallow: .....
>>
>> I think the MB developers aren't too keen to change this. Does this
>> make RDFa useless? I'm second guessing myself now - should we return
>> to Ian's idea of a robot-friend subdomain? Any ideas are most
>> welcome.
>
> I knew about that, but I am not sure it renders the RDFa useless, far
> from it. For now, I would say leave the robots.txt as it is. Later on,
> if there is a possibility to open it up a little, we could rate-limit
> and point robots to another server, but it should (imho) be entirely
> transparent (I don't know if djce is on this list, but I am sure he
> would know whether that would be feasible or not): no
> data.musicbrainz.org please :-/
>
> Right now, the most important thing is to have dereferencable
> identifiers for musicbrainz resources, so I think RDFa is definitely
> still worth it!
>

I totally agree with Yves :-)

And perhaps there could be a robots.txt user agent whitelist...

nick.

[Kurt J]

Hello all,

thanks all for the feedback. sometimes i get stuck in these second
guessing cycles ;)

please see the increased RDFa support now at

http://mbz.catfishsmooth.net/

http://mbz.catfishsmooth.net/release/aaa64bfa-2f93-46ee-b6aa-48baf0775b4b

and related blog post here:

http://linkedbrainz.c4dmpresents.org/content/rdfa-test-server-online

-kurt j

> nick.