[ANN] grobotstxt - a Go port of Google's robots.txt parser and matcher

[Jim Smart]

Hi there fellow Go Nuts and Gophers,

I have ported Google's robots.txt parser and matcher, from C++ into Go — including all the tests — and given it a quick API cleanup to make it a little more Go-like.

https://github.com/jimsmart/grobotstxt

— Maybe this is useful to someone here?

Regards,

/Jim

[Ben Hoyt]

This looks really nice! I was looking for something like this recently, but ended up using https://github.com/temoto/robotstxt. However, something that's closer to the Google version might well have made me choose that.

A couple of comments:

1) I always love packages that only depend on the Go standard library (and I know I'm not the only one who thinks this way). Your package pulls in ginkgo and gomego and x/tools, but they're only used for the tests. I think it'd be a selling point if it only depended on the stdlib's "testing" package. You could fairly easily rewrite these tests as table-driven tests and avoid those 3rd party packages.

2) I think a bit more cleanup would be good to make it a bit more Go idiomatic, for example merging the _cc and _h files into one (that's an artifact of C++).

Cheers,

Ben

[Jim Smart]

On Monday, April 27, 2020 at 12:45:47 AM UTC+1, Ben Hoyt wrote:

This looks really nice! I was looking for something like this recently, but ended up using https://github.com/temoto/robotstxt. However, something that's closer to the Google version might well have made me choose that.

A couple of comments:

1) I always love packages that only depend on the Go standard library (and I know I'm not the only one who thinks this way). Your package pulls in ginkgo and gomego and x/tools, but they're only used for the tests. I think it'd be a selling point if it only depended on the stdlib's "testing" package. You could fairly easily rewrite these tests as table-driven tests and avoid those 3rd party packages.

2) I think a bit more cleanup would be good to make it a bit more Go idiomatic, for example merging the _cc and _h files into one (that's an artifact of C++).

Thanks for the interest and for your feedback! 

I have intentionally kept the code layout fairly close to the original library, it makes it much easier should I ever have to patch the code based upon changes in the original project. I've always found that easier when maintaining ported code.

Cheers,

/Jim