Building a niche search engine

[Lukas Bogacz]

I'm considering the idea of building a niche search engine. The goal would be to only index programming related information, starting with a particular tech stack. We could do this by starting with a curated list of domains and carefully crafting a set of crawling rules. Could the group please give me an idea as to how difficult I should expect this to be, and what are the largest issues I should expect to run into? Is it a matter of paying for bandwidth or more of a technical/engineering problem?

For context this isn't intended as a solo project, but rather as a startup. We're trying to estimate how much we should expect it to cost us and how long it would take in order to decide if to pursue the idea. The engine would be used by LLMs in order to provide them with background information, so we would make design choices around that.

I would appreciate any guidance in the matter.

Lukas

[Rich Skrenta]

I have died on this hill. Happy to chat.

Rich

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/common-crawl/00dda90f-87a0-421d-a3a3-3d5f99636188n%40googlegroups.com.

[Daniel Halstrom]

We got a full backlink index working with roughly 12 months of crawls and the ability to add more crawls working a few years back.

Server cost to keep it up was roughly 400$ for 2 .   Ran 2, update 1 at a time and swap them.   Could index anchors, titles and backlinks only.    No permutations or content indexing.

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/CACprwLdOOkSvVQ1TZkz8oUGHKuktkyP8iqUP0uXcd3RQqPd00g%40mail.gmail.com.

[Sandra Borst]

I want to unsubscribe, thank you.

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/CAE1Ub8uO7XE-CjFWzXhc1Mi5Y6zdojqiYcmg5KxcydQ2%2BEo7Zg%40mail.gmail.com.