Back in December we talked about a project being launched by Wikia - the company behind Wikipedia - to create a new search engine based on some of the same principles used to create the popular encyclopedia.

Over the weekend Wikia announced that it had acquired Grub, a distributed crawler, from LookSmart:

Originally created in 2000, Grub leverages the distributed computing model to crawl the web and index pages. Users install a specialized client application on their computer, which then automatically performs indexing while idle and transmits page data back to a centralized repository. In this manner, volunteers will contribute the raw computing power that performs the indexing. Source: ArsTechnica

You’ve probably heard of other projects using this approach, like SETI@Home. You’ll download the Grub client and install it as a background process. When the computer isn’t actively being used (like when you’re at work, asleep, etc) Grub will kick in and start indexing the Internet from your computer. The information it indexes will then be sent back to Wikia and added to the database from which the company’s up-coming search engine will run.

Wikia will be releasing the Grub client as an open source project in the hopes of getting it ported to non-Windows platforms.

I’m actually looking forward to seeing how this plays out. It would be hard for any new search company to come in and catch up with existing players index-wise. I mean… Ask.com can’t catch up… and it’s had years. So maybe a distributed crawling approach will help beef up a new index quickly.

The problem with Wikia’s approach, I fear, is the human element. Don’t get me wrong - I’m a fan of humans. But with the introduction of human editing - like that found in Wikipedia - comes the introduction of corruption, misinformation, and mob mentality. You can see it on Wikipedia without much hunting.

Will Wikia’s search engine be the same way?

I’m not sure how much trust I can put into a search engine that reacts to the feelings of the masses. I trust Yahoo! and Google because I’m assured that people don’t alter search results. I’m fairly confident that when I search for something, regardless of how socially unacceptable or disagreeable the topic might be, I’m getting unbiased results.

How unbiased can human-edited search results be?

Only time will tell and I’ll be holding out final judgment on the venture until I actually have something in front of me to play with. And, as I already said, I’m really excited to see the distributed crawling play out.

What do you think? Can a wiki-style, editable search engine work? Would you use it? And how much trust would you put into something like that?