Blekko a Silicon Valley based search engine, which raised an additional $2.5 million in late 2009, has been repeatedly touted as a next generation search technology capable of storing large amounts of data in a relevance algorithm that may actually strong-arm Google with a percentage of market share.
Blekko's founders, a group of silicon valley engineers (founders of DMOZ and TOPIX) tout ScoutJet as the next generation in search technology. The relevance algorithm surfaced on many web-server logs recently as ScoutJet and though the bot runs rather quietly and is obeying our robots.txt commands, it seems to be sucking up data at a rate faster than other bots.
Looking at the run times of search engines (Yahoo, MSN, GoogleBot) related to our server which hosts approximately 25 domains, shows ScoutJet indexing approximately 4% faster. ScoutJet resembles the old fashion meta crawler engines typical of its former AOL engineers currently partnered with the project, but may be looking at data a whole different way than that of Google and Yahoo.
Typically the Yahoo Bot and MSN Bots index repeatedly over our servers not stopping to check if the relevance of the content justifies its retrieval of data. Additionally, the GoogleBot indexes less frequently and appears to verify headers or changes in content prior to deciding whether to re-index the content. Although I am being quite broad in my interpretative viewpoint, there are significant differences in the way the GoogleBot and Yahoo Bot retrieve information related to news indexes. What I am now seeing is a new search engine that is leaving our server quicker. My interest after launch is to see what they are doing with our data and if they will include this data separately from a news aggregation perspective.
It might be a refreshing change to see some new technology given the world class intelligibility behind Blekko. Combining new and existing hardware challenges with an entirely re-worked theory of relevance might just be a fun alternative to the current searches available on the web.
Yet another alternative to the current search leaders may not be a valid market without viral growth, however, could potentially provide technology advancements likely to be purchased by larger tech firms. As we know, you don't compete with companies who have billions, you get bought out. ScoutJets personality from a tech POV can be viewed in your logs but bookmark it as the project status is still in pre-release. Either way it's going to be fun watching this company emerge knowing the leadership behind it and their successes in the marketplace respectively.
By AZR ©