In short, Alexa, an
Amazon-owned search company started by Bruce Gilliat and Brewster
Kahle (and the spider that fuels the Internet Archive), is going to
offer its
index up to anyone who wants it (details are not up yet, but
soon). Alexa has about 5 billion documents in its index - about 100
terabytes of data. It's best known for its toolbar-based traffic and site
stats, which are much debated and, regardless, much used across
the web.
OK, step back, and think about that. Anyone can use Alexa's
index, to build anything. But wait, there's more. Much more.
Anyone can also use Alexa's servers and processing power to mine
its index to discover things - perhaps, to outsource the crawl
needed to create a vertical
search engine, for example. Or maybe to build new kinds of
search engines entirely, or ...well, whatever creative folks can
dream up. And then, anyone can run that new service on Alexa's
(er...Amazon's) platform, should they wish.
It's all done via web services. It's all integrated with
Amazon's fabled web services platform. And there's no licensing
fees. Just "consumption fees" which, at my first glance, seem
pretty reasonable. ("Consumption" meaning consuming processor
cycles, or storage, or bandwidth).
The fees? One dollar per CPU hour consumed. $1 per gig of
storage used. $1 per 50 gigs of data processed. $1 per gig of data
uploaded (if you are putting your new service up on their
platform).