FAQ: What is the botlist web application?
Botlist, aka the-ghost-network(ghostnet for short) is about web content. This application is designed to find and extract relevant content from the web. And the data will be accompanied by intelligent agents to mine that data.
The idea is based loosely on a lot of the aggregation web 2.0 applications there are out there. I really liked the idea of del.icio.us. That started botlist in 2003-2004. Then reddit came along and really motivated me to start redesigning botlist.
However, botlist is different. These social-networking news applications are driven by users. Botlist is driven by aggregate bots. The bots (aka intelligent agents/crawlers) are spawned to crawl useful information on the web and aggregate a public database and web application.
Also, botlist is a collection of different programs. I am working on a specification for the web front-end. Basically the web front-end is synonymous with the "reddit/digg" clone. As new data comes in, data is viewable on the botlist front-end. I am also working on other front-ends in different frameworks, including a rails/django front-end. The second aspect of botlist is the crawler/backend bot agents. The third aspect is archiving the content.
FAQ: What is the purpose of botlist? Why couldn't I have just used Yahoo or Google? Or Reddit?
Or, you could use any of the other public repositories. Botlist is different in two ways. First, all source and all data will be under some form of liberal free/open/cc license. The source code for all projects will be openly available, as well as all data crawled and mined by botlist. Secondly, botlist is just different. There is a lot of content on the web. Some analysts found that there are 170 million websites on the net and billions and billions of web-pages. That is a lot of content. It doesn't make sense for us to only use 2-3 portals to look at that information. Botlist is just another tool used to look at the web.