crawling the web

I had a job once that required that I write little robots that combed the web. These robots gathered information about things and that information went into our product that we would update our customers with. It gave me the idea that there is a lot of the web that is untamed or gets forgotten because our attention span is only so big. The web crawler solves this problem by searching for the stuff that you want to know about and then it only bothers you when it finds it. Currently I have a web crawler running for a few select people. It crawls the KSL classifieds website. I have won some really great deals off that crawler as have other people. When I get a chance I will get that crawler out here for you to play with. I have in the works another crawler that will let you gather friends/followers in twitter based on what you and they are interested in. I will post some links up when I get more of this stuff completed.

No comments yet.

Leave a comment!

You must be logged in to post a comment.