Thursday, December 24, 2009

Of Bots, Spiders and Scrapers

In Chapter 1 of his book Webbots, Spiders and Screen Scrapers A Guide to Developing Internet Agents with PHP/CURL, Michael Schrenk says :

"To be successful with webbots, you need to stop thinking like other Internet users. Namely, you need to stop thinking about the Internet in terms of a browser viewing one website at a time. This will be difficult, because we've all become dependent on browsers. While you can do a variety of things with a browser, you also pay a price for that versatility - browsers need to be sufficiently generic to be useful in a wide variety of circumstances. Webbots, on the other hand can be programmed for specific tasks and can perform those tasks with perfection."

That really captures the essence of what this book is about. Schrenk targets two very different audiences - one is the webbot developer who has a variety of projects to play with and learn the tools in the context of the different applications. The other is the business leader who is eager to develop an unique on-line strategy to gain competitive advantage and realizes that it needs to be a lot more than having on-line presence.

As a consultant, I find myself needing to roll up my sleeves and do actual development on occasion - even if only as a proof of concept. On a more regulare basis, I need to help my clients define the best use of technology to achieve their organizational goals which more often than not includes doing better than their competition.

I found this book very useful in understanding what webbots, spiders and scrapers are good for. It was easy to see how it can free up time from tedious tasks and  open up opportunities that we have not even thought about. Schrenk gives the reader in a head start in terms of how to starting thinking about a webbot project. If the reader keeps in mind his advice about stop thinking like other Internet users, they would soon find themselves coming up with ideas of their own. The possibilities are myriad. For those who have the ability and inclination to try out some of their ideas, Schrenk includes a wealth of resources in the book and there is yet more on his website.

Whether you are a developer , a business leader or have a function that is neither of them, you will find this book valuable. In the least it will help you think about the true potential of the internet and about ways to tap it.

No comments: