I had a scrapper project once where the wholesaler would provide updated pricing and limited product information to a resellers via xml which was downloadable from a passworded administrative area but the xml file failed to contain product descriptions leaving the resellers the arduous chore of manually creating descriptions for new products and the impossible chore of recognizing which existing products had updated descriptions.

The XML part was easy. A scheduled task ran nightly that used CFHTTP to log in and grab the XML file, it was processed updating prices, removing out-of-stock or retired products, and logged its results with CFLOG.

The descriptions was more difficult. The database only had a couple thousand products but that was enough to timeout the connection when trying to loop over all the products and use CFHTTP to scrap the description from the manufacturer’s website. The solution was to have a scheduled task run weekly that launched a .cfm template. That template would read a certain number of products from the reseller database (which had been updated in the XML process) which I think was 500. Then with those ids it would call a CFC which was set up as an asynchronous gateway. The CFC would loop through that smaller number of ids and use CFHTTP to get product descriptions avoiding timeout issues and of course log its results with CFLOG. Since it is an asynchronous gateway, several batches of ids could be fetched at once, that is, ids 1-500 could be scrapped simultaneously with ids 501-1000 and 1001-1500…

That was actually one of my favorite projects.