Scraping Search Engines is SLow

Cyberwebby

Hello,

I'm wondering why scraping the search engines is so slow compared to some other tools that I use like Scrapebox. I ran this GSA website contact overnight to scrape the engines and it only scraped collected 1000 URLs over a period of like 8 hours, that's slow, - and this is even including using Bing as one of the search engines. Whether using private proxies or not, I always gotten a lot of fast and plentiful results from Bing with other tools because Bing tends to be very liberal with scraping. Hey, I'm not trying to compare this head to head with Scrapebox's scraping capabilities, but I'm just wondering why this is producing drastically slower results compared to it. I was expecting it to produce more and faster scraping being all the search engines options and settings that GWC has.

romingson

Because gsa product must working all in one that means need to scrape ,check and sent will give slow.

Cyberwebby

I just had it on scrape and it still has been extremely slow. Tested several times.

TheGypsy

Just wondering have you tried the search engine parser from the advanced tools or simply using project to scrape?

I have no idea how SER actually works just what I've picked up from the forum during the years but SER does things in batches. It goes to scrape, saves the URLs, picks up some URLs then posts then verifies then can do a lot of other things that are set up. So I guess when you just set up for scraping only it just skips some of the steps but probably that takes time hence it slows down.  

Or I could be totally wrong and it's a bug