I am not really sure what the cause is, I even tried to increase search engine query time to 60 seconds and that didn't help. I have 10 dedicated proxies from buyproxies as well. Let's see if anyone else has any clues...
are you really sure this didn't happen all the time but you just couldn't knew it??!
the reason i ask is because those log-warning messages were just introduced the past few versions so it may have happened all the time, but you just didn't know it.
however, i've notice nothing special the past few days and the proxies are working as always for me. i'm not saying that google didn't change anything though.
It started recently. My proxies got banned quite often but came back in an hour if I stopped using them. I uncheck custom time between queries and 14 hours so far without ban. But I'm using only one SE tho. Experiments
@Ozz the reason i ask is because those log-warning messages were just
introduced the past few versions so it may have happened all the time,
but you just didn't know it.
I think you are correct. I need to up the amount of proxies I use for the amount of scraping SER is doing.
Ya I'm using yahoo too. I don't know if I get this all the time but I know I'm getting it a lot now. I'm also seeing my LPM drop. Is it just me getting this?
Google RND queries random IPs of googles search-servers directly. what seems to be very good idea at first glance causing IP bans quickly as it seems (and is the opposite of what it was introduced for).
@seagul here is a thread (somewhere) where some users describe they have this problem now. im one of them. noone had a real solution.
what i've tried yet (without success)
- removing advanced operators like inurl: etc. it was my best bet that these get you banned extremely fast (because no mediocre user is using them, almost only scrapers)
- changing google countries (sometimes you only have a countryspecific ban)
what seems to help/what you can try:
- removing google RND (recommendation by sven, i'll leave this unchecked because it seems to get your proxies banned faster). It took now 12 hours instead of 2 to get all my private proxies banned again (...)
- using yahoo for at least 50% of your projects
- increase the search engine query time (maybe even back to the 60 seconds default)
what definitely helps:
- go through all your proxies 1 by 1. Check 1 proxy, let a sample project run and see if its blocked. Uncheck it. Rinse and Repeat. only keep the working proxies.
- do this daily.
This is extremely painful and i only did it once, just to see that my proxies got banned again...
Im only using the global site list now and will most likely switch to another software (not because SER not beeing a good tool but because i cant solve this problem).
I bought 3 new sets of proxies from different providers and used different options to avoid getting banned. Even installed SER on my home PC to see if i have a VPS problem or anything. Same thing over and over again. The funny thing is that my proxies are working in scrapebox and gscraper. So i could scrape lists and feed GSA, but i really don't like to babysit a software like that.
Ok thank you for your replies. I am now seeing more blocked Proxies again but not as many as when I had RND enabled. Funny because as you say the proxies check out OK in scrapebox. I see lots of threads about problems with proxies but I've never had a problem until now. Makes me wonder if Google have specifically launched a campaign against SER LOL
@LeeG well either the other party/parties on the shared proxies are hammering them or SER is triggering Google's blocking filter. I hope I can get to the bottom of it. If I thought private proxies would help I would buy them but I don't think it will help somehow.
Well seagul if youre using shared proxies you can never be sure who got the proxies banned Thats the first thing i would fix.
The proxycheck in scrapebox you mention doesnt really mean much because you dont know how it tests. Maybe it just checks whether the proxy can connect to google. But what makes me a bit curious is the fact that scrapebox can scrape with these proxies, no matter what. SER can't.
@leeg the problem with this method, imho, is that you have all Googles in there somehow. And some like google malta and the like yield extremely poor results. sometimes it shows 300 results while google.com shows 2 mio.
there are ~30 google engines that are useful (from my observation, correct me if im wrong) and mixing them with the poor ones may reduce the ban-risk, but at the same time it lowers the amount of new targets.
Im desperately trying to fix GSA for me, but at the moment i dont see a "real" solution. I dont know why some people are lucky with their proxies and some are not. While most advanced users here are using almost the same settings, it is alerting that some just lose all of their proxies while others dont.
Even with only using the sitelist i harvested so far i can rank pages, but its not a good feeling if you're limited with your maximum amount of links.
I am going to try @ozz method and see if it helps.Mine are dedicated private proxies and I am having the same problem, so, I can't blame someone else hammering my proxies!
I think it is the nature of the game, if you are going to scrape with advanced operators all day long then there is a good chance that some/all will get banned.
Comments
I think you are correct. I need to up the amount of proxies I use for the amount of scraping SER is doing.
It happens to us all
Shared proxies, means just that, others are using and sharing them.
Editing how you run ser, don't mean the others using them are doing anything different to stop google bans
They might even be the cause
How do you know they are even using ser? They might be hammering the proxies with scrapebox or some other software
One of the reasons I have been telling people to use a selection of random engines
Try the method of using random googles Ozz shared today
Ozz, don't expect people to do something that involves doing something and not having it provided on a plate for them
Using their brains, seems way beyond the comprehension of some in all honesty
Served on a plate or they moan
Some results are better than no results
No results means more calls on google.
More calls on google can results in your proxie getting banned for a period
So those engines you don't like, even though they might give poor results, they buy time between searches.
End result, less proxie bans
I think it is the nature of the game, if you are going to scrape with advanced operators all day long then there is a good chance that some/all will get banned.
The best solution is to simply buy some more.....
are GSA ser , must have very dedicated private proxies , too pass this?