Skip to content

Advice on SR and SV resolving?

wts3849wts3849 US
in GSA Keyword Research
Hi Sven,

I really love the packed features on KR. However, The resolution for SR and SV alone takes so much time and resources that I have a hard time justifying the use of KR.

It seems like if I set thread above 100, proxies get banned all the time. But If I set it at 10, then it will take days to resolve SR and SV. I don't even bother to resolve SEO-score.

 Do you have any advice on best performance settings? I use 35 private proxies.

Also, Would you consider adding the option to import all the SR, SV, CPC data from KeywordsEverywhere API? Because right now it does not seem to be pulling any data from it.

Please advise.
«1

Comments

  • wts3849wts3849 US
    I can share the API if you need.
  • wts3849wts3849 US
    Sorry, I forgot keyword tools don't offer SR. It has to be scraped by Scrapebox or Keyword Researcher...
  • SvenSven www.GSA-Online.de
    SR is scraped from a normal search where the amount of results is usually displayed on top of all results. This can not be made faster really as google needs it's pause between queries. This pause is also the same on a proxy, but more parallel queries can be made.
    If you accept SR coming from bing (or other search engines) instead of google, you can improve the speed a lot here as bing doesn't ban you that quick and your proxies would be fine as well. The results are not that much different either (at least for keywords I tested with).
    Thanked by 1wts3849
  • wts3849wts3849 US
    That's a great idea. I could use Bing.

    What about the SV part? KR is not quite filling it at all and won't really import from KeywordEverywhere, either. Any means around this?

    I appreciate your time.
  • SvenSven www.GSA-Online.de
    i need details on keyword/language and your api maybe to check why keywordanywhere would not work here.
    Thanked by 1wts3849
  • wts3849wts3849 US
    Keyword file and API sent by PM.
  • SvenSven www.GSA-Online.de
    will check this tomorrow as it's already almost 8pm here.
    Thanked by 1wts3849
  • wts3849wts3849 US
    Of course. Thanks again, Sven!
  • wts3849wts3849 US
    One more question, when the results are under six figures, they tend to lose zeros below 1K upon Excel import.

    I think it is just a US number formatting issue for the thousands separator being "." because original export files in txt or csv (; separated) all numbers appear correctly.

    Is there a way around this issue?



  • SvenSven www.GSA-Online.de
    thanks for noticing this...it's fixed in next update.
  • SvenSven www.GSA-Online.de
    latest update fixes the meta-data parsing
    Thanked by 1wts3849
  • wts3849wts3849 US
    edited April 2022
    Ohoho... My life's just gotten 1000x better. A million thanks to you! Everything works great now. I no longer have to go through SB, KEverywhere, KPlanner exporting/converting/splitting/combining/vlookuping data back and forth anymore.

    Just a little suggestion, would you consider adding a pause/timeout adjustability somewhere in KR? Because, I see that KR regulates the query submissions with mandatory pauses in-between no matter how many threads are being used and sometimes it just feels too long. I understand in the case of SEO-Score and SR resolving. But I don't really think we need that much pause for "Online Keyword Scraping" part for example. For SB, we often don't really have to use proxies for normal keyword scraping and IPs hardly ever get banned. At this point I still feel more comfortable scraping the initial keywords from SB and then import into KR.

    I am just hoping if I can work everything streamlined and focused within KR.

    Thanks.
  • SvenSven www.GSA-Online.de
    Thanks for the feedback.
    The pause between searches on keywords is by default 1sec and thats per ip/proxy. If you have e.g. 10 proxies, it will do 10 parallel queries on that same site, wait 1 second and do another 10.
    Just for google it is 10 seconds (changeable in the ini files). I really think this is good as it is as you might already spend more time parsing and preparing the next task than 1 second.
    However, maybe you mean something else where the pause is to big?
    Thanked by 1wts3849
  • wts3849wts3849 US
    edited April 2022
    Thank you for the info.

    Yes, I meant Google. It's just that in KR I clearly feel pause takes much longer than scraping whereas in SB I hardly notice the pause. But I haven't really compared in depth.

    Is the ini file ranking.ini? Can you tell me exactly where I have to go to adjust google pause?

  • SvenSven www.GSA-Online.de
    it's in c:\users\<login>\appdata\roaming\gsa keyword research\search\*.ini
    Thanked by 1wts3849
  • wts3849wts3849 US
    Let me test and see if I can find a sweet spot for google.

    Thanks.
  • wts3849wts3849 US
    edited April 2022
    Google.ini had by default 60 for searches and 120 for special searches, which is perfect for SEO-Score and SR scraping but too long for keyword scraping.
    I tested KR with my usual SB KW scrape settings as below and I could scrape 15000+ long tail keywords from Google without getting any proxies banned.
    35 private proxies, Thread (Connection) - 50, Wait between searches (Timeout) - 10
    Problem is, every time I need to scrape SEO-Score and SR, I have to go back to the file to change the values. Very likely, I am going to forget about it and end up getting the proxies banned over and over.
    1. Would you consider assigning independent timeout values for keyword scraping apart from other searches that require higher timeout value?
    2. What is extendet/special searches timeout for?
  • SvenSven www.GSA-Online.de
    next update lets you define a custom delay on the scraping form.
    Thanked by 1wts3849
  • wts3849wts3849 US
    Really sorry, Sven.

    I noticed SR results are very unstable on Bing. I don't know if it is just me experiencing this, but I thought I better report.

    Below are 4 different "SR only" scraping on Bing with country option "All".  If the specific country is chosen, the result seems more unstable.

    Very often random 374.xxx or outrageous 10-15 figure or mere 2-digit numbers appear. After many attempts, acceptable value appears, too. No matter what, all initially scraped values seem to be off.









    Google is also unstable, but it is rather N/A or 1,000, which seems to be a drop. But when the value appears, they seem to be in the acceptable range.

    Thanks.
  • SvenSven www.GSA-Online.de
    so what did you do to get all the different numbers? Just change the country/language or is it the same scraping but with different results?
  • wts3849wts3849 US
    edited May 2022
    Same scraping... Took multiple samples of 5-15 keywords to confirm. Same thing happened every time.

    Initial SR values are mostly off and they rotate randomly upon rescraping.
  • SvenSven www.GSA-Online.de
    ok thanks for the feedback. I will have a closer look on this tomorrow as it's already 5:30pm here.
  • wts3849wts3849 US
    No hurries. I just ran another test and I will post the 3 screen captures by order. Sorry for large images. 



    Google will mostly will fill SR with 1,000 which is basically a drop. And next run will fill with N/A.

    Thanks.
  • SvenSven www.GSA-Online.de
    The N/A switching from a rescrape is a bug that I fixed in next update.
    Though I can not reproduce the switching of numbers. Do you use proxies and what language/country/search engine for this test?
  • SvenSven www.GSA-Online.de
    I think the different numbers can come from the proxy ip and it's regional settings in case you are not using language/country parameters.
  • wts3849wts3849 US
    edited May 2022
    I uninstalled KR and reinstalled. Now I don't see the switching of the numbers. Apologies for misinforming.

    Now google SRs are quite accurate without any N/A. But I still see wrong values from Bing SRs with N/A. I tested on two different systems with various country parameters. Same thing. Let me fiddle with KR a little more and report.


  • wts3849wts3849 US
    So where exactly is the custom delay setting? It is not the user adjustable control?
  • SvenSven www.GSA-Online.de
    That delay setting in one option when scraping for new keywords.
    --
    bing SR: I will debug this.
  • wts3849wts3849 US
    edited May 2022
    Oh yeah, now I see it. Let me try it. Thank you!
  • wts3849wts3849 US
    edited May 2022
    Google still seems to follow ini file wait time setting regardless of assigned custom delay value.

    Bing no longer scrapes keywords online. "Sorry, no keywords found"

    Other sites seem to work well with custom delay value.

    Thanks.
Sign In or Register to comment.