Skip to content

Only scrape if target urls < x

Right now I'm scraping far more urls then I post to, target urls just keeps on getting bigger and bigger and its not submitting as much as i want to

Would be great to have more control over when it scrapes/posts, maybe an option to only scrape if target urls are below a certain threshold


  • What program do you use for scraping?
  • Built in scraper
  • I'm not exactly sure what you are doing but when you scrape a list you suppose to clean that up before using it in a live project. This can be done in various ways but scraped URLs are always a big batch.
    After cleaning a million URL you may get a couple of thousands verified targets. Of course it all depends on what are exactly are you doing but you can get the idea.
  • Yes I am running a dedicated project just for scraping. Problem is that it's forever growing bigger and bigger since it seems to scrape more than it tries to submit.

    Would be great to have a setting to not have to manually switch on/off scraping
  • Are you aware that SER has an extra tool just for scraping? It's in Settings > Advanced > Tools > Search online for URLs

  • Isn't that faster than setting up a scraping project? 

    So you have a project set up where you scrape only and you manually export the scraped URLs to a different project to post?

  • Well can't tell if it would be faster actually, but what I'm looking for is more of a set and forget solution for scraping
  • Then you have 3 options.
    1. Easy:
    You set up SER to find targets and post to them. It's originally designed that way and it still works. You need google passed proxies and it will does it's job.

    2. Hard
    You set up a scraping rig with scrapebox, platform identifier and dropbox

    3. Lazy
    You buy lists.
  • Yeah I'm currently going with 1) but the problem is, as described in this topic, that the target list just keeps on growing and growing and will eventually become a problem. Would be great to be able to control it.
  • You mean the target list when you right click on a project and select show target URLs?
  • Yeah it just keeps on growing million by million, so I have to change the options for scraping back and forth from time to time

    Another feature that would be sweet could be a status like "Search only" but "Submit only"
  • If you turn off all the search engines then SER is submit only.
Sign In or Register to comment.