Skip to content

Only scrape if target urls < x

qlitheqlithe
in Feature Requests
Right now I'm scraping far more urls then I post to, target urls just keeps on getting bigger and bigger and its not submitting as much as i want to

Would be great to have more control over when it scrapes/posts, maybe an option to only scrape if target urls are below a certain threshold

Comments

  • TheGypsyTheGypsy Madrid
    What program do you use for scraping?
  • qlitheqlithe
    Built in scraper
  • TheGypsyTheGypsy Madrid
    I'm not exactly sure what you are doing but when you scrape a list you suppose to clean that up before using it in a live project. This can be done in various ways but scraped URLs are always a big batch.
    After cleaning a million URL you may get a couple of thousands verified targets. Of course it all depends on what are exactly are you doing but you can get the idea.
  • qlitheqlithe
    Yes I am running a dedicated project just for scraping. Problem is that it's forever growing bigger and bigger since it seems to scrape more than it tries to submit.

    Would be great to have a setting to not have to manually switch on/off scraping
  • TheGypsyTheGypsy Madrid
    Are you aware that SER has an extra tool just for scraping? It's in Settings > Advanced > Tools > Search online for URLs

  • qlitheqlithe
    Yes
  • TheGypsyTheGypsy Madrid
    Isn't that faster than setting up a scraping project? 

    So you have a project set up where you scrape only and you manually export the scraped URLs to a different project to post?

  • qlitheqlithe
    Well can't tell if it would be faster actually, but what I'm looking for is more of a set and forget solution for scraping
  • TheGypsyTheGypsy Madrid
    Then you have 3 options.
    1. Easy:
    You set up SER to find targets and post to them. It's originally designed that way and it still works. You need google passed proxies and it will does it's job.

    2. Hard
    You set up a scraping rig with scrapebox, platform identifier and dropbox

    3. Lazy
    You buy lists.
  • qlitheqlithe
    Yeah I'm currently going with 1) but the problem is, as described in this topic, that the target list just keeps on growing and growing and will eventually become a problem. Would be great to be able to control it.
  • TheGypsyTheGypsy Madrid
    You mean the target list when you right click on a project and select show target URLs?
  • qlitheqlithe
    Yeah it just keeps on growing million by million, so I have to change the options for scraping back and forth from time to time

    Another feature that would be sweet could be a status like "Search only" but "Submit only"
  • TheGypsyTheGypsy Madrid
    If you turn off all the search engines then SER is submit only.
Sign In or Register to comment.