Skip to content

Add Request Retry Count To Individual Projects

senginessengines us
in GSA Platform Identifier
If possible add the option of deciding how many retries in individual project/s settings. (Request retry count)

I currently have no way of setting this for rechecking unrecognized urls.

Comments

  • s4nt0ss4nt0s Houston, Texas
    Right now the request retry count is global in the settings.

    What some people will do is set a 2nd project to monitor the unrecognized URL's folder so its basically running through the identify and sort in process again and you can still have it output to the same site list as the first project so its basically rerunning it.

    I can see about adding it on a per project basis. 







  • senginessengines us
    @s4nt0s Thanks for listening to my feedback.

    Also would be nice to have recursive folder monitoring (Gscraper-Folder/*.txt) as gscraper creates folders by date when scraping. If you leave it running for a few days it will add more folders by date which don't end up being processed.

    I use the option Create new text if lines >
    which works great, but when left overnight the new folders are not scanned.







  • BigGulpsHuhWelpBigGulpsHuhWelp
    Is there a way to directly integrate gscraped urls into ser? right now i scrape then right click import target url via file into selected projects
  • s4nt0ss4nt0s Houston, Texas
    @sengines - We tried to do that in the past and we hit a snag with doing the multiple folder. It works with processing, but with monitor we couldn't do it. It had something to do with a windows limitation. 

    You might have to choose the gscraper option to save to a single file :/

    I will look into it again though

    ------

    @biggulpshuhwelp - Well with Pi you have your output directory set as your global site list in SER and it will feed them in as they're sorted but I'm not sure how you would do it unless you have another tool that sorts the URL's for you since the .txt files in the global site list need to be in the proper format to be picked up by SER. (named properly)

    There's probably another way you could do it, but you might need to write a script. Probably someone with basic script knowledge could do it for cheap. You can append the URL's to a certain file in SER, but I'm drawing a blank to which file that is :P
  • donaldbeckdonaldbeck Advanced SER Videos -> http://bit.ly/1ySrbwu | Learn SER For Free Step By Step -> http://sertips.com
    @s4nt0s

    I believe it's the .new_targets file.


    You would just write the gscraper URLs to the appropriate .new_targets file.
  • s4nt0ss4nt0s Houston, Texas
    @donaldbeck - Ah yes, that sounds right. Thanks
  • senginessengines us
    @s4nt0s a bug I found. On some monitoring projects when you press stop or pause it gets stuck on. "waiting for remaining requests (98%).
  • s4nt0ss4nt0s Houston, Texas
    @sengines - Hmm, is that only happening on some projects for you? Anything different about the projects thats happening on than the other ones? When you see that %, it is trying to exit threads. Maybe those projects are set to a higher thread count? 
  • senginessengines us
    The software handles well even with more than 6 million urls and multiple projects going at the same time. It's just that when I want to stop or pause these said projects this is when it freezes.

    I've tested this previously, (stoping or pausing projects monitor and process files) and it can stay there for a couple of days without doing anything. I did notice that in the requests tab there was one thread that was idle and that may have caused the freeze? Not sure.

    Thanks for such great software once again. This saves me a lot of time. I recommend everyone to buy this as it handles well under a lot of work.
  • s4nt0ss4nt0s Houston, Texas
    No problem. Thanks for the feature suggestions and feedback. :)
  • senginessengines us
    @s4nt0s my pleasure. My last request. :)

    How about on the next update add a minimize to tray that would be great. Thanks again.
  • magixmagix
    @sengines
    Perhaps that's the sollution for yor problem with GScraper (building new date folder over night):
    http://www.online-tech-tips.com/computer-tips/how-to-move-or-copy-files-at-scheduled-time-intervals/

    Its called dropit and you can set the little free to use software tool to periodically look inside a folder and copy all files ending with .txt to a desdination folder. You can add a ton of filters, placeholders ect. to the rules. You can set it to look inside sub-folders, too. And so, every newly created txt file inside your GScraper folder will be copied to your prefered folder which you can monitor by PI.

    I hope this helps and fits your needs...
  • senginessengines us
    @magix great tip. I do appreciate it. Never heard of it until you mentioned it.
  • magixmagix
    It works great - you can do a lot with this little helper...
Sign In or Register to comment.