Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Download failed?

Hey hey hey guys!

Just recently I noticed that I'm getting A LOT of "download failed" errors.

I just tested my proxies, and every single one of them seem to be working great.

I just opened one of the urls in a browser, and it opened well so this must be a SER related issue.

Where should I start hunting for the cause here? I'm running a lot of threads, is that it? I mean like 1000+ at the moment.

I had the site size be set to 5M but I just raised that to 50M. Any other ideas?

Best Answers

  • cherubcherub UK
    Accepted Answer
    How many proxies are you using with your 1000+ threads?
  • cherubcherub UK
    Accepted Answer
    You may well be hammering your proxies. Downloading 1 page to test a proxy wont tax it, but running 1000+ threads with 30 proxies may be pushing it. Try lowering your threads and see if that has an effect.
  • Accepted Answer
    Try running 300 threads with 30 proxies.
  • Accepted Answer
    Well,  I had already created a thread on this that's pretty long and we discovered the issue.  It's with Lists,  If you scrape with SER and let it post you'll be fine but as soon as you start using a list is when you get the download failed.  

    https://forum.gsa-online.de/discussion/comment/79593#Comment_79593
  • Accepted Answer
    Yeah Sorry No fix for it yet. :(  Just something we gotta wait on @sven to come back for. :) 

Answers

  • I have 30 right now.

    However, I don't do any kind of scraping with SER so I figured a little less should be just fine.
  • @Hunar, well that might explain, since I have been running lists I've scraped myself. Thanks!
Sign In or Register to comment.