Skip to content

Download failed?

ArtsiArtsi
in Need Help
Hey hey hey guys!

Just recently I noticed that I'm getting A LOT of "download failed" errors.

I just tested my proxies, and every single one of them seem to be working great.

I just opened one of the urls in a browser, and it opened well so this must be a SER related issue.

Where should I start hunting for the cause here? I'm running a lot of threads, is that it? I mean like 1000+ at the moment.

I had the site size be set to 5M but I just raised that to 50M. Any other ideas?

Comments

  • cherubcherub SERnuke.com
    How many proxies are you using with your 1000+ threads?
  • ArtsiArtsi
    I have 30 right now.

    However, I don't do any kind of scraping with SER so I figured a little less should be just fine.
  • cherubcherub SERnuke.com
    You may well be hammering your proxies. Downloading 1 page to test a proxy wont tax it, but running 1000+ threads with 30 proxies may be pushing it. Try lowering your threads and see if that has an effect.
  • itsmeitsme
    Try running 300 threads with 30 proxies.
  • HunarHunar
    Well,  I had already created a thread on this that's pretty long and we discovered the issue.  It's with Lists,  If you scrape with SER and let it post you'll be fine but as soon as you start using a list is when you get the download failed.  

    https://forum.gsa-online.de/discussion/comment/79593#Comment_79593
  • ArtsiArtsi
    @Hunar, well that might explain, since I have been running lists I've scraped myself. Thanks!
  • HunarHunar
    Yeah Sorry No fix for it yet. :(  Just something we gotta wait on @sven to come back for. :) 
Sign In or Register to comment.