Skip to content

Http Timeout

AlexRAlexR Cape Town
in GSA Website Contact
Hi

What does this setting do?

I assume it's how long it waits for a website to respond. But what kind of response.

Let's say in my list of 1M urls 100K are dead websites. 

If I set this to 120s. 
I assume it will check the response codes before waiting 120s. 

Wondering if this could be slowing down submissions.




Comments

  • SvenSven www.GSA-Online.de
    That timeout is used for crawling a website. If just a single bit is delivered from the website after a request is made, that timer is reset to 0 and again it's waiting 120sec (as in your sample).
  • AlexRAlexR Cape Town
    So in the case of a dead website what would happen? Would it wait the full 120s?
  • SvenSven www.GSA-Online.de
    yes
  • AlexRAlexR Cape Town
    That's why my projects might be so slow.

    Any chance it could check for dead websites and immediately remove these rather than wait on the timeout!

    I saw @loopline suggested 120s. But surely it makes sense to rather have this set to 5s?
  • wolfvanweenwolfvanween
    @AlexR, a lot of websites have slower response rates, specifically on first byte. This would disable a lot of potential candidates. 
    And think about it, the faster a website is on the net, the more likely its owners are putting  care into spam control as well, so the harder to automate. For GSA SER we love those old and slow boys, forgotten by admins but having accumulated a lot of authority over the years...
  • AlexRAlexR Cape Town
    @wolfvanween - what do you use CF for? Guest post?

  • wolfvanweenwolfvanween
    Oops, sorry, I just realized this was in "GSA Website Contact" -- that of course is an entirely different discussion. ;-)
  • SvenSven www.GSA-Online.de
    you can not say if a server is down unless you wait a certain time for a response. There is no other way.
  • AlexRAlexR Cape Town
    What timeout do you recommend then? is 15s too short?
  • SvenSven www.GSA-Online.de
    depends on your network, amount of threads.
  • AlexRAlexR Cape Town
    300 threads.
    1GBPS port.

  • SvenSven www.GSA-Online.de
    30 sec should be fine then I guess.
  • looplineloopline autoapprovemarketplace.com
    I use 120 because if a proxy is slow or a site is slow (and some just are slowwww) I still want to capture that submit. Im happy to slow things down a bit on posting for accuracy. 

    There isn't a wrong answer, the wrong answer is one that causes success rate to drop below what you want. 

    Its trial and error, I have a lot of experience with different programs and timeouts and I know what I want to achieve and sometimes its more of a shoot from the him combined with gut feel when I make my choices.  Its hard to explain. 

    But like Sven said, 30s is nice and if your happy with it go with it. 

    Really the only way to know is to take a decent sized list and split test it over and over with the same list and different timeouts.  But there is other things at play, if you submit too many times the same message or ip or domain the end site could limit it and then if you wait a day then the next day the site might be down or over loaded, well for that matter 5 mins from now it might be over loaded or not. 

    Its just there are a lot of factors at play, but you can use big data, like run 1 million submits, log the success, run it a week later with a different timeout and see if it impacts things.  Its hard to know for sure though because may be you get 0.0X% higher success with a longer timeout, but thats not worth it, you would rather have the speed.  Thats why there isn't a wrong answer except the one that produces results your unhappy with (assuming changing it makes results you are happier with). 


  • AlexRAlexR Cape Town
    Thanks! The issue I've got is that I want to filter new domains to find prospects. So a longer timeout is wasted on sites that are dead. 

    I'll try 30s and run at 120s when doing submits.
  • davidc1davidc1
    Hello,

    Is GC make a domain resolver before trying to connect ? if not, it could lose a lot of time waiting for the delay.
  • SvenSven www.GSA-Online.de
    of course this resolving of domain to ip is done before connection and so this timeout does not apply to that part.
    Thanked by 1davidc1
  • AlexRAlexR Cape Town
    I thought you said:
    So in the case of a dead website what would happen? Would it wait the full 120s?
    You said yes.


    Does that mean if the domain is dead it won't wait for it to load?
  • SvenSven www.GSA-Online.de
    well define the "dead domain".
    a) if the IP could not be resolved, it would not even try to connect (domain expired e.g.)
    b) if the server is just down, the ip usually still is getting resolved from the domain and connection is performed.

  • AlexRAlexR Cape Town
    a) Great. That's exactly what I was after. Didn't want it to wait 120s for a IP that couldn't resolve! 
Sign In or Register to comment.