What is causing threads counts to get almost 0 waiting for a while then increases but not too much

edited March 2015 in Need Help
No proxies selected
No search engine selected
1.2 GB new targets file i can see it

more than enough cpu and ram

I did set to 200 threads but do not pass even 50 threads and it decrases to like 2 3 threads then increases like 30 40 threads
any idea ?


ok it seems like working this way which is extremely incorrect

lets say pick 1000 urls to process

process process and the very last ones are the ones that will give timeout which is 90 seconds at me

so when url repository ends, it waits till all tasks completed to fetch new urls into repository which makes it wait and wait and wait

However this still also does not explain why i can not reach set thread count

windows server 2012 standart 64 bit

resource monitor shows only 60-65 threads while i did set 200 threads

@Sven


Also any way to save donwload failed urls to somewhere ?

Comments

  • SvenSven www.GSA-Online.de
    Sorry I have no clue. Poste some logs please (pastebin.com).
  • edited March 2015
    @Sven ; sent you url for check

    no ping service enabled
    no search engine enabled
    nothing single tier single project : fresh
    no proxies enabled
    no pause project enabled

    single email address

    very strong server a lot of free resources
    after i deleted cache and only imported 500mb of urls thread count constantly about 60

    DpM : 1745


    however i want to utilize more threads to fetch more pages

Sign In or Register to comment.