What is causing threads counts to get almost 0 waiting for a while then increases but not too much
No proxies selected
No search engine selected
1.2 GB new targets file i can see it
more than enough cpu and ram
I did set to 200 threads but do not pass even 50 threads and it decrases to like 2 3 threads then increases like 30 40 threads
any idea ?
ok it seems like working this way which is extremely incorrect
lets say pick 1000 urls to process
process process and the very last ones are the ones that will give timeout which is 90 seconds at me
so when url repository ends, it waits till all tasks completed to fetch new urls into repository which makes it wait and wait and wait
However this still also does not explain why i can not reach set thread count
windows server 2012 standart 64 bit
resource monitor shows only 60-65 threads while i did set 200 threads
@Sven
Also any way to save donwload failed urls to somewhere ?
No search engine selected
1.2 GB new targets file i can see it
more than enough cpu and ram
I did set to 200 threads but do not pass even 50 threads and it decrases to like 2 3 threads then increases like 30 40 threads
any idea ?
ok it seems like working this way which is extremely incorrect
lets say pick 1000 urls to process
process process and the very last ones are the ones that will give timeout which is 90 seconds at me
so when url repository ends, it waits till all tasks completed to fetch new urls into repository which makes it wait and wait and wait
However this still also does not explain why i can not reach set thread count
windows server 2012 standart 64 bit
resource monitor shows only 60-65 threads while i did set 200 threads
@Sven
Also any way to save donwload failed urls to somewhere ?
Comments
no ping service enabled
no search engine enabled
nothing single tier single project : fresh
no proxies enabled
no pause project enabled
single email address
very strong server a lot of free resources
after i deleted cache and only imported 500mb of urls thread count constantly about 60
DpM : 1745
however i want to utilize more threads to fetch more pages