Skip to content

Running GSA WC

jacobt3jacobt3 earth
edited May 12 in GSA Website Contact
Hello admin,

I was running GSA WC with proxy while sending and to 400000 urls which are not cleaned to if having really the form or not but i was having good results.

However today i saw the log only scrapes 5-10 threads whilst my thread count is 100
using a laptop 4 core 4 thread intel
LAN connection / Fiber line
While i test on browser i get  40 Mbps. 


Why GSA WC not using all the threads to scrape and send which was working yesterday as well. 


Today i tried to add multiple projects 5 may be and added 200000 urls for each project.


Then this situation started and it is very slow now. 


System has enough resources to be used. 3 GB RAM usable 90% CPU usable.


What am i doing wrong?


* Possible to assign num of threads each project?



Thanks

Comments

  • backlinkaddictbacklinkaddict
    However today i saw the log only scrapes 5-10 threads whilst my thread count is 100

    Because you put GSA software on high number of threads does not mean it will use them all every time. When this happens to me (If I think it's issue) I will stop, refresh projects, make sure I didnt misconfigure any setting by accident. Sometime "refreshing" the state helps too.

    using a laptop 4 core 4 thread intel
    You appear to be using an older/slower laptop and 40Mbps... on a fiber line? For me, if less then 100 MBps it means likely a wire broke or some hardware is throttling the connection.

    3 GB RAM usable 90% CPU usable.

    Is this what it says in GSA software or the task manager? Have a peek in your OS task manger and see if any background process like windows defender or antivirus is actively running and using/hogging resources as well. Set those things to run outside of active use time if you can.

    Maybe someone most currently using WC can help better? But this would be my first few thoughts if trying to troubleshoot...



  • jacobt3jacobt3 earth
    backlinkaddict said:
    However today i saw the log only scrapes 5-10 threads whilst my thread count is 100

    Because you put GSA software on high number of threads does not mean it will use them all every time. When this happens to me (If I think it's issue) I will stop, refresh projects, make sure I didnt misconfigure any setting by accident. Sometime "refreshing" the state helps too.

    using a laptop 4 core 4 thread intel
    You appear to be using an older/slower laptop and 40Mbps... on a fiber line? For me, if less then 100 MBps it means likely a wire broke or some hardware is throttling the connection.

    3 GB RAM usable 90% CPU usable.

    Is this what it says in GSA software or the task manager? Have a peek in your OS task manger and see if any background process like windows defender or antivirus is actively running and using/hogging resources as well. Set those things to run outside of active use time if you can.

    Maybe someone most currently using WC can help better? But this would be my first few thoughts if trying to troubleshoot...





    Hi,


    Thanks for the reply. Appreciate it.



    it was the proxies. I was hammering the proxies and it broke / stopped responding.
    So, added more proxies and increased threads. Now, it works fine but i needed to tweak things.



    Internet speed is fine, it 60 MBps line. GSA can utilize all of that but it all falls into checking/submission proxy. As the most check then send GETs are via proxy. The more proxy the better. 

    Or Else

    Ruin the own IP and get kicked  by the ISP. :)




    Can you help about checking the list for compatible forms? 
    I am using scrapebox to harvest the urls from Google /  Bing / Yahoo.




    Thanks a lot 
Sign In or Register to comment.