Skip to content

[Feature Request] Identification on GPU (CPU is very slow)

Lord_AlfredLord_Alfred world
edited January 2018 in GSA Platform Identifier
Hi.

Yesterday I scraped very small urls list and I try to identify this in GSA PI. It's my first time for using this program.
In my own home server (2 x Intel XEON X5660 2.8GHz = 24 cores with hyper threading and 96GB RAM) I "eat" all CPU usage on 50 Threads in GSA PI. And my small urls list (1.5 million unique domains) will be identified in 134 hours (or more)... It's very, very slow.

I think, maybe I use not optimal settings for my project (tell me if its true), but in 2018 year maybe it's time for move to GPU identification? I known, its hard process for programmers, but then identification increase in 10X or 100X times.

My project (and global) settings:





Comments

  • s4nt0ss4nt0s Houston, Texas
    @lordalfred - I see you have "limit bandwidth" unchecked on the main UI. You should have that checked and try lowering it a bit and it should help with CPU usage.
  • andrzejekandrzejek Polska
    https://www.researchgate.net/post/Can_web_crawling_be_done_in_parallel_using_Nvidia_GPU_in_CUDA_kernel
  • Lord_AlfredLord_Alfred world
    edited January 2018
    s4nt0s, hmm.. Why? My bandwith is not limited and "Down speed" is lower 1MB/sec. If I check on "limit bandwith" its lowering CPU usage, but it's also lowering my "URL/sec", right? Maybe there are another solutions or advices for increase speed?

    andrzejek, no, I tell about only move "identification" process onto GPU. It's use more response checks, regexps and some other very heavy tasks on CPU. I think, on GSA PI already use async on "identification" process, but using GPU in this (I guess) will be increase overall speed in more than 10X times.

    PS: http://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/competition/bkase.github.com/CUDA-grep/finalreport.html

    We successfully created a parallel regular expression matcher using CUDA for GPUs. Our implementation is anywhere from 2x-10x faster than grep depending on the workload and about 68x faster than the perl regex engine. We think that this makes it a viable candidate for use in the real world

  • Lord_AlfredLord_Alfred world
    I Increased threads to 200, CPU usage not changed (100% usage), but count of URLs/sec is increased. Thats is mistake or I really can increase threads to very bigger value without loss of checked urls?


  • s4nt0ss4nt0s Houston, Texas
    @Lord_Alfred- Give the bandwidth limit a try, you should see decreased CPU and you might not notice any slow down in URL process speed.
  • Lord_AlfredLord_Alfred world
    edited January 2018
    @s4nt0s , I limited bandwidth to 512 kb/s, my average "down speed" is 100-300 kb/s, but cpu usage lowed to 20-40%. Why? It's magic (or bug?). Processed urls speed will be decreasing or not?
  • s4nt0ss4nt0s Houston, Texas
    @Lord_Alfred - You can see the amount of URL's processing speed so you can judge it by that. The bandwidth limit helps with the CPU issue which is why I always tell people to adjust that to bring down CPU. It's not a bug, that's how it works. :P
  • Lord_AlfredLord_Alfred world
    @s4nt0s , hehe, thanks for giving this advice! And what you think about move identification process on GPU? It's possible and does this make sense?
  • Lord_AlfredLord_Alfred world
    After some hours I get this very strange stats:



    "URLS/min" upped to ~600!!!..

    I don't understand how this is possible with limited bandwith... Heh.. But I hope that all is secret feature )))
  • s4nt0ss4nt0s Houston, Texas
    @Lord_Alfred - Adding the GPU support would require a lot of work for very minimal gains so most likely that's not a road we're gonna take right now.

    What are the strange stats you're referring to?


  • Lord_AlfredLord_Alfred world
    edited January 2018
    @s4nt0s I mean "URLS/min" upped in 2-3 times and average "down speed" is less "limit bandwidth". I don't understand how it's works, but is strange for me - it is not  logical.

    About GPU support: your company tested it? I known some programs, where adding GPU gave best results.
  • s4nt0ss4nt0s Houston, Texas
    No GPU hasn't been tested because it would require a lot of extra work for what we believe will be minimal gains. Most people run Pi on a VPS where CPU and RAM are the main resources they go after when purchasing. I've had a lot of VPS and I couldn't tell you what GPU any of them ever had.

    For now we're letting the CPU handle it.
Sign In or Register to comment.