Skip to content

Where is saved physically imported sites list to a project ?

yavuzyavuz
edited March 2015 in Need Help
Where is saved physically imported sites list to a project ?

Does GSA save them with zipping algorithm ?

Currently i am trying to import over 4gb text file and application became unresponsive :D

Comments

  • SvenSven www.GSA-Online.de
    The size should not matter. It is however not saved packed as SER has to read that file in 1MB matches and packing would then require it to save in 1MB packets. I think thats just a waste of time and resources as it also would become slower. However do you get any bugreport window when importing and it freezes?
  • yavuzyavuz
    @Sven i just checked after 8 hours. No report displayed and i see that it is not saved. Application uses 2155 mb ram memory atm.
  • SvenSven www.GSA-Online.de
    you just imported this to one project?
  • yavuzyavuz
    @Sven yes. I just checked and the file is created about 320mb atm so in 9 hours it imported only 320mb of 4.5gb :) i am going to write a c# file to remove duplicate, randomize and split into 500mb GSA ser seems like extremely slow
  • yavuzyavuz
    @Sven ok i removed duplicates myself, randomized and split as 500mb files

    It imports really fast that way

    However i believe if you choose randomize very big file gsa fails :D
  • SvenSven www.GSA-Online.de
    might be indeed a bottleneck. Can you provide that huge file for testing?
  • yavuzyavuz
    @Sven i think you can generate random urls file about 5.4gb in size and try yourself :) mine has about 95.000.000 unique urls.

    Also randomizing this file took quite a while at my monster computer (8 cores, 4.5GHZ, 32gb ram, 2x ssd raid 0)

    My ram got fulled probably also used pagefile
Sign In or Register to comment.