Skip to content

Sharing sitelist on LAN

royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
I have GSA SER installed on my main PC at home and then another copy on my Laptop at home, and soon i will add a 3rd and maybe fourth machine.
All the computers are connected on my LAN.

Would it be possible ( or advisable ) to make the site list folder on my main PC shared and then let all the other PC's on the LAN use the same sitelist ?

So i will have one central sitelist and all PC's read and write to the same list.

The reason i will has LAN share is because i had too many problems using Dropbox for this kind of thing as many time it create conflict copies.


Appreciate your thoughts.

Michael

Comments

  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    I was hoping someone might be able to help me with my question
  • i guess you need to test it by yourself as the most are using VPS and don't have a datacenter at home ;)
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    lol - its not a data centre just a laptop and a desktop pc ,and using some spare pc's including my daughter's when she is at work.

    I have tested it in a LAN environment  and seems to be working. The import site list also seems to work much better, as i can keep my main copy of GSA Ser running whilst importing site list from a idle computer.

    will continue to monitor it.
  • yeah, i was just kidding :)
    great that it seems to work out for you. sometimes you have to test things out by yourself if it isn't a common task.
  • @royalmice so it's one gsa running at a time, isn't it? I can't imagine if more than one instance access the same file at the same time.

    Please keep us updated if that works. :)
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    NO i have 2 or 3 instances of GSA ser running same time

  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    same like i have 3 machines connecting to 1 GSA CB install over the lan

  • Really, now I'm even more eager to know.

    Connecting to GSA CB is not a problem as CB is like a web server waiting and serving requests. With sharing global lists on LAN, file locking might be the problem.

    Ex: when I edit a Word file, others can only read. Otherwise it might be corrupted. New technology allows multiple edits and conflict resolution. I don't think this could handle the situation well. But I could be wrong.

  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    @audioguy

    Yes i understand what you are saying, but i foresee that being maybe a problem when you have a large number of people writing to a file, but for my set up where i have only a few machines sharing the site-list files it seems to be working just fine.

    Since i have started the shared site-list, i have the main pc running  64 projects (16 main projects of 4 tiers each), running at 100 - 150 threads, set to use both site list as well as targeted urls.

    I have about 6 projects running on the laptop, with about 50 threads. The laptop is also running a search for new urls and adding them to the sitelist, it has been running for about 18 hours searching identified around 50K urls that its adding to the site-list.

    Touch wood,  but thus far i see no errors or any problems and GSA SER is running just fine.

     
  • @royalmice So you have identified, submitted and verified lists shared?

    Added it to the stuff I'd try whenever I have time. It'll save a bunch of time shuffling and processing lists from multiple VPSes.


  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    Yes i share identified, submitted and verified, below is how it is set up on my laptop, which use shared folder on main pc.

    image
  • AlexRAlexR Cape Town
    Do you use dropbox for this at all?
  • AlexRAlexR Cape Town
    How would you do it across a VPS?
  • Dropbox, as @royalmice mentioned, caused problems. My experience has been the same.

    You can do it if your VPSes are from the same companies in the same network. Just use typical Windows file sharing, but consider security very seriously as you don't want anyone to be able to access, modify or delete your files.
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    Have not tried it on a vps buti guess you would just have to specify the file path like this  \\vps ip\user\appdata\\roaming\gas ser\......

    It would be just a normal ftp connection
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    I just thought of another way for VPS -- why not just add a mapped network drive then just browse to it from within GSA SER ==  Might be easier in case authentication is needed.

    I am just guessing here so try it and see..
  • Just map appdata as a drive, like z:\ or something. Then point SER to the path and files as if it's local. Ask Windows to automatically map the drive every time it starts. Works anywhere, VPS or not as long as network sharing is enabled.
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    UPDate
    ;

    I have noticed that when one of the pc's are writing to the sitelist file, then another cant write at the same time. How ever the other pc will keep trying until the file is available, we are talking a wait time here in seconds

    Below was the status message i saw briefly on one machine, but it did not show again, as the sitelist file it wanted to write to became available.

    I was importing a few thousand links at one time into the site list at that time :

    The status showed the below momentarily.

    11:54:37: [-] Unable to save URL to site list file (no rights to write?).
    11:54:37: [-] Unable to save URL to site list file (no rights to write?).
    11:54:37: [-] Unable to save URL to site list file (no rights to write?).
    11:54:37: [-] Unable to save URL to site list file (no rights to write?).
    11:54:37: [-] Unable to save URL to site list file (no rights to write?).
  • Are you sure it was retrying, not skipping the line?

    It's kind of expected anyway but perhaps good enough for sharing a few instances of SER.
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    If i have to consider the alternative - dropbox, then i think the Lan share is more reliable.
    I think the next time i import a new bunch of urls into the sitelist files, i should first stop all instances of GSA SER, as i think the import of say a few thousand URLs  takes a little time and during that time no other instance can write to the Verified file.

    For normal normal operations where it only write one url at a time ( i think ) it should not be much of a problem.

    Will continue to keep an eye on it.
  • hi,

    if i am not wrong what you are trying to do is access same file by muitple instances to write? This is not possible as file is locked by single instance. It's not in GSA but in any windows file system. To overcome this we use database which allows multiple I/O inputs. Or maybe i didn't understand what you are trying to say. It's just not possible to write to same file using different instances.
  • royalmiceroyalmice WEBSITE: ---> https://asiavirtualsolutions.com | SKYPE:---> asiavirtualsolutions
    dexterproxy multiple instances  = 3 max
    The file is only locked for the milli  sec or so that a instance write to it.

    This is just normal windows file sharing.

    If you maybe have any other suggestions how multiple instances can share the same sitelist then i would love to know.
  • For now I guess I'll just do it semi manually every 3 days or a week.

    Collect, merge and put the lists back. It can be done with a script.
  • MrXMrX Germany
    It would suggest you to use sitelists on every machine! Im not quite sure how often SER does perform file requests but it might be slowing it down if it has to pull data via LAN! And of course there are problems with read/write on a shared file.
    You could just use a backup script or tool to compare all the sitelists and update new entries to every one of them on a regular basis like every 24 h!
Sign In or Register to comment.