Sharing sitelist on LAN

royalmice

I have GSA SER installed on my main PC at home and then another copy on my Laptop at home, and soon i will add a 3rd and maybe fourth machine.
All the computers are connected on my LAN.

Would it be possible ( or advisable ) to make the site list folder on my main PC shared and then let all the other PC's on the LAN use the same sitelist ?

So i will have one central sitelist and all PC's read and write to the same list.

The reason i will has LAN share is because i had too many problems using Dropbox for this kind of thing as many time it create conflict copies.


Appreciate your thoughts.

Michael

royalmice

I was hoping someone might be able to help me with my question

Ozz

i guess you need to test it by yourself as the most are using VPS and don't have a datacenter at home

royalmice

lol - its not a data centre just a laptop and a desktop pc ,and using some spare pc's including my daughter's when she is at work.

I have tested it in a LAN environment  and seems to be working. The import site list also seems to work much better, as i can keep my main copy of GSA Ser running whilst importing site list from a idle computer.

will continue to monitor it.

Ozz

yeah, i was just kidding

great that it seems to work out for you. sometimes you have to test things out by yourself if it isn't a common task.

audioguy

@royalmice so it's one gsa running at a time, isn't it? I can't imagine if more than one instance access the same file at the same time.

Please keep us updated if that works.

royalmice

NO i have 2 or 3 instances of GSA ser running same time

royalmice

same like i have 3 machines connecting to 1 GSA CB install over the lan

audioguy

Really, now I'm even more eager to know.

Connecting to GSA CB is not a problem as CB is like a web server waiting and serving requests. With sharing global lists on LAN, file locking might be the problem.

Ex: when I edit a Word file, others can only read. Otherwise it might be corrupted. New technology allows multiple edits and conflict resolution. I don't think this could handle the situation well. But I could be wrong.

royalmice

@audioguy

Yes i understand what you are saying, but i foresee that being maybe a problem when you have a large number of people writing to a file, but for my set up where i have only a few machines sharing the site-list files it seems to be working just fine.

Since i have started the shared site-list, i have the main pc running  64 projects (16 main projects of 4 tiers each), running at 100 - 150 threads, set to use both site list as well as targeted urls.

I have about 6 projects running on the laptop, with about 50 threads. The laptop is also running a search for new urls and adding them to the sitelist, it has been running for about 18 hours searching identified around 50K urls that its adding to the site-list.

Touch wood,  but thus far i see no errors or any problems and GSA SER is running just fine.

 

audioguy

@royalmice So you have identified, submitted and verified lists shared?

Added it to the stuff I'd try whenever I have time. It'll save a bunch of time shuffling and processing lists from multiple VPSes.

royalmice

Yes i share identified, submitted and verified, below is how it is set up on my laptop, which use shared folder on main pc.

AlexR

Do you use dropbox for this at all?

AlexR

How would you do it across a VPS?

audioguy

Dropbox, as @royalmice mentioned, caused problems. My experience has been the same.

You can do it if your VPSes are from the same companies in the same network. Just use typical Windows file sharing, but consider security very seriously as you don't want anyone to be able to access, modify or delete your files.

royalmice

Have not tried it on a vps buti guess you would just have to specify the file path like this  \\vps ip\user\appdata\\roaming\gas ser\......

It would be just a normal ftp connection

royalmice

I just thought of another way for VPS -- why not just add a mapped network drive then just browse to it from within GSA SER ==  Might be easier in case authentication is needed.

I am just guessing here so try it and see..

audioguy

Just map appdata as a drive, like z:\ or something. Then point SER to the path and files as if it's local. Ask Windows to automatically map the drive every time it starts. Works anywhere, VPS or not as long as network sharing is enabled.

royalmice

UPDate
;

I have noticed that when one of the pc's are writing to the sitelist file, then another cant write at the same time. How ever the other pc will keep trying until the file is available, we are talking a wait time here in seconds

Below was the status message i saw briefly on one machine, but it did not show again, as the sitelist file it wanted to write to became available.

I was importing a few thousand links at one time into the site list at that time :

The status showed the below momentarily.

11:54:37: [-] Unable to save URL to site list file (no rights to write?).
11:54:37: [-] Unable to save URL to site list file (no rights to write?).
11:54:37: [-] Unable to save URL to site list file (no rights to write?).
11:54:37: [-] Unable to save URL to site list file (no rights to write?).
11:54:37: [-] Unable to save URL to site list file (no rights to write?).

audioguy

Are you sure it was retrying, not skipping the line?

It's kind of expected anyway but perhaps good enough for sharing a few instances of SER.

royalmice

If i have to consider the alternative - dropbox, then i think the Lan share is more reliable.
I think the next time i import a new bunch of urls into the sitelist files, i should first stop all instances of GSA SER, as i think the import of say a few thousand URLs  takes a little time and during that time no other instance can write to the Verified file.

For normal normal operations where it only write one url at a time ( i think ) it should not be much of a problem.

Will continue to keep an eye on it.

dexterproxy

hi,

if i am not wrong what you are trying to do is access same file by muitple instances to write? This is not possible as file is locked by single instance. It's not in GSA but in any windows file system. To overcome this we use database which allows multiple I/O inputs. Or maybe i didn't understand what you are trying to say. It's just not possible to write to same file using different instances.

royalmice

dexterproxy multiple instances  = 3 max
The file is only locked for the milli  sec or so that a instance write to it.

This is just normal windows file sharing.

If you maybe have any other suggestions how multiple instances can share the same sitelist then i would love to know.

audioguy

For now I guess I'll just do it semi manually every 3 days or a week.

Collect, merge and put the lists back. It can be done with a script.

MrX

It would suggest you to use sitelists on every machine! Im not quite sure how often SER does perform file requests but it might be slowing it down if it has to pull data via LAN! And of course there are problems with read/write on a shared file.
You could just use a backup script or tool to compare all the sitelists and update new entries to every one of them on a regular basis like every 24 h!