Is it possible to Merge Multiple Site Lists?

kyloe18

Hey, Heres a question which i cant get my head around - I have multiple copies of GSA Running on different VPS' and each of them are scaping sites and saving the verified and successful sites to their site list folder locally. My question is: is it possible to combine site lists from several different instances of GSA so that i can copy a bigger and better verified/successful across each of them? This would mean that each will be scraping for different keywords etc and contributing to a big "master list" that i could share back acrosss my multiple installs allowing for a nice big site list for T2/3 when i need to set to submit to Global Site List only? Cheers

Sven

you can do that manually when using import/export site lists.

kyloe18

Thanks Sven, so when you import a site list, does it merge with the existing SL as i thought it replaced it?

gooner

@kyloe18 - It merges, i use that all the time

royalmice

Since you can specify the location of the site list on each instance, why not just point all to a master location, like a network drive, on a LAN just use share folder.

I work in a Home LAN enviroment, with a master GSA install (powerful overclocked PC , of which the app data  \  sitelist folder is shared (R/W) with my notebook,. I also run GSA CB, set to run as a webserver and listening on IP address of master PC where it is installed at, port 80

ON the notebook i have another GSA ser install, and i set all the global site-list to shared location on master PC
For capcha solving on the notebook i set it to decapcher, but then edit the host file so decapcher points to internal IP of the master PC - which mean GSA SER on main PC will solve the capchas from notebook as well as the main pc itself.

I suck at explaining, so if this don't make sense, it is not you  "> I will draw a diagram one day.

magix

@sven Does this work:

Move your "identified" folder to a dropbox folder and point both installations to the same "folder" - it synchonizes this folder automatically all the time.

If this works it's perhaps a sollution for keeping one big list on all vps installations...

JudderMan

@royalmice, that sounds awesome dude

"Hub and spokes"

Sven

@magix Never tried it but yes, I think there have been customers doing this in the past.

krushinem

this will cause you problems using dropbox; I can attest. looks good in the beginning eventually you will get conflict errors and multiple files will get created wish someone could tell how to get around that

ron

That's interesting @krushinem - I was thinking of tying multiple vps's into the same verified list.

I know that I can use a static file like I do with keywords where I use the same keyword token in multiple projects. But it's only reading/importing the keywords from a static file.

So you're saying using a common sitelist definitely won't work (well), right? Are you saying it works, but multiple files are created for the same engine? So in effect you aren't using the entire global sitelist? Does that mean they are renamed copies like Article_Beach(1), Article_Beach(2), etc. - is that correct? I'm just trying to get my arms around this too.

krushinem

if you don't mind this site_list-identified (Selective Sync Conflict) and that is just for the main folder ---sitelist_Article-vBulletin - Blog (HOME's conflicted copy 2013-12-18 (2))-----sitelist_Article-vBulletin - Blog (HOME's conflicted copy 2013-12-18 (4)) and so on for every individual site list----that there is an issue I combined 89 drupal site lists today because of this

ron

That sounds pretty bad. I think it defeats the genius behind the whole idea.

Thinking that maybe the best way is to import each vps's site lists once a week and de-dupe?

Satans_Apprentice

Remove

Satans_Apprentice

Sorry, did it with my iPad.

OK boys and girls. Here is how to make this work. Guaranteed. A centralized verified list won't work because of contention creating new .txt files. When both servers access a verified file at the same time, a duplicate .txt file with a number appended to it is created. There is another way to have multiple servers share verified URLs without a centralized verified list. Here is how to do it using a round robin technique: 

1. Create separate verified folders for each server on Dropbox.  Name them "Verified A" and "Verified B". Save your verified lists from each server to their respective folders. 

2. For every project on both servers, for {options=>how to target URLs} select "submitted" and "verified" as sources. (By the way, @sven, please pick a single naming convention. On global {options=>advanced} it's "successful", not "submitted") 

3. Go to global {Options=>advanced} for server A, and using the dropdown arrow for "successful" select the "Verified B" folder you created above in Dropbox. 

4. Make sure that you DO NOT CHECK "build site lists" for "successful". This will duplicate the contention problem.  

5. Using the dropdown arrow for "verified", make sure you have selected the "Verified A" Dropbox folder for Server A.  Check "verified"

6. Repeat the process for server B, but switch the folders. "Successful" for server B should point at folder "Verified A". Verified should point at "verified B". 

What will happen is that server A's verifieds will move to server B through "successful" queries, and vice versa. It's not as sexy as the centralized concept, but the verifieds will be shared without contention since only 1 server is writing to a folder/file. You could string together as many servers as you want with this technique. You just need 1 Dropbox verified folder for each server: A>B>C>D>A. @gooner, @2take2, @ron - check it out...

krushinem

nice

Satans_Apprentice

To add another concept, you could use "identified" and "failed" to create a mesh with multiple servers.

Tim89

#smart

mamadou

@Satans_Apprentice ... very nice and smart idea.

Sorry for the stupid question but How can I apply this for more than 2 servers ...let's say I wanna do it for 10 servers , How can I achieve that ?

Satans_Apprentice

@Mamadou: You can chain together any number of servers using Dropbox. Say you have 4 servers - A, B, C, D.

For server A, in Global Options point "Identified" at Server B's "Verified" folder. Make sure "Identified" is checked in all of your projects. For Server B, Point "Identified" at server C's verified list. In server C, point Identified at Server D's Verified list. For Server D, point identified at Server A's Verified list. The newly acquired links will move in a circle.

You could also create a "Mesh" topography by using "Failed" and "Submitted" and pointing them at other Server's verified lists. You could also go hub and spokes by pointing B, C, D's Identified folder at Server A's Verified List.

The possibilities are endless.

Just make sure that on any server, the only the "Verified" folder gets a checkmark in Global Options. If you check anything else, servers will be sending bad links to other servers verified folders. Big Mess! You only want to read from other server's verified folders. You never want to write to them.

mamadou

@Santans_Apprentice

Thanks for your help and the detailed answer. You are the man!