Skip to content

How can I create a central database of site lists from all my vpses?

OvidiusOvidius
in Feature Requests
I have 10+ vpses and I want to be able to save the site lists on a global folder that all my vpses can read/write it.

My hosting can provide a solution for this as long as gsa ser has the ability to pick it up.

@sven Can you do something about this?

Comments

  • rodolrodol
    you can use dropbox.... 
  • JustinJustin
    Dropbox
  • OvidiusOvidius
    @rodol and @justin do you use dropbox for this?

    My knowledge is that Dropbox can't be used for this beacause it creates a lot of temporary files, but maybe I'm wrong ...
  • rodolrodol
    temporary?, dropbox is like a cloud harddrive, just register for a dropbox and use it in all your vps servers, create the folders on your dropbox (verified, submitted, etc) and use that url on all your ser list
  • OvidiusOvidius
    I already have dropbox, I'll try that. Thank you

  • goonergooner SERLists.com
    You will get duplicate files in Dropbox if all VPS' are reading/writing to the file.
    Try it and you'll see.
  • rodolrodol
    why that could happen @gooner?
  • goonergooner SERLists.com
    I don't know, but it creates another file with "_copy" or something like that at the end.
    Myself, @ron, @2take2 and @satans_apprentice tried to use this method to build a joint site list.
    Unfortunately it didn't work.
  • OvidiusOvidius
    My hosting assured me that it can create a "local folder" that all my vpses can read/write.

    @sven Is gsa ser able to handle it? Can you code gsa ser to build a joint site lists? What specs should this folder have?
  • ronron SERLists.com
    edited March 2014
    @Ovidius - It creates a "conflicted copy" in your sitelist. So what happens is that when VPS#1 writes to the same folder, it will create the first conflicted copy. Example:

    Article_Beach.txt
    Article_Beach (conflicted copy).txt
    Article_Beach (conflicted copy 2).txt
    etc.

    You will then shoot yourself, lol.

    The problem is not with reading the file @Ovidius , it is because of writing to the same file.

    But you raise a great point because all those VPS's could 'read' some master verified file that you store in a different location - it just can't be the exact same folder that any of the VPS's write their verified to.

    It would require some manual weekly process where you intervene, take all 10 individual verified folders, merge them into a giant sitelist, de-dupe, then stick that in a place (identified, submitted or failed) where all the projects can read that file. 

    I hope this makes sense.
  • rodolrodol
    mmmm, looks like when ser write the list it use some kind of unique string per software or something like that only a coder will understand, maybe sven can code an option for this for people using 2 or 3 ser installations.
  • OvidiusOvidius
    Thank you @ron, yes it makes perfect sense and I'll implement this if I can't find a more solid solution. I'll load all projects from "failed" :)
  • OvidiusOvidius
    If multiple projects of one gsa ser install are able to write to the same file than I can't see why multiple gsa sers can't write to the same files.

    maybe @sven can create a global option "time to wait before writing to network site lists" and we will enter a different prime number other than 1 in every gsa ser we want to joint the site lists.

    I don't know how different projects of the same vps write to site lists right now, maybe somebody can explain!


  • HunarHunar
    Oh,  I would love something like this.  I have a Lot of installs running.   This would literally save me a LOT of time. :)
  • JustinJustin
    edited March 2014
    RE: It would require some manual weekly process where you intervene, take all 10 individual verified folders, merge them into a giant sitelist, de-dupe, then stick that in a place (identified, submitted or failed) where all the projects can read that file.

    This set of processes could be automated relatively easily.
  • rodolrodol
    edited March 2014
    SEr projects write and read from same address, you cant set an address to read and another to write, so that is impossible right now, it will require that you change the address for each project to write on your local machine, then after the merge process and creating a global list change all project address to the global one in dropbox or whatever for Read.
  • DzilRelDzilRel ____'s Backyard
    @Justing - Winautomation can do that pretty easy.
  • HunarHunar
    @DzilRel Is that something that you know how to do?
  • DzilRelDzilRel ____'s Backyard
    @Hunar Yes sir.
  • Satans_ApprenticeSatans_Apprentice SERLists.com
    edited March 2014
    We actually sent out a workaround in our first SER Tips on the SERlists.com website. Essentially if you have 2 servers set them to read from each other's verified list, but save to their own verified list. It creates 2 central repositories.You can do this with Dropbox.
  • RayBanRayBan
    @satans_apprentice - did not see this in first tutorial. Can you share in which para was it mentioned ?
  • goonergooner SERLists.com
    @rayban -

    Basically, you can set the projects to post from "Identified", "Submitted" or "Failed". In this example, let's go with "Identified".

    Then:
    Set server A identified folder as the verified folder of server B
    Set server B identified folder as the verified folder of server A

    VERY IMPORTANT: Where you set the identified folder, ensure the checkbox next it is unchecked or SER will overwrite that file. You only want it to read from identified, not write to it.

  • RayBanRayBan
    @gooner - cool, will give it a try.
Sign In or Register to comment.