Feature Request : remove old non working domains from our global list
HI,
I use gsa ser for the past 2 years. This time you can imagine i gathered decent sized databases, the problem now is that they are too big for me. I need a way to delete [Download Falied] websites. I have 80%+ sites with this message because my database is old and it just saved all. Thing is that it takes me too much time to work with global lists, it should be faster but not in my case...
So if there could be an option that would check the platform of that target url, if it still is phpfox for example it will stay in the list, if it gets download falied message and its not due to proxy issue then it will get removed.
My identified list is 298+ mb and yes i DID removed the duplicate urls but duplicate domains aswell before checking the identified list size !
Let me know what you think @Sven and others.
I use gsa ser for the past 2 years. This time you can imagine i gathered decent sized databases, the problem now is that they are too big for me. I need a way to delete [Download Falied] websites. I have 80%+ sites with this message because my database is old and it just saved all. Thing is that it takes me too much time to work with global lists, it should be faster but not in my case...
So if there could be an option that would check the platform of that target url, if it still is phpfox for example it will stay in the list, if it gets download falied message and its not due to proxy issue then it will get removed.
My identified list is 298+ mb and yes i DID removed the duplicate urls but duplicate domains aswell before checking the identified list size !
Let me know what you think @Sven and others.
Comments
may be something like re-verification of URLs elsewhere in SER
a ping to all sites in global list by category (identified, submitted, verified)
a simple ping to each site
started manually as needed
while a built in SER version of live check surely would be fastest and best,
there is an instant working solution for SB owners I just tested (still running) after above post
use live check addon from SB
load one by one the URL lists (from identified, etc) one engine at a time,
run live check with HIGH connections (may be 2-3 times faster than for submissions or scrapes)
run,
then save / overwrite LIVE knks to original file in SER
one at a time
my list is only about 3 months old and most from recent 4-6 weeks SB
yet some 35-50% are dead
by tomorrow I may be finish and let you know the LpM improvements in efficiency
it's many operations because you have to do file by file
but it is an instant solution until same feature as one click verify in SER available
I would prefer to see the sites removed from SER if they fail X times. After X fails they're removed from Verified Site Lists.
last night I did most using SB ive check addon
depending on engine up to 80-90 % were dead
average about 35-40%
its a solution while waiting for sven to have time to implement live checks of global lists into SER
One question...I have hundreds of thousands of URLs for some platforms. If I tell it to recheck 3 times, will it do all three checks immediately?
My concern is that if a site is overloaded and doesn't respond now, it might respond in a few hours after the other checks are done. Is there a way to check once, then move it to the back of the line if it fails?
I don't see why it should work differently for me, as I haven't made any custom modifications to Ser, everything is set up as default
ie for verified links GSA will check them a few times, and if they persistently fail they will be removed. Could we not just check if a domain is failing each time we try and post to it and remove it after $x retries?