@judderman - how do you manage to keep your proxies alive ? Is this possible due to removing advanced footprints, like inurl, powered by, node etc or smth. different ?
Whenever I choose to scrape with SER my proxies die within one hour. I have even tried to put waiting time to 160 seconds, when using 50 semidedicated proxies and 5 to 161 search engines, but it does not help.
Honestly, I don't know. I go off the fact that you run 100 threads per proxy (OK 700 threads with 40 proxies is taking it too far but it's worked so far). I have 50 dedicated proxies from SolidSEOVPS now, as Proxyhub were crap (even not pushing them as hard). I actually add hundreds more footprints and use Santos' footprint tool so it's not that.
@JudderMan Thanks for mentioning Santos footprint tool, never saw that one.
So are you saying that
A: you added more footprints to SER, so when it now scrapes it finds more targets (= only good for scraping with SER, not with other tools)
or
B: you modified the engines in a way that SER is now able to identify more platforms from the same amount of targets it scraped. So instead of scraping 100 sites and only being able to post to 10 of them (rest: "no engine matches") it now scrapes 100 sites and can post to 20 of them?
If it's "B" then...wow...I gotta get to work.
In case of a new SER update, though, will all engine files be overwritten again?
Both! I did b) first and the modified engines are pulling 1000x results compared to the standard SER engines. Drupal being one of the best. I still get a lot of no engine matches...
In case of a new SER update - when using Santos' tool you rename the file to say Article Beach Modified. This automatically shows up in SER and once you've gone through and clicked all your modified (notepad ++ will make this fast no doubt) then your settings are saved - or you back up all projects and save it as a zip file in case any updates wipe your previous settings.
With a) I'm still trying this out, but not sure how to split test it to see if it's having that much of an effect. It's 'easier' to see with Santos' tool as it gives you the amount of targets before you save the modification..I'm sure you can use that footprint file to good use now
Hmm...not sure about that, I thought that once you select engines then update it will have saved the previously selected engines. Like when you uncheck poor performing engines...I can't access my f**king dedi for some reason (possibly another DDOS attack) and haven't heard from my provider so I can't check that for you.
I don't think sharing is a good idea. It's why modifying engines/setups makes SER work even better. If people can't be bothered to put the effort in and want everything as default then that's no fair on the rest of us. I think it would be good to have a private forum where some stuff gets shared (ie. BHW JrVIP)...Sven could profit from that if it was a paid forum.
@johnmiller - What I meant was that after all the work I did improving the engines, a lot of engine 'scripts' changed. This basically left those engines worthless. Which meant I had to keep changing engines. There are 250 engines. After doing this for a very long time, you don't want to do it anymore. End of story.
JudderMan I agree, sharing wouldn't be a good idea. Like a modified xrumer vs. the vanilla version.
@ron So are you saying you did all that engine editing but after a while it got too time consuming and now you returned running with out-of-the-box SER again? Maybe it would make sense to just focus on the important engines, i.e. articles/contextuals.
You mod an engine, you rename it, and sven improves the scripting with 'original' engine. It happens all the time. The scripting is what makes it work correctly, and it has nothing to do with footprints.
So now you have a situation with a modded engine that may no longer work. Would you like to monitor every day whether sven found a problem on an engine, and then keep remodding his revised engine so it has your new footprints?
At a certain point you lose your business focus because you have now become a professional "SER babysitter". Been there, done that.
I assumed with each SER update the version history would tell if an engine was modified or not but looks like that isn't the case? Knowing which engines were changed would cut down heavily on the babysitting part.
150LPM on contextuals only. Changed up all of my projects today and it's been running very well since. I have noticed that when I update SER all my engines change to 'all' checked...very annoying.
The only other change is 50 dedicated proxies from Solid and no global list being used - all SER searching. Binned my proxyhubs as they were starting to be crap.
@JudderMan, What is your verfiied ratio? I get good LPM 100+ consistently but verified is low at 10-15%. I have semi-dedicated proxies from buyproxies.
Do you think dedicated proxies give better results?
I think good dedicated proxies make a difference, I had semis and dedis from proxyhub and the dedis were terrible, slow as hell and died all of the time. Solid proxies have been great so far.
My verified ratio is 20-30% at the moment, will give it a few more days to give a more accurate figure as they are contextuals so it takes a little time.
@ron So right on babysitter. I get fed up too every time even I think to attempt it. So I don't do it. I agree you lose your REAL focus doing and continuously tweaking stuff.
@JudderMan Curious if you post more than once on a site (there's an option in "options" tab) to have such an high LPM? Because I did so and I can see an improvement.
Nope, I stopped that as it was a bit too excitable to post to the same domain. If I change it I'll get 300+ LPM again. I'm happy with 140-150LPM with my current setup for mostly contextuals and letting SER scrape, post and not over and over to the same domain.
Comments
Whenever I choose to scrape with SER my proxies die within one hour. I have even tried to put waiting time to 160 seconds, when using 50 semidedicated proxies and 5 to 161 search engines, but it does not help.
So are you saying that
A: you added more footprints to SER, so when it now scrapes it finds more targets (= only good for scraping with SER, not with other tools)
or
B: you modified the engines in a way that SER is now able to identify more platforms from the same amount of targets it scraped. So instead of scraping 100 sites and only being able to post to 10 of them (rest: "no engine matches") it now scrapes 100 sites and can post to 20 of them?
If it's "B" then...wow...I gotta get to work.
In case of a new SER update, though, will all engine files be overwritten again?
Glad to hear a new update won't overwrite the files but what did @ron mean that with every SER engine update you have to do all again?
Maybe it would make sense to share the modified engines so SER can be improved? Or maybe only the hardworking should be rewarded...
@ron So are you saying you did all that engine editing but after a while it got too time consuming and now you returned running with out-of-the-box SER again? Maybe it would make sense to just focus on the important engines, i.e. articles/contextuals.
You mod an engine, you rename it, and sven improves the scripting with 'original' engine. It happens all the time. The scripting is what makes it work correctly, and it has nothing to do with footprints.
So now you have a situation with a modded engine that may no longer work. Would you like to monitor every day whether sven found a problem on an engine, and then keep remodding his revised engine so it has your new footprints?
At a certain point you lose your business focus because you have now become a professional "SER babysitter". Been there, done that.
I assumed with each SER update the version history would tell if an engine was modified or not but looks like that isn't the case? Knowing which engines were changed would cut down heavily on the babysitting part.