Ah I knew it! That means if I don't have an engine selected I'm missing out on possible targets for when I do use those engines. Better do create a fake campaign like you said with bing as the url and load all my random lists into 1 project and select every engine and see what I get then.
When I test a list, I check all engines to see what it produces. Sometimes people get all picky and then wonder why they aren't producing many links. Our lists cover all platforms. Sure I am not a fan of Indexer, Ping, Exploit, etc., but on lower junk tiers, I say what the hell, lol. It's not like you are getting graded on quality on junk tiers.
Just to add, I like having a generic project or two that have all platforms and engines checked because that way the entire list is getting processed into my verified file. Then I have them for when I need them
Well I basically crawled the web at every forum I'm a member of and got a bunch of random lists. Some a week old to some a couple of years old. I just created a do follow contextual project and let it do it's thing. I loaded them manually into the project and selected them all.
And then after you are done with all of that, remove your duplicate URLs and duplicate domains in your verified list, and you are good to go. Always remove dups from your verified daily if you are using it to feed your projects.
We have had a lot of issues using that, so I wouldn't use it. What happens is that a lot of websites have a bad hair day, a missing electron in the internet connection, a microsecond of line noise in the proxy - and wham - a perfectly good link is dropped because it couldn't get confirmed.
So you lose a lot of perfectly fine links. I would rather have a little garbage mixed in with all the good links - and settle for that inefficiency - as opposed to complete annihilation of some legitimate links.
ron Does "Automatically choose URLs from engines used in projects" really gives you wrong numbers?
I just tried it yet again on two sets of projects with different engines and it gives me exactly the same results as if I were to select engines manually
I just don't like how it differs in the message it delivers, which is different with each method. I like to know I am getting "x" per project, not some total.
Removed all URL Cache and History and tried again and the same happens
Reduced threads from 1250 (which SER purrs at all day long when pulling from sitelists) to 600 and no change
@Sven does SER handle the targets differently when you import a sitelist directly into projects compared with just using the built in sitelists? I assumed it would just write it to the target files and then pull from there as it needs new targets, is that not the case?
@davbel when you import a list, it could be stored within the memory, therefore increasing your memory usage which causes more 'out of memory' issues when you're importing directly, Which is quite a catch 22 situation because SER flies through direct imported sitelists and is significantly slower when it is pulling sources from the global lists, so what do you go for? Lower the amount of projects your running.. to ramp up the threads as high as you can, then import directly? OR, ramp your threads up to normal and simply use global lists.
I too suffer from this situation, I have around 250 projects which, if run pulling from verified lists can run all day long @ 1200 threads, However, If I were to import a sitelist of 40k urls to each project, I would need to lower my threads to 400 - 500 or I'll get those dreaded memory warnings and SER turns to shit.
The list I was using was about 80k and I've tried turning it right down to 200 threads and it still tops out at 100% CPU + memory and stupid bandwidth, but only about 1/2 the time.
I think I'll try a yet smaller list and see if I can get that working as you say.
Is there an easy way of exporting sitelists to a specific size?
@davbel How many projects are you importing that list to? If you have 10 projects for example, I believe what basically happens is you're importing 80k x 10, so in theory SER treats this all seperately and creates a sitelist profile for each project resulting in major CPU / MEM usage so an 80k list imported to 10 projects is a massive amount to import & store.
I tend to import llists at around 20 - 30k size limits, I split my sitelists up using scrapebox Export & Split feature, I just type how many urls I want in each file and hit save and scrapebox splits my lists up.
What would be much much easier, is if we could pull our sites from the verified lists in order instead of how it works now... "pulls sites at random from global sitelist" I don't understand why it is that way... If I set a project up and select my verified sitelist, that means I want SER to run through my verified sitelist from A to B and post to that list, but actually, it only pulls urls at random from that list, this may result in the same url/source being pulled from the list over and over and over again, which will then spit out "Already Parsed" etc.
It will still pull URLs randomly and not by sequence, it pulls everything randomly if you select any list within your global sitelist, that is why it's bad, That is why it is way faster to import direct to your projects, this will work through the entire list until it is depleted, the downside is I'm assuming it takes up more memory/cpu usage causing SER to actually run slower or crash out and lag.
either select from verified lists = high threads but hit or miss with posting direct import sitelist to project = much lower threads depending on amount of projects and run through more unique domains. (no double parsing, more effecient yet run at much lower threads.
@tim89 it was all (60ish) as per @ron, but now I'm thinking I've misunderstood him.
I always forget about SB and some of the extra stuff it can do...if I don't lose the will to live, I think I'll try what you do and see if I can get them into some sort of working system...
Strange thing is that this PM reverting back to my previous set-up, I've got the exact same LPM at 400 threads as I get at 1250 with much less download failed.
@davbel download failed ocasionally has to do with the HTML timeout too, don't forget to increase your timeout accordingly when increasing your threads.
What I've been doing recently is gathering up all my contextual verified lists and merging them into 1 file, importing that file into scrapebox and the de dup them, removing duplicate domains too, which should leave you with a huge arse contextual verified sitelist with no dups, importing a list like this to projects will make your projects fly so high.
@tim89 have it set at 160s which should be OK me thinks.
I think there may be something funky with how this install of SER has "evolved". No matter what I change, I just can't seem to improve the LPM etc to be more than I was getting before I became tempted by the evil Sith Lords @ron and @judderman and their promises of more links from the dark side...
In fact some of the things that work for them have the opposite effect for me.
I'm tempted to back up and re-install.
How did you "collate" your super contextual list? Is it just a case of opening each of the relevatn site list files in SB or is there a way to export only contextuals (or any other type) from within SER?
@davbel - You can edit a site list in a folder, and just delete the non-contextual platforms and engines. Then import it into your favorite folder. And then rinse and repeat with other lists, and then keep importing into the same folder. The contextuals get meshed together nicely. There may be a slicker way to do it, but that way was fast enough for me.
@davbel how many projects do you have? I mean per server. I have one server with 150 projects in various groups and it's my oldest server and that is the only one that has been struggling. I assume it's 'clogged' up but I can't see where or what yet. Maybe this is your problem?
My other servers, younger and max of 50 projects per server, are totally fine. Better than fine, actually, and every update they get better. I am, however, just about to switch everything up and wipe one server to turn it into a 'filter' server of my Gscrapering lists, which according to the other 'evil Sith Lords' you can run 150 projects simultaneously with no problem.
Not much help, sorry, I'd do a backup (double, triple, quadrulple) backup, reinstall, de-dupe your lists, check your overall settings, proxies and emails for any dead wood.
Hello guys i need to know how you can build alot of contextual backlinks, i have tried letting gsa scrape and post for urls and i have tried gscraper and scraped 1m unique url list. All over the past month and what i get is 500-1k contextual ( PR 1+ ) links per project .I know i PR drastically lowers possible links to post to. But i am asking how to get more links with those two filters ( contextual links and PR1+ )
@davbel have to disagree, ive been 'flying high' with the new versions with between 250-350(my personal rec ord! :P) but ive dropped them down now as I am not going for speed again. but it was great to see.
That was with me not even trying to push it to the max, still had verify and reverfiy set on.
Also Im not using lpm any more to monitor speed, jumped over to VPM (I encourage others to move to this more accurate performace figure), much more useful metric
@judderman I've added a few over the last couple of days so there are now 70ish projects, but this was also my first install of SER as well, which has been running for at least 18 months.
I had some success today in getting the speed back to somewhere near where it used to be, but I think the long term solution is going to involve doing some work with my lists and splitting them into types and then feeding them into projects as relevant, as well as a back up and complete re-install.
Comments
When I test a list, I check all engines to see what it produces. Sometimes people get all picky and then wonder why they aren't producing many links. Our lists cover all platforms. Sure I am not a fan of Indexer, Ping, Exploit, etc., but on lower junk tiers, I say what the hell, lol. It's not like you are getting graded on quality on junk tiers.
Just to add, I like having a generic project or two that have all platforms and engines checked because that way the entire list is getting processed into my verified file. Then I have them for when I need them
http://prntscr.com/3l4e7c
http://prntscr.com/3l4ew8
We have had a lot of issues using that, so I wouldn't use it. What happens is that a lot of websites have a bad hair day, a missing electron in the internet connection, a microsecond of line noise in the proxy - and wham - a perfectly good link is dropped because it couldn't get confirmed.
So you lose a lot of perfectly fine links. I would rather have a little garbage mixed in with all the good links - and settle for that inefficiency - as opposed to complete annihilation of some legitimate links.
Tried again using a smaller list and the same thing happens, CPU up at 100%, massive memory usage, massive 5 x network usage.
Tried doing it @Ron's way and same thing happens.
Removed all URL Cache and History and tried again and the same happens
Reduced threads from 1250 (which SER purrs at all day long when pulling from sitelists) to 600 and no change
@Sven does SER handle the targets differently when you import a sitelist directly into projects compared with just using the built in sitelists? I assumed it would just write it to the target files and then pull from there as it needs new targets, is that not the case?
I too suffer from this situation, I have around 250 projects which, if run pulling from verified lists can run all day long @ 1200 threads, However, If I were to import a sitelist of 40k urls to each project, I would need to lower my threads to 400 - 500 or I'll get those dreaded memory warnings and SER turns to shit.
The list I was using was about 80k and I've tried turning it right down to 200 threads and it still tops out at 100% CPU + memory and stupid bandwidth, but only about 1/2 the time.
I think I'll try a yet smaller list and see if I can get that working as you say.
Is there an easy way of exporting sitelists to a specific size?
I tend to import llists at around 20 - 30k size limits, I split my sitelists up using scrapebox Export & Split feature, I just type how many urls I want in each file and hit save and scrapebox splits my lists up.
What would be much much easier, is if we could pull our sites from the verified lists in order instead of how it works now... "pulls sites at random from global sitelist" I don't understand why it is that way... If I set a project up and select my verified sitelist, that means I want SER to run through my verified sitelist from A to B and post to that list, but actually, it only pulls urls at random from that list, this may result in the same url/source being pulled from the list over and over and over again, which will then spit out "Already Parsed" etc.
Oh how I wish life was easier.
either select from verified lists = high threads but hit or miss with posting
direct import sitelist to project = much lower threads depending on amount of projects and run through more unique domains. (no double parsing, more effecient yet run at much lower threads.
I always forget about SB and some of the extra stuff it can do...if I don't lose the will to live, I think I'll try what you do and see if I can get them into some sort of working system...
Strange thing is that this PM reverting back to my previous set-up, I've got the exact same LPM at 400 threads as I get at 1250 with much less download failed.
What I've been doing recently is gathering up all my contextual verified lists and merging them into 1 file, importing that file into scrapebox and the de dup them, removing duplicate domains too, which should leave you with a huge arse contextual verified sitelist with no dups, importing a list like this to projects will make your projects fly so high.
I think there may be something funky with how this install of SER has "evolved". No matter what I change, I just can't seem to improve the LPM etc to be more than I was getting before I became tempted by the evil Sith Lords @ron and @judderman and their promises of more links from the dark side...
In fact some of the things that work for them have the opposite effect for me.
I'm tempted to back up and re-install.
How did you "collate" your super contextual list? Is it just a case of opening each of the relevatn site list files in SB or is there a way to export only contextuals (or any other type) from within SER?
That was with me not even trying to push it to the max, still had verify and reverfiy set on.
Also Im not using lpm any more to monitor speed, jumped over to VPM (I encourage others to move to this more accurate performace figure), much more useful metric
I had some success today in getting the speed back to somewhere near where it used to be, but I think the long term solution is going to involve doing some work with my lists and splitting them into types and then feeding them into projects as relevant, as well as a back up and complete re-install.