Brumnick said "...Don't forget to ramp up the threads...". If I am using 30 semi-dedicated proxies, do yo think 300 threads is good while doing manual verification? :-?
@ron, would you really use proxies for verification and identification? Currently I'd never use proxies for these, but I'm curious to know your reason.
As for me, I don't use proxies for identification, but for verification - yes. Imagine verifying hundreds (or thousands) of urls from the same domain using your real IP, your IP may be flagged and reported as spam/abuse...
I see @ron, my reason for not using proxies for verification is that I don't want my proxies to slow me down even a fraction of an inch whenever possible. I appreciate the answer, you just had me curious as I thought everone were doing the same as me.
@davbel yeah dude I have all of my RDPs pinned to my taskbar and just highlight the projects and Active V. Again as mentioned above, you can run f**k loads of threads when you're just verifying and 100+ projects at the same time. Some of my dedis I've limited to 50 projects per server so they verify so quick then it's back to submissions after it's done (usually after I've been to the gym LOL).
@ron where is the link to where you say how to edit projects with notepad++ thought it was this thread but cant see it now must have been another one and forgot which. How do you do it en masse?
@PeterParker - It's in SERLists.com as a subscriber. I think you have joined and unsubscribed, etc. If you did, just join back in there, and a list of all the tutorials will be given you with links.
@ron I started taking your advice and I scraped the forums for AA list. I got about 20 Mil URLS and added them to my GSA Verified list only. The I removed duplicate URLS, Duplicate domains, and the other option clean the list. I've been uploading list for about 3 days straight now.
Found a 870K keyword list and use %spintax% in the keywords field to randomly choose a file.
Turned verification off
Turned off Web2.0
Turned off indexing
Checked try to have an anchor text
Changed filters to PR 0 and OBL 100
I kept the bad words filters though.
I got a charity hand out of 30 yahoo emails. Which was a huge help!
My question is they were all in .txt format. I added from file and chose "identify platform and sort in" will this add the URLS to my verified list? How does GSA know which list to add it to?
To verify I guess I just make the project "active V", but then where do I get the links to add to SEO indexer?
I ran a campaign and my LPM went from 4 to 10. The only other variable is the proxies. I need to buy more.
lol im kind of on the same position as you now, been trying things ppl have said and cant get anyhting steady above 40 lpm whereas i used to sit pretty at 200 ~:(
Looks like im back into the land of the mortals.
Have you filtered for only high performance platforms?
I've only had GSA about a month, so I haven't made enough post to actually do that yet. I plan on it once I run enough numbers to get some solid statistics.
@silverdot - I would just import the target urls into a project (like 30 projects that are all clones) with all platforms turned on, and definitely leave verification turned on for automatic 1440.
This scraping and processing you are doing is best done with dedi's, one for scraping and one for processing them through SER projects to create the verified file. You need about 10 emails per project.
It sounds like you picked up some old lists. Never import them into verified. Huge mistake. Most targets are probably dead. The only way to find the gems is to process them through projects as I described above. I would create those 30 cloned projects with fake URL's to post to. In other words, I wouldn't even waste my time actually trying to post them to my real projects. What comes out as verified from all that processing should be used for your real projects.
In other words, you are mixing a very inefficient process with your real projects. Processing massive lists to find a relatively few legitimate targets is a painful and slow process. It really should be separated from running your own real projects. Yes, it is expensive to do it the right way, but if you want to take that route, then a separate server with a second copy of SER + CB is the right way to do it.
You basically have 3 options: 1) Have SER scrape for targets and directly feed your real projects (which is what most people do, and how I did it for a year and a half - and did just fine); or 2) set up a separate scraping server to find targets and feed a separate processing server to run those targets through SER - and then take the product and use on your 'real projects' server; or 3) buy lists.
I feel that you are mixing approaches, and that is holding you back from speed and success.
I think with 10 proxies this is as fast as I'm going to get.
Here's what I did differently.
I loaded an AA list directly into the project. It was an older one, but it's all I got right now. About 500K from April I got from a friend.
I changed my proxy settings to have everything on private except search engines. I kept those on public with a 60sec timeout. With an AA list I don't think this setting matters because I'm giving it the target URLS.
I kept verified link must be exact URL on, but disabled verification. I also kept the indexer on because I don't want to have to copy all of those URLS and put them in the indexer myself. (I don't know how, show URL's > Verified > export?)
Q: I'm assuming that the links that do get posted and verified will be added to my verified list?
Q: On the Submission section what does the "Skip for identification" tick mark do?
Q: What does the green and yellow highlights mean on the right hand side? Submitted / Verified?
Once a week for re-verifications for me. Just to check. Hardly any die-off in my experience.
A few hours? I guess it's running through more attempts or parsing emails then? I just run it for 30-45 minutes and run all projects at the same time. Runs at 2GB RAM and when I see more than 90% error message "all links verified once" I switch to Active again.
Sorry @judderman I meant when it's posting links it slows down.
It'll run for a couple of hours hovering at it's setting of 1250 threads and post 10s of 1000s of submissions, but then it'll slow right down to 200-500 threads and the number of subs slow too, sometimes just a few hundred.
Still has lots of targets to post to and as soon as I stop and restart it speeds back up, but it just seems to get stuck after a few hours.
I'm guessing it doesn't happen to you or the others and its something unique to my set up
@davbel - I got into a routine now where I have verification disabled on all projects, and then switch to to setting Active(V) later in the day for a couple hours to clear it all out. And then flip it back to Active.
I also clear cache and reimport the targets once per day. That helps it to run fast and with plenty of targets.
@ron are you reimporting the targets into projects or into options -> Import Site Lists?
It appears as I'm not the only one with the slow down - there's another thread on the forums with users getting the same slow down issue whilst it's posting and then remedying it in the same way with a stop / start.
They've asked @sven for a feature for SER to restart automatically every 2 hours
@davbel - I am importing target URLs>sitelists. Under older versions of SER, we ran into issues where importing straight into projects was causing a RAM/CPU bottleneck. But since v8.38, importing directly has been awesome.
I still set projects to sitelist 'on' with identify (which is where I store my latest list). So that is a safety net, if you will, if targets run out. I typically purge cache and re-import the same list daily. And with fantastic results.
I'm running 150 projects simultaneously, no scheduler. When I import, I do them as a group. By group, I mean that all projects have exactly the same engines checked (otherwise the import gets screwed up and various engines will not import). So I do all contextual-only projects together as a group (all highlighted together), and all junk tier projects together, etc.
@ron that's something I was thinking about in reference to importing list. I imported the AA list directly to the project but only selected do follow contextual platforms. Most of the SER diagrams I've seen show the tiers being build with secondary no follow links.
Point being, should I import the same AA list to the secondary links as well.
or
Am I missing out on possible targets for platforms because I only select do follow contextual engines?
Normally I've been running about 60ish projects, no scheduler at 1250 threads and SER has been at about 30-50% CPU and 1.3-1.7Gb memory.
However when I import the site list into all the projects, SER runs out of memory within a few minutes and struggles to run more than 10-15 projects without 100% CPU.
This is how I had been doing it:
Selecting all projects
right clicking
Import Target Urls -> From Site List
Selecting the correct sitelist
Then clicking "Yes" from the Automatically choose URLs from engines blah blah pop up
But based on what you say @ron, what I should be doing is:
Selecting alike projects i.e. only contextuals or only junk
right click
Import Target Urls -> From Site List
Selecting the correct sitelist
Then clicking "No" from the Automatically choose blah blah popup
Select only the relevant engine types from the Choose Files window
@davbel - That is exactly how I do it. Especially the "No" part on #5. I don't like what it does at all if I were to click "Yes". I'm glad you wrote that out. I hope people are paying attention.
@silverdot - Follow the exact 6 steps as @davbel mentioned. Yes, you are using the same sitelist to supply targets to all projects at all levels of your tiers.
Comments
Found a 870K keyword list and use %spintax% in the keywords field to randomly choose a file.
Turned verification off
Turned off Web2.0
Turned off indexing
Checked try to have an anchor text
Changed filters to PR 0 and OBL 100
I kept the bad words filters though.
I got a charity hand out of 30 yahoo emails. Which was a huge help!
My question is they were all in .txt format. I added from file and chose "identify platform and sort in" will this add the URLS to my verified list? How does GSA know which list to add it to?
To verify I guess I just make the project "active V", but then where do I get the links to add to SEO indexer?
I ran a campaign and my LPM went from 4 to 10. The only other variable is the proxies. I need to buy more.
What else am I missing?
Looks like im back into the land of the mortals.
Have you filtered for only high performance platforms?
@silverdot - I would just import the target urls into a project (like 30 projects that are all clones) with all platforms turned on, and definitely leave verification turned on for automatic 1440.
This scraping and processing you are doing is best done with dedi's, one for scraping and one for processing them through SER projects to create the verified file. You need about 10 emails per project.
It sounds like you picked up some old lists. Never import them into verified. Huge mistake. Most targets are probably dead. The only way to find the gems is to process them through projects as I described above. I would create those 30 cloned projects with fake URL's to post to. In other words, I wouldn't even waste my time actually trying to post them to my real projects. What comes out as verified from all that processing should be used for your real projects.
In other words, you are mixing a very inefficient process with your real projects. Processing massive lists to find a relatively few legitimate targets is a painful and slow process. It really should be separated from running your own real projects. Yes, it is expensive to do it the right way, but if you want to take that route, then a separate server with a second copy of SER + CB is the right way to do it.
You basically have 3 options: 1) Have SER scrape for targets and directly feed your real projects (which is what most people do, and how I did it for a year and a half - and did just fine); or 2) set up a separate scraping server to find targets and feed a separate processing server to run those targets through SER - and then take the product and use on your 'real projects' server; or 3) buy lists.
I feel that you are mixing approaches, and that is holding you back from speed and success.
http://prntscr.com/3jdi36
I changed my proxy settings to have everything on private except search engines. I kept those on public with a 60sec timeout. With an AA list I don't think this setting matters because I'm giving it the target URLS.
Also I've noticed that when running SER like this after a few hours it tends to slow down for no apparent reason. Do either of you get that?
I've just been stopping and restarting which seems to fix it.
It'll run for a couple of hours hovering at it's setting of 1250 threads and post 10s of 1000s of submissions, but then it'll slow right down to 200-500 threads and the number of subs slow too, sometimes just a few hundred.
Still has lots of targets to post to and as soon as I stop and restart it speeds back up, but it just seems to get stuck after a few hours.
I'm guessing it doesn't happen to you or the others and its something unique to my set up
@davbel - I got into a routine now where I have verification disabled on all projects, and then switch to to setting Active(V) later in the day for a couple hours to clear it all out. And then flip it back to Active.
I also clear cache and reimport the targets once per day. That helps it to run fast and with plenty of targets.
It appears as I'm not the only one with the slow down - there's another thread on the forums with users getting the same slow down issue whilst it's posting and then remedying it in the same way with a stop / start.
They've asked @sven for a feature for SER to restart automatically every 2 hours
@ron how many projects are you running when submitting? Are you using the scheduler?
And you are importing the list into every project aren't you?
Normally I've been running about 60ish projects, no scheduler at 1250 threads and SER has been at about 30-50% CPU and 1.3-1.7Gb memory.
However when I import the site list into all the projects, SER runs out of memory within a few minutes and struggles to run more than 10-15 projects without 100% CPU.
This is how I had been doing it:
- Selecting all projects
- right clicking
- Import Target Urls -> From Site List
- Selecting the correct sitelist
- Then clicking "Yes" from the Automatically choose URLs from engines blah blah pop up
But based on what you say @ron, what I should be doing is:Correct?