GOAL : 100 Projects Running on 5000 Threads, and LPM Over 1000+ ( Dedicated Server )

Pazalaza

Hi Everyone,

I plan to buy a dedicated server to achieve this goal: 100 Projects running on 5000 Threads, and LPM Over 1000+

Dedicated Server DXE5-26-2 : Dual Xeon E5-2620 Processors

• Dual Intel Xeon E5-2620, 2GHz,7.2GT/s, 15MB Cache, 12Core/24Threads
• 2TB SATAII, Enterprise Hard Drive, 1.2 Million-hours MTBF
• 16GB DDR3-1600 ECC Reg. RAM (upgradble to 256GB)
• 20TB Public Bandwidth

Platform: Windows Server 2012 R2 Standard Edition 64-bit

Upgrade Options:

> RAM Upgrade to 64GB DDR3-1600 ECC Reg.      
> Hard Drive 2 (Additional 2TB, SATA3, Enterprise)    
> Network1 (1000Mbps (1 Gigabit))    

+ Captcha Sniper + 500 Private Proxies + The Incredible Indexer

Do you think I will achieve my goal with this dedicated server characteristics ?

PS: I'm actually running 10 projects using GSA, on 500 threads using 50 Private Proxies, Captcha Sniper on ELITEBOX VPS from http://blackboxrdp.com/ and I'm at 80-110 Lpm.

ghetum

Processor is very good in my opinion . About the ram , maybe you should go with 24gb . My biggest advice to you is to buy an SSD dedicated server instead of a normal Hdd . I ran GSA on a SSD and you cant compare the speed with a normal HDD . No lag , faster processing .

I run gsa with captcha sniper too . In this moment I run it with 200 Threads , Captcha sniper on normal HDD , (Intel Core I5 3.40 GHZ ) , 8 GB ram and i have 60 LPM .

On a SSD I got with the same list, 100+ LPM . So its a big difference . 

gooner

I don't think it's possible. I have a better server than the one you are looking at and the best i can get is 400 LPM.
SER will "out of memory" at much less than 5000, you'll be lucky to get 1000 threads.

CPU/RAM really makes no difference in this case, the bottleneck will always be SER out of memory.

Brandon

I don't think it's possible. You're better off getting a few smaller servers to achieve your goals. Too many bottlenecks that one server can't overcome.

libero

If Your goal is 1000+ lpm, it`s not really necessary such a monster configuration. Really.
 With 500 threads,
50 private proxies (usualy 30-40 are working) we have
300-1,200 LpM (fluctuates a lot depending on what GSA is doing, if I import a fresh list its near the top)

HTML timeout: 140 seconds
time between queries: 3 seconds
Keywords: 500+K list

I just wonder: why captcha sniper? CB performs much more better and has a lot of available options for 3d party integrations with OCR solvers..

It`s all with 8GB RAM, dualcore cpu, and yes, SSD is better.

fakenickahl

I also must say that I doubt you will be able to run more than 1000 threads. My server has a rather good CPU, but ser is always only using 30-40%. Also, 16gb ram is already overkill so you shouldn't consider upgrading this. Remember that ser is only a 32bit application, so you'll never be able to use more than 4gb ram (?). I have once seen ser use 3gb ram at once, but usually it's just at 1-1½gb ram usage (besides between version 7.4x-7.50). The out of memory error will kill your dreams and hopes, unfortunately. However, an alternative could be to create virtual machines on your server and buy more licenses. Though I don't know if that'll be a cheaper solution compared to getting multiple small servers or vps's.

You should also be prepared to buy for a very large amount of proxies if you want to run a lot of threads, and I'm pretty sure that 20TB data per month will not be enough.

Please correct me if I'm wrong, but I recall hearing once that ser will not even be able to write and read fast enough to the harddrive that a SSD would make a difference if you've got a good SATA drive.

Anyways, good luck with your project. Hope you figure something out that'll work for you. Though you'll end up with a lot of unused power if you get such a server.

Edit: When I was running 2k threads (still with a good deal of unused power), I kept running into problems (which I described here https://forum.gsa-online.de/discussion/8101/frozen-threads), and that was before I would get out of memory errors at a little more than 1k threads. Now I can't even get near 2k threads before running into out of memory errors.

gooner

@fakenickahl is right, 20TB will probably not be enough at 1000 LPM

@libero i assume is importing verified lists to get those figures, if you plan on letting SER scrape or importing raw scrapes from SB/GS i would be amazed to see those numbers.

Personally, i import verified lists from another server and my LPM is 150 - 400... I have never seen 1000 LPM and i would say the chances of achieving that are close to 0.

Kaine

I think one cpu is good (In your place I would favor the frequency and the amount of cache that the number of core).

Take the minimum for 4 or 8 GB ram, SER uses only maximum 2 GB (32-bit). 

If you hesitate between a normal hard drive and an SSD, make Ramdisk that is 100,000 times faster at least  

I think 1GPS is more important ...

JudderMan

I have a better server and I've seen 350LPM once. Usually 150LPM but currently 40LPM...Maybe your amount of proxies will help but I can run 1500 threads but only for verification and can run 450 projects at once this way.

Kaine

Do not forget that CB has its limits too ... 

If too many captcha come together, it'll crash.

@JudderMan 350 LPM with succes list i think ?

I think a 64-bit version will use all our resources (personally I often dream)

1 good proxy (elite) for 1 thread is good ... for me

micha

@Pazalaza, check, you can give GSA SER max 2k threads only.

micha

@ Kaine, you are running GSA SER from ramdisk? How many LPM you have compared to HDD?
And how you do this? You running GSA SER from ramdisk, or you are keeping projects on ramdisk?
What ramdisk you are using?

JudderMan

@Kaine - no 350LPM was just blog comments no list. I'm running it now at 100LPM with 700 threads. Proxies are my downfall, I know that, I only have 40 now as Proxyhub's dedicated weren't good enough. 

A 64bit SER would be ideal.

Kaine

@micha Yes i running GSA i ramdisk. 

I can not give you stats because I use other software in parallel but the results are better (captcha breaker is also in ramdisk). 

The trouble is there to get all the files in the installation directory, it seems to me only going SER in place elsewhere ... it would nice if Sven could put all together. 

The software I use is Dataram ramdisk but any will do. the only thing he should do is save to extinction.

@JudderManI scrap elites and anonymous, I was always around 3000 now well, ever Up and very fast. I do not know how you made ​​​​with as less of proxy. (google traduct)

bingopro

You need more BW if you gona run it 24/7

I run 300 threads almost 24/7 and I have usede 14TB in  12 days

fakenickahl

@Kaine this really sounds very interesting! I have just had a quick look around dataram's website. My server has 16gb ram, so basically what I do is purchase the 12gb version, make a drive, and install ser on it? If I'm not mistaken then the problem with a ramdisk is that the data from ser in \AppData\Roaming\GSA Search Engine Ranker is placed there permanently and can't be moved right? I have heard before that when ser has to spin large spintax it takes a lot of resources, but having the data from \AppData\ not remove this issue for example? I'd also imagine having large url files would be easier handled on a ramdisk? And at last, since ser is a 32bit application, would it even be necessary to use the 12gb version of ramdata instead of the 4gb one?

Sorry for all these questions man. I read through some of the information and faqs on the website, but I'm still having a little hard time understanding this. Any help would be greatly appreciate, whether I'm totally wrong or spot on. I'm always looking for ways to further optimize my link building efforts.

Brandon

Nobody ever wants to realize that LPM varies wildly based on what types of platforms you run. You can get a very high LPM with blog comments. You'll get very low verifieds. Would you rather have a lot of links submitted or would you rather have a lot of links verified and viewable online?

Kaine

@fakenickahlThe ramdisk'll create a virtual disk that would be a real hard drive for windows  

Simply install and SER CB above. For software it makes no difference, that's why Dataram must save virtual disk on a real disc at pc extinction. 

I did not try to put the data in Appdata, i think SER not surely would arrive to find them, but it probably would improve performance. Maybe they have option for change path.

To install SER CB and it takes less than 1 GB ramdisk enough. If that's not enough (after), it is sufficient to expand the Ramdisk and put the data in  (copy data, delete Ramdisk, new ramdisk with same letter, paste data in)

@Brandon +1 

I speak only boost the overall average

fakenickahl

@Kaine thanks a lot for helping me out man! I'll surely give it a try, my server has 12gb ram which is never in use so I'd love to put some of it to good use

If the feature to change the location of the data in AppData would have a benefit to some of us, I'm sure that's a valid reason to suggest it as a feature to Sven. I only have a very limited programming knowledge, but it doesn't sound like the hardest feature to add. I know you can already change the location of your identified, submitted, verified and failed global lists, but I imagine it'd help a great deal if all the other data also could be located at the ramdisk.

Appreciate your time and help man.

Kaine

It's a plaisure

I think there are test version before buying

JudderMan

@Brandon Yep blog comments yield high LPM - it was for a one-off client who just wanted blog comments....might as well make the most out of SER if people just want certain engines

@Kaine interesting stuff, let us know how it all works for you with your setup.

Pazalaza

@ghetum @gooner @Brandon @libero @fakenickahl @Kaine @JudderMan @micha @bingopro

Thank you all for your advices

What I'm gonna to do is buying 10 GSA License, and I will run these 10 GSA (using 10 projects on 500 threads and 100 private proxies per GSA) on 10 VPS using CaptchaSniper to reach my goal. And I will add a list of 30K verified /GSA each month.

VPS Settings /1 GSA :

4 GB Dedicated RAM

(8x Core) Intel XEON Processor

240 GB Disk Space (RAID 10)

Unlimited Bandwidth @100Mbps

What customizations should I do on this vps to be the more efficient possible ?

Regards,

2Take2

I own both, but unless you've already got the Captcha Sniper licenses, I'd use GSA Captcha breaker as it is more stable and needs less baby sitting, whereas CS tends to crash all the time (which would be a right pain across 10 servers).

JudderMan

Same as above, I've used both extensively and CB (+EVE) is better than CS all day long even though my CS was fairly stable.

2Take2

Hi @JudderMan, that's interesting about CS, I found that mine was crashing after every couple of days continuous use - is yours running better than that?

JudderMan

I ran it from October to January and had no real hiccups. It wasn't as successful as CB but not bad. CB is far better, especially with EVE as secondary. I know that's the opposite of what's recommended but I'm not a fan of paying over and over again for captchas, DBC cost me thousands over the years.

2Take2

Ah, ok, it must have just been that my version of CS is a bit flakey. I just assumed that the program was like that so never bothered contacting their support about it. In hindsight perhaps I should have lol.

I know what you mean about DBC, but I still use it for small projects like web2 account creations etc. I probably wouldn't use it with SER though, well maybe for some small isolated projects perhaps, but not en mass. Unless of course, I was feeling a bit flush.

AlexR

What sort of LPM are you guys able to get with only contextual platforms?