Operation since last couple of updates

Stanger

I've noticed that currently it's running like this for me:

1. memory usage is WAY down
2. CPU usage is almost maxed and I'm running 1/2 the projects I was and took the threads way down
3. the number of public proxies is very low too

Wasn't sure where to post this so if this isn't correct please let me know.

s4nt0s

I'm not sure about the first two but for the 3rd option regarding public proxies - You know you can adjust the proxy sources where the software scrapes from right? If you click the add/edit proxysites button there are a ton of different proxy sources you can choose from instead of the default that everyone else is using. Might be something worth looking into if you're using public proxies.

Stanger

Thanks I didn't know that. Ill play with it

AlexR

I have had the same with the first 2. Very high CPU and low memory. (Memory sit's at 200mb and memory at around 85% on a 6.8ghz dual core). Probably from 5 updates ago. I've now lowered my thread count to 50. 

Stanger

@s4nt0s, I see where you're saying. How do I know which ones to chose or why wouldn't I chose all of them?

s4nt0s

That's more of a trial and error type thing. I use private proxies so I don't mess with it but it would be wise to test out other sources if you do use public proxies.

IF you choose all of them you're gonna be scraping a tonnnnnnnnnnnnnnn of proxies and it will probably take a long time. It's probably best just to select a few and go from there.

Ozz

@Stanger: Maybe I've some good tips for you, but you need to test them by yourself and keep and eye on the proxy counter in the status bar. I only use puplic proxies on my test machine, so there is no need for me to optimise this as best as possible.

This is my settings I use atm:

You first have to take a look at what proxy list give you good and fresh proxies every day. Some lists get updated more than once a day, some lists gets only updated once a week or even less. You need to get rid of all providers that  don't update their lists daily at least. To do that its best to visit the sites by hand and take a look into their proxy posts. To get the URL of the list click "Add/Edit ProxySites" and copy the URL.

If you got plenty of good lists (~10) activate them and download the proxies. Click "Add Proxy" -> "Find online". Keep an eye on the log and notice which provider gives you new proxies. Many proxies will be duplicates and won't give you any new proxies. Now just use the ProxySites which give you the most "new" proxies and get rid of the rest. The less ProxySites you use, the less threads you waste when searching for new proxies.

Test the proxies now (Test Proxies / don't skip unchecked). In my settings I use 50 threads with Timeout: 2 seconds. You need to figure out what works best for you.

If I reduce the timeout to 1 seconds for instance my router can't take the heat and I lose my connection. Same could happen if you increase the threads.

If you think that you don't get enough working proxies you could increase the timeout a litte bit (~5 seconds) but the more threads you use and the less timout you set, the faster will be your proxy test.

Automatically search for new proxies (could be everything from 30 to 720 minutes). Figure out what works best for you. 

You can also re-test flagged "not working" proxies again. Just mark the fastest proxies (up to 2 seconds) -> right mouseclick -> Test selected.

Let the proxy scraper run with your best settings for some days. Then you sort the list by speed and check what sources gave you best results in terms of working proxies, number of proxies and speed. If you see that some ProxySites give you bad results with no/few added proxies or only really crappy ones than you can get rid of them, too.

I for instance use just 3 proxy list providers, but I have my own ones

But I'm sure you'll find some good ones with the lists of providers that SER has installed. Most proxy lists are duplicating each other anyway.

Stanger

Thanks Ozz! When I get home ill crack into this. Appreciate the help.

mafcra

After latest updates i constantly getting trouble with links verifications for tier2 links (tier 1 looks working ok, I mean project using tier 1 links for creating of tier 2). I am using private proxies.

Example of this trouble : 610 submitted links, 1 verified link only. Extremely huge captcha spending because gsa creating lot of links but unable to verify it. 

It really strange problem, because if project not using tier links - everything working ok.

Can Sven take a look into it? I can send him that projects via email.

Stanger

That doesn't sound like a GSA problem. My tiers are building huge links with ease using public proxies.

mafcra

it happens only for tier 2 or tier 3 project (projects which using links of previous project). For tier 1 everything work fine. Really strange problem.