Thanks for the tip @seo4all! I got myself the FpF, and I got some footprints from a couple of article sites that were pretty low on verified for me. I'm now scraping, so let's see how it goes
I've done some testing, and for whatever reason it seems that the CPU is melting even when I check the option of using verified lists. This happens in both 8.27 and 8.0, which is just completely beyond my understanding.
As it turns out, the problem was between my screen and where I sit.
As I had played around with different versions, somewhere along the lines the proxy settings (threads and timeout) had changed into pretty unsustainable numbers, thus, low LpM.
It's still not quite where I want it to be - or where I think it should for running scraped lists - but at least I'm driving them again.
If you're here and your processor is melting, download one of the earlier versions here:
@Artsi so you think there really is something that make CPU go wild in newer versions? Did you experiment between them by repeating projects under same conditions except SER version? :-?
@seo4all If you scraping with unorthodox footprints wouldn't the majority of targets be ignored for unmodified engines?
I mean, the engines themselves contain some "conditions" like:
page must have1=Powered by XOOPS|The XOOPS Project|href="http://www.frxoops.org/"|<meta name="generator" content="XOOPS" />
submit success=Vous кtes maintenant enregistrй|Merci pour votre enregistrement|an email containing an user activation key has been sent to the email account you provided|please follow the instructions in the mail to activate your account |You are now registered.|Please wait for your account to be activated by the adminstrators.|You
using custom footprints may lead to some targets which do not have these or I am wrong?
How do you find "Default footprints scraping" VS "Custom footprints scraping" situation overall? Does it really much better for you without tweaking the engines themselves? :@)
Well, now that I'm running the 8.28, everything is in order.
I sure did. I have two VPS's, and I'm currently blasting one property with just about similar projects. One thing that changed for me were the proxy settings, so I'm sure that contributed to the problem as well.
Anyway... Everything is working smoothly now, which is most important
@seo4all, you can download two 7.xx versions if you scroll up and see those links I posted.
I must say though, there's something not quite working for me even with those versions... I wonder if my lists are full of the thing itself or what? Curious to see when this gets solved.
Hmm... Could this be something outside the SER? A Captcha Breaker issue perhaps? Something's going on here, and it aint' solving for me by changing the version. I've restarted everything to no effect.
no, as Im sure it has something to do with other things on your system. The CPU usage is as low as it can get. At least I don't see hwy it would be different as in any other version.
@Sven -i checked and double checked everything i even lowered the number of threads from 300 (which i was using with great success before this problem) to 100. Still the cpu goes wild.
As you can see i`m not the only one which is still having problems. Take a look at @freaky01 post below. He faces the same problem. And i`m sure that others are having this issue too. Chances for everyone to have their seetings set up in a bad way are very low.
As an example . Before having this issue i was running at 300 threads and with the list imported my cpu rarely went to 30%. It was below that. I would say arround 10% but sometimes was fluctuating. And that was when i was running more campaigns in the same time)
With the same seetings but with only 100 threads this time (instead of 300) and by only running one project it goes to 90% and up. So i`m quite sure this is not a problem with my seetings
@Artsi i`m constantly building and increasing my verified list. This means i`m importing new list of targets and run the tool to see how many verifieds i get. Then i repeat the process with another list.
It's just that I started suspecting this problem would have something to do with the list itself, so I tried running a verified list through a project, and that seems to be working all right.
So, the problem must not be in importing urls per se, but importing urls that SER needs to figure out where they belong or something like that
Artsi - thanks for version 8.0 - just installed and working without any problem, runing 1000 threads now and no crashes, with version 8.31 crashed with only 300 threads, also lagged all vps because of to high CPU usage..
what does the CPU load shows guys ? i`ll downgrade as well but i don`t know which version should work better
@Sven -as you can see from @jakubsas post below latest version is buggy while previous versions are way more stable. This confirm once again that the problem is within the software and not in one`s seetings
When I hit start, it was glued to 99% for some time - as it naturally happens in the beginning - but then it started to come down. Not that much, but I'm visiting the 60s and 80s here and there... I'll just let it run and see what it says.
But looking pretty good to me this far.
In 7.78 CPU usage seems a bit normal to me right now.
@Artsi I'm pretty sure 7.51 was before the update to the CPU monitor within SER and you can't use that. You need to open resource monitor to figure out how much CPU SER is actually using to compare it with current versions. Before the update to the CPU monitor it was very normal for it to say 99% all the time even though it wasn't true.
Right now I have 2 SER's running the lists I've scraped.
Here's the first one - 7.78:
As you can see, running those 4 projects turns the CPU into max. Here's the other one (on 7.51):
Here the exact opposite happens, and seems like SER is not doing nothing at all.
As a matter of fact, I've had it happen several times now, that I find the CPU to be at 0%, and all the projects are marked light blue, as in doing nothing at the moment. Stopping and restarting SER takes it active again.
I'm suspecting this might have something to do with the lists I've scraped, but how and what exactly? I've cleaned the lists to the best of my ability (remove dupes, dead, unindexed less than PR 1 urls), so what could it be about that one? Is it that SER doesn't recognize the engines or what?
@Sven, you have any ideas on what I should look about those lists, if that would help me get to the bottom of this?
How about the other guys? @fakenickahl@davbel@gooner@JudderMan ... Have you ever had a scraped list turn into a CPU-melter? It seems like that's the one shared variable here for me, since it seems to be happening regardless of the version. Maybe I should tick some setting ON / OFF here?
Here are my project settings:
I have the identifies folder enabled there to see if it would make any difference, but doesn't seem to have nay effect...
Oh and by the way!
On both of those SER's something kind of weird happens when I stop the projects...
It seems like as if there was like 10 000 threads going on, since SER will just be unwinding for like a minute or two. I'll just see this crazy flow of "donwload failed" and "no engine matches" and "no form at all" and similar red-text messages... How could there be so many processes going on in there? Is there some sort of cache that gets clogged or what?
Comments
I'm going to use it too.
I've done some testing, and for whatever reason it seems that the CPU is melting even when I check the option of using verified lists. This happens in both 8.27 and 8.0, which is just completely beyond my understanding.
As it turns out, the problem was between my screen and where I sit.
As I had played around with different versions, somewhere along the lines the proxy settings (threads and timeout) had changed into pretty unsustainable numbers, thus, low LpM.
It's still not quite where I want it to be - or where I think it should for running scraped lists - but at least I'm driving them again.
If you're here and your processor is melting, download one of the earlier versions here:
7.78
http://www.mediafire.com/download/f60sf1gpyzbast7/SER_7.78.exe
7.98
http://www.mediafire.com/download/53pl9gtbjlr771c/SER+7.98.exe
8.0
http://www38.zippyshare.com/v/55616772/file.html
Then put that .exe file into your SER folder and name it Search_Engine_Ranker (better to store your actual version somewhere safe.)
Well, now that I'm running the 8.28, everything is in order.
I sure did. I have two VPS's, and I'm currently blasting one property with just about similar projects. One thing that changed for me were the proxy settings, so I'm sure that contributed to the problem as well.
Anyway... Everything is working smoothly now, which is most important
While i`m running without any imported target list and let SER scrape my cpu stays at less than 10% (normally 3-5)
With an imported list of targets it goes to 70% and up.(while running only one project) This doesn`t make any sense . It should be the exact opposite.
I even lowered significantly the number of threads. Still nothing. What is happening ?
I missed the versions 7.xx of the software. I didn`t had any problem with them. All my problems started with the version 8 of this tool.
Please let me know how can i downgrade to v 7xx @Sven
Thanks in advance
I must say though, there's something not quite working for me even with those versions... I wonder if my lists are full of the thing itself or what? Curious to see when this gets solved.
I'm just running 7.51, and STILL melting the CPU.
Hmm... Could this be something outside the SER? A Captcha Breaker issue perhaps? Something's going on here, and it aint' solving for me by changing the version. I've restarted everything to no effect.
As you can see i`m not the only one which is still having problems. Take a look at @freaky01 post below. He faces the same problem. And i`m sure that others are having this issue too. Chances for everyone to have their seetings set up in a bad way are very low.
As an example . Before having this issue i was running at 300 threads and with the list imported my cpu rarely went to 30%. It was below that. I would say arround 10% but sometimes was fluctuating. And that was when i was running more campaigns in the same time)
With the same seetings but with only 100 threads this time (instead of 300) and by only running one project it goes to 90% and up. So i`m quite sure this is not a problem with my seetings
It's just that I started suspecting this problem would have something to do with the list itself, so I tried running a verified list through a project, and that seems to be working all right.
So, the problem must not be in importing urls per se, but importing urls that SER needs to figure out where they belong or something like that
I'm actually at 7.51 right now... Just imported 4 lists in there, and BOOM! LpM back to 75+
@Sven -as you can see from @jakubsas post below latest version is buggy while previous versions are way more stable. This confirm once again that the problem is within the software and not in one`s seetings
When I hit start, it was glued to 99% for some time - as it naturally happens in the beginning - but then it started to come down. Not that much, but I'm visiting the 60s and 80s here and there... I'll just let it run and see what it says.
But looking pretty good to me this far.
In 7.78 CPU usage seems a bit normal to me right now.
Right now I have 2 SER's running the lists I've scraped.
Here's the first one - 7.78:
As you can see, running those 4 projects turns the CPU into max. Here's the other one (on 7.51):
Here the exact opposite happens, and seems like SER is not doing nothing at all.
As a matter of fact, I've had it happen several times now, that I find the CPU to be at 0%, and all the projects are marked light blue, as in doing nothing at the moment. Stopping and restarting SER takes it active again.
I'm suspecting this might have something to do with the lists I've scraped, but how and what exactly? I've cleaned the lists to the best of my ability (remove dupes, dead, unindexed less than PR 1 urls), so what could it be about that one? Is it that SER doesn't recognize the engines or what?
How about the other guys? @fakenickahl @davbel @gooner @JudderMan ... Have you ever had a scraped list turn into a CPU-melter? It seems like that's the one shared variable here for me, since it seems to be happening regardless of the version. Maybe I should tick some setting ON / OFF here?
Here are my project settings:
I have the identifies folder enabled there to see if it would make any difference, but doesn't seem to have nay effect...
Oh and by the way!
On both of those SER's something kind of weird happens when I stop the projects...
It seems like as if there was like 10 000 threads going on, since SER will just be unwinding for like a minute or two. I'll just see this crazy flow of "donwload failed" and "no engine matches" and "no form at all" and similar red-text messages... How could there be so many processes going on in there? Is there some sort of cache that gets clogged or what?