Why SER isn't sending all the links it creates to IncredibleIndexer (API) ?

grax1

Hi guys, I've noticed weird thing, I don't know what's wrong but it looks like SER is not sending all the links it creates to indexer through API. Every project I run got an option to send links to "other indexers" and I got API for Incredibleindexer in global options. As you can see on the screen below, SER created 5k verified and sent only 87 links to indexer. Does anyone also experience something similar ? http://i.imgur.com/7JLtK1X.png

Olve1954

SER sends in batches of 100 links every x minutes to your indexer. I think is every 10 mins. To check how many links SER has sent, it's better to check with your indexer - how many links it has received. Did it receive 5k links? I use InstantLinkIndexer, and I can see the number of links it receives is about the same as the no. of verified links SER creates per day...

Sven

It sends it every 5 minutes or 100+ URLs

grax1

In this thread I wanted to let you know guys that my SER don't send all the links to the indexer - incredible indexer for the time I was writing it and I didn't notice it only in statistics but also in Serpbook. There was a time where I was manually pasting links to indexer, in parts. I switched to API and the number of movements in Serpbook decreased noticeably, even when I was running some aggressive campaigns.It looked like these links wasn't indexed, at least a big part of them, simply because they wasn't even sent to indexer.

Yesterday I bought second indexer - linkprocessor - the interface is so much better there comparing to what incredible indexer offers. When you check out this screen you will see the list with number of URLS linkprocessor received and the date when it happened. When I bought this 2nd indexer, I used it for 150k list of URLS I had to index from other tool and then I just pasted API into SER and it is the only source the indexer gets links from. My account allows 300k urls/day. For the moment I'm writing this post, there are already 7 pages on the list with these links indexer received. There is one time ~ 150k URLS like here and the rest are links like from the 1st screen where SER sends up to maybe 80 links for one time. 

Here's the problem:

1. http://i.imgur.com/5GObo3h.png
   
2. http://i.imgur.com/9jowrD9.png

299,541 Credits means that today indexer received only 459 URLS. You could tell me timezones is the problem, it's not. I bought access to linkprocessor 3.4.14, and up to now I built more than 10k+ (just don't know how many was yesterday, so just number for today) and indexer hasn't received that amount of links, not even 10%. 

In my settings I got 2 indexers selected, I got this option in every project I run.

@Sven, @s4nt0s I hope you guys can help me, I can give you access to my VPS and to linkprocessor as well so you can have a close look, PM me if you need it

boghi

I have the same problem and i solved it very easy by adding links manually into theincredibleindexer ,2 times a day .

grax1

@boghi maybe it's a solution for you, but this task should be automated, @Sven please do something with this issue, it's annoying when you pay 55$/monthly for indexing and don't even use 10% of your plan

Tim89

@grax1 none of my customers are suffering from this... could be a problem with the indexing services API or functionality.

grax1

@Tim89 thanks for information, maybe in the future I will give your service a try when the price will be more affordable for me @cloudattack and @linkprocessor can you tell something about this problem ? Have you noticed some problems with API or something ?

Tim89

@grax1 as you wish.

Sven said many times before that links are sent either in batches of 100 or every ten minutes I think.

I know incredible indexer have been suffering from some downtime as many of their customers have been complaining about recently, maybe that could be your problem?

Can't comment on link processor but they are a new service and could possibly need to iron out some bugs, who knows.

I would personally start with the services first.

Jesse

I used to have this same problem as @grax1 with Instantlinkindexer. So I moved to other indexing provider.

On a side note, Instantlinkindexer is too costly to be considered anyway.

linkprocessor

@grax1 I have posted in our thread that yesterday we had some technical issue on server. So, it might be that our server didn't receive all links yesterday. Now issue is fixed. Let me know how does it work in next few days.

grax1

@linkprocessor unfortunately nothing has changed, 6.4.14 I built ~ 6k verified so far, this are available credits for this day: http://i.imgur.com/jJegOYV.png ...

linkprocessor

@grax1 Please check my last post in our thread:
https://forum.gsa-online.de/discussion/10106/link-processor-indexing-crawling-link-pushing-the-best-gsa-indexer-9-95#latest

We have solution for this - to send links to us via RPC.

grax1

@linkprocessor - great! I just changed it the way you suggested and deleted the normal API as well. I haven't checked your thread so that's why I wasn't aware of this solution, would have changed this if I only knew before Hope this will do the job now

grax1

@linkprocessor and @sven I still hive some problems with indexers, today I built 25k verified so far, I logged to linkprocessor to check the usage of my plan, guess what, not even 5k links sent. I's better than it was, but still sucks. I decided to stop all the projects and now SER started to send all the pings (I'm using linkprocessor through ping and not API as @linkprocessor recommended).

So all I see in the log after stopping SER is just a long, long list, still active, with successfully pinged/failed to ping(for linkprocessor) and 99 Links sent to Indexer (incredible indexer) one after another. When I log into incredible indexer, there are 7847 links for today, similar problem, maybe @cloudattack can say something?

When I refreshed interface from both indexers, there are already a few thousands new links. But it SER started to send massive amounts of links once I stopped it! And most of the time I let it run for a few days, then stop to finally update and let it run again. I hope someone can finally solve all this issue with indexers as it's becoming frustrating !!

linkprocessor

@grax1
I have tested API again (4th time) and everything is fine.
I have stopped GSA campaigns and used option send verified links to indexers...
GSA has started with sending links like "100 links sent, 60 links sent, etc.." and all the numbers are the same like those made in campaigns on Link Processor side.
So, whatever amount of links is sent from GSA is received on LP side and added to your campaign (except duplicate links in one campaign).

grax1

@linkprocessor SER sends links, but as I said it first started to really send those links when I stopped it, that's weird, maybe @Sven knows the reason for it

grax1

Today I stopped SER to update to 8.16 and problem disappeared, SER didn't start to suddenly send tons of links once it stopped. I also checked both indexers and looks like the amount of links it should get was sent, so far so good