too many "Unable to extract content for"

April 2021

Hello

How we can improve the "Unable to extract content for". Log attached

PD: let me know if this log can leak any relevant info about we using this tools.

https://pastebin.com/raw/mrhmV1nt

April 2021

since many of the urls have an <article> tag, it should be easy for it to extract the content. I guess it skips it as the keyword is not found on that site. Just check for yourself as I don't know what keywords you use.

Keep in mind that if you use "mobile app development" as keyword, it has to be present like that, and not with stop words in between.

May 2021

@Sven I did a simple example using one single source

take a look at the report https://pastebin.com/QQkBH4RX

see this log https://pastebin.com/ebm6NEtm

May 2021

well ok, seems like duckduckgo needs some fix.

May 2021

I have added that as source for next update.

May 2021

Nice, thanks,

keeping in the same thread, using bing as content provider, there are some results that I would like to extract, from my perspective it could be related to some bot blocket like cloudflare, it's possible to visualize the raw response of the request?

log https://pastebin.com/8WTht5jx (ignore the applied filter)

Let's talk about these as an example

they are very good source of data, what does means unable to extract content for?

[20:39:04] Result 06 - Unable to extract content for https://www.bizjournals.com/southflorida/news/manufacturing/
[20:39:04] Result 07 - Unable to extract content for https://seekingalpha.com/news/3583836-perficient-acquires-productora-de-software
[20:39:04] Result 08 - Unable to extract content for https://gorillalogic.com/blog/gorilla-logic-adds-custom-software-development-center-in-medellin-colombia

May 2021

1. just sentences of headlines...I don't see any real article here

2. javascript protected

3. the only thing thats worth here...but maybe due to a missing keyword its not finding relevant content

too many "Unable to extract content for"

Comments