too many "Unable to extract content for"

CRYPTOLATAMMILLER

Hello

How we can improve the "Unable to extract content for". Log attached


PD: let me know if this log can leak any relevant info about we using this tools.

https://pastebin.com/raw/mrhmV1nt

Sven

since many of the urls have an <article> tag, it should be easy for it to extract the content. I guess it skips it as the keyword is not found on that site. Just check for yourself as I don't know what keywords you use.

Keep in mind that if you use "mobile app development" as keyword, it has to be present like that, and not with stop words in between.

CRYPTOLATAMMILLER

@Sven I did a simple example using one single source

take a look at the report https://pastebin.com/QQkBH4RX

see this log https://pastebin.com/ebm6NEtm
 

Sven

well ok, seems like duckduckgo needs some fix.

Sven

I have added that as source for next update.

CRYPTOLATAMMILLER

Nice, thanks,

keeping in the same thread, using bing as content provider, there are some results that I would like to extract, from my perspective it could be related to some bot blocket like cloudflare, it's possible to visualize the raw response of the request?

log https://pastebin.com/8WTht5jx  (ignore the applied filter)

Let's talk about these as an example

they are very good source of data, what does means unable to extract content for?

1. [20:39:04] Result 06 - Unable to extract content for https://www.bizjournals.com/southflorida/news/manufacturing/
2. [20:39:04] Result 07 - Unable to extract content for https://seekingalpha.com/news/3583836-perficient-acquires-productora-de-software
3. [20:39:04] Result 08 - Unable to extract content for https://gorillalogic.com/blog/gorilla-logic-adds-custom-software-development-center-in-medellin-colombia

Sven

1. just sentences of headlines...I don't see any real article here

2. javascript protected

3. the only thing thats worth here...but maybe due to a missing keyword its not finding relevant content