Question for incrase scraping speed

Kaine

Hello Sven 

Are there is no way to increase the speed of scraping? 

I use other software but it is detrimental to the freshness of the link on which it publishes. 

The advantage of automatic scrap, is to arrive first on a new post (but scape must be nervous).

I'm sure this directory you can improve the X10 power easily. 

In this regard how trhead are allocated to this operation? say for over 100 requested. 

I think it deals with a small amount of data instead of storing data hugely 

Currently it looks like he tried to deal directly scrape it. Congestion in ram > dedupe > blast could surely boost everything.

This could be dependent on the memory that the user owns or up to what we used 32 bit. 

When do you think of that ?

The idea came with this software: IntelBurnTest.
This software is used to sort the good intel cpu bad for a same category.
And it allows them to more different declination for maximal frequency (frequency/volt).

The peculiarity of it is that it completely filled the ram before "blowing" literally cpu instruction. 

The benefit is optenir the maximum number possible operation (gigaflop) which is not possible otherwise. Optimization sequenciel way.

I think it can be used both for scratching, but also for the next treatment.

jpvr90

It looks like you are over-thinking the scraping process. No need to over complicate this process...scraping is not as hard as some make it out to be.

Kaine

I know this process is not complicated.

I'm just trying to not only make a request but also to participate.

jpvr90

dedicated machine + good footprints + lots and lots of proxies = more targets than you will ever need