Skip to content

Big scrape domain (by extension) reward

1235

Comments

  • KaineKaine thebestindexer.com
    edited March 2014
    Thank for review vifa, 
    I think a project like this we will soon be able to sort a good or bad good service. 
    In this regard, we start to have bad return on proxies service gscraper. Many people fail to scraper or very slowly. We recommend to avoid them. 
    If you have good addresses shared here. If the service is not good, if service is not good it will be said here too. 

    Thank you
  • KaineKaine thebestindexer.com
    @vifa have finished .GOV 

    They unlock Full acces to Database.

    Thanks
  • Sammy262Sammy262
    Kaine is there a reason for these crazy footprints in the list? I notice the scraping speed is super slow compared to what I get normally. Usually I see 34k URLs per minute with my settings and now I only get round about 250 URL per minute.
  • KaineKaine thebestindexer.com
    edited March 2014
    Yes it's really heavy to scrape, that why all want database ;)

    That why i want too only power user.
  • KaineKaine thebestindexer.com
    edited March 2014
    I receive message of member who wishes to buy our lists. 

    It is not for sale. 

    I banish without hesitation if this rule is not respected, do not take this risk !
  • jpvr90jpvr90
    edited March 2014
    @Kaine i agree i hate the idea of people selling list...it only hurts SER in the long run.

    anyways on topic to your post...my understanding is you are using footprints from SER and just adding "site:" operator to them? is this correct? please send more info i might be interested in this.
  • beamerbeamer
    if there is 1 extension left? i will start to scrape it right away. if i am not too late..
    PM me
  • ThundermanThunderman
    Hi Kaine 

    I added you to Skype, see you on there.
  • goonergooner SERLists.com
    .edu will be finished in approx 10 hours
  • ThundermanThunderman
    @Gooner  - Are you using Gscraper or Scrapebox?

    And how many threads are you running?

    I am running 600 threads through Gscraper and my CPU and Memory isn't high at all, but its moving along nicely at the moment. I don't know when it will be finished though, I have sent it to email me when when its done.


  • goonergooner SERLists.com
    edited March 2014
    @thunderman - I'm running Gscraper on 1500 threads.
    These scrapes are very slooooooow. Currently at 1500 URLs/minute.
  • ThundermanThunderman
    I have just up'd mine to 1500 threads and the CPU and Memory didn't really respond to much so I am leaving it at that rate.

    Warp Speed Ahead!!

    I normally average twenty three thousand urls per minute on a normal scrape i do for my personal projects, what is your normal average @Gooner?

  • goonergooner SERLists.com
    @thunderman - It depends a lot on the type of scrape, like an "inurl" scrape is really slow too.
    But yea, roughly the same as you i think.
  • KaineKaine thebestindexer.com
    edited March 2014
    @Thunderman Done.

    @gooner ok send me that at job done.

    PS: If it is long, is that your IP is burned. Try changing (vpn) i have 200ko deduped every second. + 3 800 000 at this time on .NET (not finished).


    Only the first list of work (TLD) is similar. There are currently 3 declination on different levels > more in progress.

    I would not say more. 

    @beamer @jpvr90I put you on the waiting list all the seats are Occupied, places will be expensive now.

    After discussion with the group average members want a small workgroup. To qualify a maximum of backlinks that we can collect.

    I will remove more member than add. Many not respond.
  • KaineKaine thebestindexer.com
    edited March 2014
    .NET job done, i add 4 566 182 at database.

    Full unique Domain_Database:

    image

    Obective are 10 000 000 for this week !

  • KaineKaine thebestindexer.com
    accelerator_dd can't finish the work (.CO)

  • beamerbeamer
    I will take the free spot if it is available.
  • KaineKaine thebestindexer.com
    Sorry @beamer at this time we do not add other worker, i say that her for people not on Skype.
  • KaineKaine thebestindexer.com
    @Sammy262 Split8 job done

    Full acces to database.
  • KaineKaine thebestindexer.com
    @Hjunc and me finish special job added on blackhatfusion.


    Full unique Domain_Database:

    image
  • KaineKaine thebestindexer.com
    @gooner .EDU job done too !
  • KaineKaine thebestindexer.com
    edited March 2014
    @cefege .INFO job done
    @jjumpm2 SPLIT11 done too

    Objective for week finished ! ;)

    Full unique Domain_Database:

    image



  • KaineKaine thebestindexer.com
    edited March 2014
    Too much TLD are not finished:

    .AC.UK
    .BE
    .BIZ
    .CA
    .CH
    .CO
    .COM
    .DE

    Members will be excluded

    I do not know exactly who have that list, but it will be treated by other workers. I ask you to come before they are redistributed.

    Special pack in run:

    Tier: 1 Done
    Tier: 2 Done
    Tier: 3 In run


  • jpvr90jpvr90
    edited March 2014
    @Kaine I am interested in doing .co ...but before I commit to it can I see file footprints you are using too make sure I can complete it.
  • rabbit23rabbit23
    Hey man, can Scrapebox do the work? PM me, I'm ready.
  • KaineKaine thebestindexer.com
    I thank you, we are trying to purify more than add. 

    He begins to have a nice collection now.

    .AC.UK is now taken by jjumpm2
    .US is now taken by Thunderman (original scrape) and coneh34d (special footprint).

    I have added personnel TLD to database, total is now:

    image

  • rabbit23rabbit23
    Too bad it's full now. By the way, are you going to sell it some day?
  • KaineKaine thebestindexer.com
    I do not think, means there is to make money with a collection like this.
  • KaineKaine thebestindexer.com
    edited March 2014
    @jjumpm2 finish .AC.UK

    He take now .BE .BIZ .CO

    Workers was above are now excluded.
  • candercander
    edited March 2014
    Hi, I'm on if is one hasn't be done. I've 4 gscraper with gscraper proxies on dedicade servers
Sign In or Register to comment.