Skip to content

Big scrape domain (by extension) reward

1356

Comments

  • KaineKaine thebestindexer.com
    edited March 2014
    Here is the correct configuration to use (adapt it with the necessary language of your list):


    image


    image
  • KaineKaine thebestindexer.com
    image
  • JudderManJudderMan UK
    Hi guys, Sorry I will have to pull out of this, I have too much work on at the moment and don't want to let you guys down. 

    .ac.uk is back up. 

    Sorry @Kaine
  • KaineKaine thebestindexer.com
    edited March 2014
    dariobl :) (PM sended)

     By cons you lose the right to recover the work of others.


    All seats are taken again. 
    Normally several lists will be finalized today.
  • KaineKaine thebestindexer.com
    @dariobl: Warn me when you received your list.
  • vifavifa
    Scrapebox is a bit undersized for these kinds of larger projects and crashing very often. Now I have bought GScraper, what a difference, fast as hell. :-)
  • KaineKaine thebestindexer.com
    Sure, Scrapebox is limited to something like 1000 000 line. This is a good investment to take Gscraper :)

  • fakenickahlfakenickahl
    I have also had a hard time getting much done with scrapebox due to crashing. I will buy gscraper tomorrow to get this scrape done.
  • KaineKaine thebestindexer.com
    Ok no worries 

    I just bought a field where you can to work as a team (with live chat and access dropbox).
  • ThundermanThunderman
    Cool Kaine, That sounds good.
  • JustinJustin
    Really need to be using Hrefer with a scrape like this...
  • KaineKaine thebestindexer.com
    edited March 2014
    @jjumpm2 have finished is list domain.CO.UK: About 2.2 Million domain trim to root/dedupe.

    I look forward to my second too.

    Time to finish the site with vort3x and you will get your access.
    The lists will be unlocked when the member has made ​​his work.


  • KaineKaine thebestindexer.com
    .EU is done to: 184170 unique domain.
  • comancsmcomancsm
    edited March 2014
    Hello

    I want to join too. Are there any seats available?

    I have gscraper + paid proxy service from Gscraper

  • KaineKaine thebestindexer.com
    edited March 2014
    Until we have the dedicated website that will allow us to speak freely of what we do (almost finished) I ask for a vote to see if we still take the world. 

    Since we have around the world, it'll be difficult to have all the answers quickly. 

    I ask 5 answers on adding new members. Whether or not and how much. 

    There are exactly 26 more to process lists. For those who finish their part, made ​​me know if I charge you to new ones. 

    If someone does not happen, do not want, or changed his mind, he makes his list immediately to give without delay at @comancsm.


  • comancsmcomancsm
    Thanks @kaine. Let me know as soon as a spot opens. I can run Gscraper @1500 threads with no issues.
  • KaineKaine thebestindexer.com

    Do not go too fast otherwise you scrape moin statement. Look at the tuning post previously (especially timout 60). 

    Ok I added you. I PM you for the section number 2.
  • KaineKaine thebestindexer.com
    Update on first big splitted list.


    image
  • KaineKaine thebestindexer.com
    edited March 2014
    Ok new update

    @kaine have Split1
    comancsm have Split 2
    @coneh34d have Split 3 now


    image

  • KaineKaine thebestindexer.com
    edited March 2014
    For example with the list 1 (Split-1): 

    At about 10% of scraping, I get 1.2 million unique .COM domain.

    Consumption of proxy is obviously proportional ;)


    Requires that some of the workers we do a check-up of the evolution of their list.

    For those who do not yet done so add me on Skype: kaine.project


    Update

    At this time our available Database has already reached about 3 445 986 unique domain.

    image

  • KaineKaine thebestindexer.com
    Update

    jjumpm2 take Split4 now.

    I see that the project is gaining momentum and that some member begins to stand out. I invite others to do the same, the result is worth it. I will try to ensure that we all deserve our part.

    image


  • cefegecefege
    @Kaine do you still need help scraping? I use GScraper with 100 proxies.
  • KaineKaine thebestindexer.com
    edited March 2014
    Thank @cefege, but i think 100 proxies is not enough :)

    We'll only recruit power user now.

    I think not delay going to make an announcement on all biggest blackhat website to recruit teams.


    @Hjunc have taken Split5 now.

    image
  • KaineKaine thebestindexer.com
    I just posted a request on http://bestblackhatforum.com

    Wait and see
  • cefegecefege
    @Kaine what specs do you want? I don't mind investing some in this :)
  • KaineKaine thebestindexer.com
    edited March 2014
    @cefege

    The lists that we spend, burn hugely proxies. 

    Must have service like (such as gsrapper) or access to a good source. 

    If you fixed it I will take with us. 

    I'm talking about taking Power User, because they have the resources to do the work quickly and well. This avoids me to ask at what level are the members. Every one is doing its part and receives are accessible on http://www.blackhatfusion.com


  • KaineKaine thebestindexer.com
    edited March 2014
    Member for participating in the projects contact me and gives me one IP and pass to acces at:


    image

    Our Domain_Database is now available in full acces for worker with job done.
    Other workers have restricted access. 

    Thank you.
  • KaineKaine thebestindexer.com
    edited March 2014
    No news of @antonyalston: responsible of domain.INFO

    antonyalton loses access to this collective work. 

    The list of domain.INFO is available now.


    cefege: Are you still ready to start?

  • cefegecefege
    @Kaine yes
  • rodolrodol
    hey kaine i can do some scraping
Sign In or Register to comment.