Skip to content

[Tutorial] Web 2.0 - How to code a Web 2.0 engine (Noob Friendly)

OzzOzz
edited September 2012 in New Engines/Platforms
Ok, that "Noob Friendly" is propably a lie, but I have to attract your attention somehow :D

Nonetheless is scripting a Web 2.0 engine "kind of easy" compared to other platforms, because of its static nature. You have to define all fields just one time, they will (hopefully) never change as there is only one URL to target. The only disadvantage they have is that they are more difficult to debug.

The main advantage to other platforms: There are a lot of them!


Before we dig harder into that coding stuff I want to recommend you to use/download/install these tools:

- Firefox + Firebug Addon: https://getfirebug.com/ 

If thats too "hardcore" for you already, you can stop reading right now. Anyone else, fasten your seatbelts ;)

image
Tagged:
«1

Comments

  • OzzOzz
    edited September 2012
    The site I like to script an engine for is called "alivenotdead.com". This is a good example as it's not using too much javashit or something else that makes coding more difficult for us. If you start to script a Web 2.0 try to find an "easy" target and register to the sites with help of a thrash mail provider like 10-Minute-Mail or Wegwerfemail.de (highly recommended but in german language).
    Once you've registered an account you should have an idea how difficult it might be to code and what kind of backlinks are possible. Most common types of backlinks are profiles and contextual blogs/articles for Web 2.0's. In this tutorial we are going to create both links, but I've only managed to post a profile link so far. We'are gonna need some help from Sven in the end for sure.

    Now it's time to give alivenotdead.com a visit. Copy/Paste the register- as well as the login-url into a new document or right into the template.

    Paste the urls right into the template:
    [setup]
    fixed url=http://www.alivenotdead.com/register.html
    url must have=*://*www.alivenotdead.com*
    description=Creates a blog entry on www.alivenotdead.com

    [REGISTER_STEP1]
    modify url=http://www.alivenotdead.com/register.html
    form url=http://www.alivenotdead.com/register.html

    [LOGIN_STEP1]
    modify url=http://www.alivenotdead.com/login.html
    form url=http://www.alivenotdead.com/login.html

    Once we've done that we open the register page of alivenotdead.com to inspect the fields. My advice to you is to use Firefox + Firebug. Firebug is a well known tool for all website developer and helps us to identify the fields and copy the names into our script.

    image

    Try to find the following forms and names with firebug:
    - form id (many sites don't have a proper form id though and it isn't necessary in most cases)
    - get the names of all visible text fields, drop down menus, checkboxes and radioboxes
    - find the name of the register-/sign up-/next-button. 

    image

    image

    Add your results to the template:

    [REGISTER_STEP1]
    form name=*regsubmit|*Sign Up (Please notice the "*")
    form id=form_register

    firstname=%spinfile-names.dat%
    lastname=%spinfile-lnames.dat%
    username=%login%
    emailnew=%your e-mail%
    passwordnew=%password%

    As this is a 3 step registration process we have to create..... exactly, 3 register steps. Please notice the changes of each form for each step. 

    [REGISTER_STEP2]
    form name=*regsubmit2|*Next
    form id=seccode

    seccodeverify=%captcha%

    [REGISTER_STEP3]
    form name=*shortregsubmit|*Next
    form id=register

    gendernew=%random_option%
    birthyear=%random_option%
    birthmonth=%random_option%
    birthday=%random_option%
    keepagesecret=%leave%
    region=%random_option%

    Next step is to paste unique registration success- or failed messages to the template. To get most of the "submit failed=" messages a good tip is to just click the "Sign Up"-Button with empty fields. Success  messages you will get once your registration was successful (e.g. "Account activated. Email sent to..")

    submit success=Hi, %login%!
    submit failed=This field is required|This field must contain at least 6 characters.
    submit success skip verify=This username already exists.|This email is already being used.
  • OzzOzz
    edited September 2012
    Once we have registered to the website we need to log out and log in again to obtain the field names.

    image

    [LOGIN_STEP1]

    submit success=>Logout<
    submit failed=Sorry, but the login information you provided was incorrect.

    verify on unknown status=1

    form name=*loginsubmit|*login
    form id=form_login

    set unknown variable=%leave%

    email=%your e-mail%
    password=%password%
    cookietime=%leave%

    Once we've logged in we have to find the URL to our settings because our profile or blog post is not been viewable for anyone (not public) for now. We need to change that first in "Settings" -> "Member Settings".

    [STEP1]
    modify url=http://www.alivenotdead.com/spacecp.php?action=userspaces&op=basicsettings

    form name=*Submit|*basicsettingssubmit
    form id=theform

    set unknown variable=%leave%

    firstname=%leave%
    lastname=%leave%
    language=en <-- you get those values with firebug/html source code
    birthyear=%leave%
    birthmonth=%leave%
    birthday=%leave%
    mobilephone=%columnspinfile-address_data.dat-6%
    email_more=%leave%
    privacy_pm=%random_option%
    privacy_guestbook=%leave%
    privacy_artists_communication=%leave%
    privacy_viewable=anyone
    privacy_partner=%leave%

    image

    As you can see we %leave% most fields because they are not definable or already be defined. 

    Now its time to change our profile ("About" -> "Edit my details") in Step2.
    Again, we change the URL to that site directly by "modify by=", copy/paste all fields to our template and define the variables.

    [STEP2]
    modify url=http://www.alivenotdead.com/spacecp.php?action=userspaces&op=userfield

    form name=*Save all
    form id=theform

    set unknown variable=%leave%
    variable must be used=url <-- GSA is forced to post our link somewhere to the site

    city=%random_option%
    city_other=%leave%
    languages_spoken[]=%random_option%
    marriage=%random_option%
    interested_in_dating_men=%random_option%
    bloodtype=%random_option%
    smoke=%random_option%
    drink=%random_option%
    privacy_age=%random_option%
    privacy_birthday=%random_option%
    privacy_astrology=%random_option%
    privacy_chinesezodiac=%random_option%
    privacy_chinesezodiac=%random_option%
    about_me_text=%about_yourself%
    hometown=%leave%
    job=%leave%
    address=%leave%
    school=%leave%
    affiliations=%leave%
    interest[]=%leave%
    favorite_music=%leave%
    favorite_movies=%leave%
    favorite_tv_shows=%leave%
    favorite_tv_shows=%leave%
    favorite_books=%leave%
    favorite_faves=%leave%
    myspace=%leave%
    xanga=%leave%
    sina_blog=%leave%
    qq=%leave%
    friendster=%leave%
    other_website_links=%url%

    If you have done that the script should look something like this: http://pastie.org/pastes/4675070/text?key=sytqzd3y4ps9lucdnugw

    One last word to the "verification" and "submission" lines in the script. Submission success/failed needs only to be defined in very first step (=Register_Step1, Login_Step1, Step1). Verification need only to be defined if GSA has to verify your registration (by email) or to verify if your link is posted (by url). If we "verify by url" you have to define where GSA will find your backlink. This should also be defined in the very first step, if needed.
  • OzzOzz
    edited September 2012
    Extra-Step/submit blog/article - TEXT  TO THIS STEP COMING SOON

    [EXTRA_STEP1]

    submit success=
    submit failed=

    verify submission=1
    verify interval=1
    verify timeout=1
    verify url=
    first verify=1
    verify by=url
    verify on unknown status=1

    form name=*Publish entry
    form id=theform

    set unknown variable=%leave%
    variable must be used=url

    subject=%article_title%
    catid=31
    ;31 >Standard Entry<
    ;32 >Video<
    ;33 >Information<
    ;34 >Press Release<
    ;35 >Event<
    ;79 >News<

    itemtypeid=%leave%
  • OzzOzz
    edited September 2012
    [Script] alivenotdead.com

    [setup]
    enabled=1
    default checked=0

    url must have=*://*www.alivenotdead.com*

    engine type=Web 2.0
    description=Creates a blog entry on www.alivenotdead.com

    dofollow=1
    anchor text=1
    uses pages=0

    extract keywords=0

    skip ext links on=0
    skip content on=0
    skip url content on=0
    posted domain check=1
            
    [Your E-Mail]
    type=email

    [URL]
    type=url

    [Anchor_Text]
    type=text

    [About_Yourself]
    type=memo
    allow html=0
    allow return=1
    must be filled=0
    hint=Please fill this with some information about yourself. Use the spin syntax heavily here.
    custom mode=0
    auto modify=1
    default=%file-about_yourself.dat%

    [Article_Title]
    type=text
    allow html=0
    must be filled=1
    hint=capitalize each word.
    max length=45

    [Article]
    type=memo
    allow html=1
    html to bbs=1
    must be filled=1
    hint=The full article comes here.
    auto modify=1

    [Login]
    type=login
    must be filled=1
    hint=The login for websites that need an account. Use numbers and letters only.
    min length=5
    static=1
    upcase=0

    [Password]
    type=password
    must be filled=1
    hint=A password used for websites that need an account. Use numbers and letters only.
    min length=6
    static=1

    [verify_url]
    type=extract
    find link=%article_title%

    [REGISTER_STEP1]

    submit success=Hi, %login%!
    submit failed=This field is required|This field must contain at least 6 characters.
    submit success skip verify=This username already exists.|This email is already being used.

    form name=*regsubmit|*Sign Up
    form id=form_register
    ;|register

    verify submission=1
    verify interval=10
    verify timeout=30
    first verify=0
    verify by=email
    verify on unknown status=1

    set unknown variable=%leave%

    firstname=%spinfile-names.dat%
    lastname=%spinfile-lnames.dat%
    username=%login%
    emailnew=%your e-mail%
    passwordnew=%password%
    seccodeverify=%captcha%
    gendernew=%random_option%
    birthyear=%random_option%
    birthmonth=%random_option%
    birthday=%random_option%
    keepagesecret=%leave%
    region=%random_option%

    [REGISTER_STEP2]
    form name=*shortregsubmit|*Next
    form id=register

    [LOGIN_STEP1]

    submit success=>Logout<
    submit failed=Sorry, but the login information you provided was incorrect.

    verify on unknown status=1

    form name=*loginsubmit|*login
    form id=form_login

    set unknown variable=%leave%

    email=%your e-mail%
    password=%password%
    cookietime=%leave%

    [STEP1]

    submit success=
    submit failed=

    verify submission=1
    verify interval=1
    verify timeout=1
    first verify=0
    verify by=url
    verify on unknown status=1

    form name=*Submit|*basicsettingssubmit
    form id=theform

    set unknown variable=%leave%

    firstname=%leave%
    lastname=%leave%
    language=en
    birthyear=%leave%
    birthmonth=%leave%
    birthday=%leave%
    mobilephone=%columnspinfile-address_data.dat-6%
    email_more=%leave%
    privacy_pm=%random_option%
    privacy_guestbook=%leave%
    privacy_artists_communication=%leave%
    privacy_viewable=anyone
    privacy_partner=%leave%

    [STEP2]

    form name=*Save all
    form id=theform

    set unknown variable=%leave%
    variable must be used=url

    city=%random_option%
    city_other=%leave%
    languages_spoken[]=%random_option%
    marriage=%random_option%
    interested_in_dating_men=%random_option%
    bloodtype=%random_option%
    smoke=%random_option%
    drink=%random_option%
    privacy_age=%random_option%
    privacy_birthday=%random_option%
    privacy_astrology=%random_option%
    privacy_chinesezodiac=%random_option%
    privacy_chinesezodiac=%random_option%
    about_me_text=%about_yourself%
    hometown=%leave%
    job=%leave%
    address=%leave%
    school=%leave%
    affiliations=%leave%
    interest[]=%leave%
    favorite_music=%leave%
    favorite_movies=%leave%
    favorite_tv_shows=%leave%
    favorite_tv_shows=%leave%
    favorite_books=%leave%
    favorite_faves=%leave%
    myspace=%leave%
    xanga=%leave%
    sina_blog=%leave%
    qq=%leave%
    friendster=%leave%
    other_website_links=%url%

    [EXTRA_STEP1]

    submit success=
    submit failed=

    verify submission=1
    verify interval=1
    verify timeout=1
    verify url=%verify_url%
    first verify=1
    verify by=url
    verify on unknown status=1

    form name=*Publish entry
    form id=theform

    set unknown variable=%leave%
    variable must be used=url

    ;add fixed data=message=%article%
    ;add fixed data condition=message

    subject=%article_title%
    catid=31
    itemtypeid=%leave%
    message==%article%
  • SvenSven www.GSA-Online.de
    wow Im always impressed by the input you give. Seeing people care is more important than a sale for me :) Thanks a lot. I keep this engine out of the next release OK?
  • OzzOzz
    yes, we need to add the extra-step for the blog first. for this I need your help anyway as I couldn't solve the hardest part: %article%. the rest of my write up will be done tomorrow.
  • SvenSven www.GSA-Online.de

    Well Im afraid Im offline from tomorrow the whole weekend. I can't help you here. But basically it would work like this...

    [Article]
    ; 1 = means to add a url if this form must use one (variable must be user).
    auto add anchor url=1

    ;you can also define what to put at the end of the article (use %url% and %anchor%). This however gets only added if no word like %anchor_text% can be found in the article to get transformed into a link
    auto add anchor url content=..

    [EXTRA_STEP1]
    variable must be used=url
    article=%article%

  • OzzOzz
    I meant to define the field for %article%. I always struggle with that. I've edited a basic code for the extra-step in post#3

    This is the field for the article:
    image
  • SvenSven www.GSA-Online.de

    Ahh you mean the form variables? Well I always use Opera (my standard browser) for that. There you can right click on a website and choose -> "Elemente untersuchen" and you get the correct form variable. But in this case it seems to be a iframe!? Might need to debug with WireShark or alike and see what packet is sent. and if there are some undefines variables that are not seen in the source you need to add this...


    add fixed data=<formvar>=content;<formvar2=content;..


  • BytefakerBytefaker
    edited September 2012
    @Ozz: Just wanted to say huge thanks for your impressive input again. That's not naturally. PS: I like your taste of humor beside this technical stuff :D (this comment can be delete as I don't want to mess up this wonderful thread)
  • OzzOzz
    edited September 2012
    @bytefaker: No problem. I'm happy that at least one person reads this wall of text (or scrolled down to the comment box), hehe

    @Sven: I think I've found the right form variable.
    <textarea id="message_textarea" style="display: none;" name="message"> </textarea>

    I did not test this, but if thats working I think to search for "<textarea id="" in the source code is a good identifier in general.
  • SvenSven www.GSA-Online.de
    well in this case it is message=%article% as it has a proper name. In this case you don't even have to add it with "add fixed data" if it is part of the form we want to submit to.
  • OzzOzz
    thx. I always thought if the field is hidden or "greyed out" in the source code than I need to define "add fix data".

    I'm just testing this beast now, but had to change some things in the registration steps which was odd because it worked yesterday. However, I've got 1 submission atm and hopefully a blog entry soon.
  • OzzOzz
    edited September 2012
    IT'S FU**ING WORKING!!
    (somehow)

    I posted the script in post #4. But I think that the "verify_url" is not correct atm. I set a "default=" in [verify_url] and hoped that SER will join this link to find the blog-url. I fear that this is not right and need to be fixed. It has posted a profile and a blog :)

    However, I'm happy that I've finally submitted anything successful 
    :bz

    Edit: @Sven: You can add the script to SER if you like to
  • SvenSven www.GSA-Online.de
    Cool, will have a look on that verify_url tomorrow. thanks again for this awesome work.
  • LWDLWD
    What an AWESOME instructions! Exactly what I was after for! I will give it a go for sure and post back result in the next couple of days...
  • mamadoumamadou
    Thanks man , but I don't think I will be doing this any time soon! , I think it will take 2 hours to add just one site and the script have to be fixed 2 days ago after the site's admin change the site!!.

    You did a very good job with lots of efforts but it is really very complicated way to add sites.
  • nicericenicerice
    Wow Ozz, this thread is brilliant. Thanks for this as it was exactly what I was looking for. I will be looking into using this for some platforms I would like to post to. If I prove they are working then I will share here.

    Thanks
    NR
  • skohskoh
    edited March 2013
    Awesome Tut ozz! quick question, I'm working on a web 2.0 that is similar... I'm good up until one part, it's saving the post instead of publishing it... How does GSA know what button to push on submitting the content? I'm not quite clear on the last step (publishing) Thanks!
  • OzzOzz
    hmm, maybe you are pushing the wrong button for that Web20? instead of "save" search for a "publish" button. apart from that its hard to tell without knowing anything about the site.
  • skohskoh
    Yeah that just made me realize what I was doing wrong. Thanks
  • skohskoh
    form name=*regsubmit|*Sign Up (Please notice the "*")
    form id=form_register

    Ozz in this code... Is this where GSA knows where to click register? I don't understand how GSA knows what to do next after all of the fields are filled... Ie. Click the register or submit button on the form

    Also the * just means that anything can come after or before that text right?

    Thanks and sorry for the noob questions, trying to learn this stuff isn't easy for a noob lol
  • OzzOzz
    edited March 2013
    yes, form name/link/id help to determine the form where the script should acting or fill out the fields before moving forward.
    the "*" is a wildcard like you've stated right.

    PS: those aren't noob questions as you've more knowledge about scripting than 99% of all SER users already.
  • skohskoh
    Thanks bro, appreciate it. I have a question on verifying, but I'll save that for another day ;)
  • OzzOzz
    i hope you will share the script once its ready to use.
  • skohskoh
    ^ absolutely, I plan on doing several and all will be shared for the benefit of this great community... That is I can get it working and I'm almost there 
  • skohskoh
    Ok, so this is bugging me... On this platform I am trying to engine, after the submission is successful the page has the output URL or successful URL found on it inside the admin panel.. How can I skip the verify url by search and try to extract the url from inside the admin panel? Any ideas? 
  • SvenSven www.GSA-Online.de

    Something like this...maybe have a closer look at existing engines.


    [verify_url]

    type=extract

    url=the url where the link can be extracted

    find url=*url mask of the link*

    find link=*name or mask of the link*


    ...


    verify url=% verify_url% 

  • awintawint
    @Ozz
     alivenotdead.com [STEP1] seem only need to do one time, but it seem GSA will do STEP1 each time , right?  how to fix it?
  • SvenSven www.GSA-Online.de
    Depends what STEP1 does, you can use "optional form" or "optional find url link" to let it continue in case it would fail here.
Sign In or Register to comment.