Request to update the script manual with the latest variables used in the current engine scripts

sashilover

Hi @sven, I tried to read the engine scripts used in the SER, and found it's a bit difficulty to understand because many of variables are not described in the script manual, so could you please update the script manual to describe all variables, this will help us developer custom engines or refine the current enginues...
Here are a few variables not found in the manual:
------
[LOGIN_STEP1]

pass=%password%

name=%login%

mail=%your e-mail%

pass[pass1]=%password%

pass[pass2]=%password%

files[picture_upload]=%leave%

language=%leave%

timezone=%leave%

captcha_response=%captcha%

edit[name]=%login%

edit[pass]=%password%

*mail*=%your e-mail%

*pass*=%password%

mimemail_textonly=0

files[picture_upload]=%leave%

name=%leave%

timezone=%leave%

language=%leave%

adtype_select=%random_option%

adstatus=%leave%

signature[format]=*html*
------

Sven

Actually everything is documented as far as i know. That stuff you have here are the names of the forms on the webpage like
<input name="pass" type="password">

sashilover

Thanks for your answer, two questions:
1). Does this mean I can add custom field in the script if I find a new field in the form?
2). What does the "*html*|1" mean in the script?
---------------
comment_body[und][0][format]=*html*|1
----------------

Sven

1) yes of course
2) It means it will try to match that field first finding *html* in the options and 1 next.

sashilover

I am still confused with the second one, could you please give me a sample about how it looks like?

Sven

there is a select field on the webpage with....
<select name="comment_body[und][0][format]">
  <option value="1">html</option>
  <option value="2">plain text</option>
</select>

So SER tries to make the selection based on the mask in that variable, it tries to select the "html" entry and not the "plain text" one.

sashilover

Got it, thanks.

sashilover

Hi @sven, I have questions about the [*_step1][*_step2]...scenario, let's take a look at the Joomla K2 script:

----------

[LOGIN_STEP1]

find link=Login

find url=%targethost%*component/users/?view=login|%targethost%*view=login*

alternative url=/index.php/component/users/?view=login

;1 = Download the new URL and continue with STEP2 or finish the submission if nothing else should be done.

just download=1

submit success=>Log out<|value="user.logout"|Edit Profile<|Profil bearbeiten<|Editar perfil<|<fieldset id="users-profile-core">|>Logout<|>Your Details<

submit failed=Your account has either been blocked or you have not activated it yet.|Username and password do not match or you do not have an account yet.|Invalid Token|<div class="alert alert-warning">|<h4 class="alert-heading">Warning</h4>|data-dismiss="alert">׼/a>

;1 = if a submission is not detected as successful or failed it will still be taken as successful (appearing in log with “unknown submission status”)

verify on unknown status=1

username=%login%

passwd=%password%

remember=1

[LOGIN_STEP2]

modify step=3

modify step condition=="username"|=username|='username'|="passwd"|=passwd|='passwd'

modify url=/index.php?option=com_user&view=login

just download=1

[LOGIN_STEP3]

form name=login - Login|FORM 1 - Log in|*Log in

form url=*/component/users/?task=user.login|*option=com_user*

form id=login

seconds to wait before submission=%random-5-10%

set unknown variable=%leave%
---------------------

What exactly does [Login_step1][Login_step2][Login_step2] do? why do we need 3 steps?
My understanding is:
1). Login_step1 find the login URL like: /index.php/component/users/?view=login? then download it? fill the fields with specified value, then submit the form? then match the message to check if it's submitted successfully or fail?

2). Why do we need the [Login_step2] and [Login_step3]? will the login_step2 and login_step3 execute no matter successful or failed of [login_step1]?

Please advise, thanks.

Sven

pay attention to the "modify step" and "modify step condition" statements.
LOGIN_STEP1 is performed and then LOGIN_STEP2. On that second step the condition is checked if we are really on the login page, if so, we move to the last step and perform he actual submission, else we try to go to another login page.

This is usually not needed but for K2 you usually have no login link or form. It's hidden and you have to try several login-urls till you find the right one.

sashilover

I understand the "modify step" and "modify step condition", what I don't understand is when the URL and form found in [login_step1], I suppose the submission will proceed?
In this case, will the [login_step2] be executed?

Sven

login_step1 - try to dl one possible login-url
login_Step2 - check if thats correct, if so...jump to step3 and submit else try another url
login_step3 - fill form and submit

sashilover

I think I understand it, it works like this:
[Login_step1] --- Download the link from the "login_step1".
[Login_step2] --- Check condition on the page from "login_step1", if yes, go to "login_step3", fill form and submit, if the condition doesn't match, download the page in the "login_step2", then go to "login_step3" fill form and submit.
Please correct if I miss something, thanks.

Sven

yes, thats correct.

sashilover

Hi @Sven, I have a question for the K2 engine, I want to ignore the post step if the %user_id% large than certain number let's say 10000, how to modify the script?
----

[Step1]

modify step=2

modify step condition=%user_id%>10000

[Step2]
Abort post article...

.....
------
Please advise.

Sven

you can do that by:
modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&amp;user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&amp;user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&amp;id=[1-9][0-9][0-9][0-9][0-9]*

sashilover

Hi @Sven, it doesn't seem to work, I add the snippet in the [Step1] as following:
----

[STEP1]

modify step=2

modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&amp;user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&amp;user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&amp;id=[1-9][0-9][0-9][0-9][0-9]*

.....
[STEP2]
just download=1
------

I tested a few sites and checked the debug log, the [Step2] never executed no matter whether the condition meets or not...

Please advise.
Thanks.

Sven

well you have to give step1 something to download...then the modify step is applies based on the condition. you might have to add an extra step here.

sashilover

---

[STEP1]

modify step=3

modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&amp;user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&amp;user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&amp;id=[1-9][0-9][0-9][0-9][0-9]*

modify url=%edit_profile_url%

just download=1

[STEP2]

;Defines the type of backlink created. Can be anything you want but you might want to use the types already used in other scripts.

link type=Article

;This is used to change a found URL to something else.

modify url=%edit_profile_url%

submit success=Your settings have been saved|Edit Profile</a>|Profil bearbeiten<|Profile successfully saved|Editar perfil<|El perfil se ha guardado correctamente|Profil enregistr꽐rofil başarıyla kaydedildi.|Profilo correttamente salvato

submit failed=Benutzername und Passwort falsch oder das Benutzerkonto existiert noch nicht!|Login denied!|Your account has either been blocked or you have not activated it yet.|<div class="alert alert-warning">|<h4 class="alert-heading">Warning</h4>|data-dismiss="alert">׼/a>

;After a successful submission the URL is put to the verification list and checked in intervals if the link appears there.

verify submission=1

;url = downloads the URL in ”verify url” and checks for the present of “verify search for” which is by default your URL

verify by=url

verify url=/?option=com_k2&view=itemlist&task=user&id=%user_id%|/component/k2/itemlist/user/%user_id%|/index.php/component/k2/itemlist/user/%user_id%

first verify=0

verify interval=20

verify timeout=30

;1 = if a submission is not detected as successful or failed it will still be taken as successful (appearing in log with “unknown submission status”)

verify on unknown status=1

;???? Don't understand

verify search detail url=0

;Will wait a given number of seconds before the form is actually submitted. This is useful to not trigger some robot detection system for some engines.

seconds to wait before submission=%random-5-10%

;Try to find a form on the current webpage that has a name like the variable content. If no name is used in the <form> tag, the caption of the submit button is used.

;Example: form name=*Sign Up|*Register

;This tries to find a form there the name or the submit button's name is either matching “*Sign Up” or “*Register”.

form name=userform - Save*|userform -  Sauvegarder|userform*|*Save*

;Try to find a form where the submission URL would match the variable content.

;Example: form url=*/signup.php|*/signup/

;This tries to find a form where the resulting submission URL after pressing the submit button would match “*/signup.php” or “*/signup/”.

form url=*/index.php/component/users/

;This will add a form field if the thing in “add fixed data condition” is found on the webpage. This works only if both variables are defined. Don't worry about this to much as you will probably never use this. It is only required if some sites add data to forms by javascript to prevent automated submissions. You can use “remove fixed data” to just remove that from the posting.

;Example: 

;add fixed data=code=%captcha%;hidden=0

;add fixed data condition=code_sign;hidden_data

;If the webpage has anywhere the word “code_sign” in it we add a new form field called code that will be filled with a captcha. And if the word “hidden_data” ;is found we also add a new form field called “hidden” with the content “0”.

;-----

add fixed data=url=%leave%

add fixed data condition=="description"

;If a form field is unknown as we didn't define how to fill it in our engine, we could still fill it by something you define here. The submission aborts if this is not defined and something is unable to get filled. The “set unknown variable condition” doesn't have to be used but if it is, the string on that variable must be present to use this.

set unknown variable=%leave%

;A form is only submitted if certain variables have been used in that form. Example: variable must be used=url,login|email

;The above example requires a form to use the variable “url” or “login” and “email”.\\

variable must be used=url

jform[name]=%article_title%

jform[email1]=%rnd_email%

jform[email2]=%rnd_email%

jform[password1]=%leave%

jform[password2]=%leave%

;性别随机

gender=%random_option%

;描述修改为文章

description=%article%

;Avatar

image=%Profile_Image%

;URL链接

url=%url%

jform[params][editor]=%leave%

jform[params][timezone]=%leave%

jform[params][language]=%leave%

jform[profile][address1]=%columnspinfile-address_data.dat-5%

jform[profile][address2]=%leave%

jform[profile][city]=%columnspinfile-address_data.dat-3%

jform[profile][region]=%columnspinfile-address_data.dat-2%

jform[profile][country]=%columnspinfile-address_data.dat-1%|%random_option%

jform[profile][postal_code]=%columnspinfile-address_data.dat-4%

jform[profile][phone]=%columnspinfile-address_data.dat-6%

jform[profile][website]=%url%

jform[profile][favoritebook]=%leave%

jform[profile][aboutme]=%article%

jform[profile][dob]=%leave%

[STEP3]

just download=1
---------

THis is the script I modified, it still doesn't work, NO posts are submitted at all, I don't understand the concepts of [Extra_step*], what's it, when would be used? why don't just use [step1,2,3....x] instead?

Please advise.

Sven

you have to keep the "submission success/failed" and "verify *" stuff in step1...always.

sashilover

Hi @sven, I appreciate your help, the scripts blow still doesn't work, it posts no matter the condition meets or not..

-----

[STEP1]

;Defines the type of backlink created. Can be anything you want but you might want to use the types already used in other scripts.

link type=Article

;This is used to change a found URL to something else.

modify url=%edit_profile_url%

submit success=Your settings have been saved|Edit Profile</a>|Profil bearbeiten<|Profile successfully saved|Editar perfil<|El perfil se ha guardado correctamente|Profil enregistr꽐rofil başarıyla kaydedildi.|Profilo correttamente salvato

submit failed=Benutzername und Passwort falsch oder das Benutzerkonto existiert noch nicht!|Login denied!|Your account has either been blocked or you have not activated it yet.|<div class="alert alert-warning">|<h4 class="alert-heading">Warning</h4>|data-dismiss="alert">׼/a>

;After a successful submission the URL is put to the verification list and checked in intervals if the link appears there.

verify submission=1

;url = downloads the URL in ”verify url” and checks for the present of “verify search for” which is by default your URL

verify by=url

verify url=/?option=com_k2&view=itemlist&task=user&id=%user_id%|/component/k2/itemlist/user/%user_id%|/index.php/component/k2/itemlist/user/%user_id%

first verify=0

verify interval=20

verify timeout=30

;1 = if a submission is not detected as successful or failed it will still be taken as successful (appearing in log with “unknown submission status”)

verify on unknown status=1

;???? Don't understand

verify search detail url=0

;Will wait a given number of seconds before the form is actually submitted. This is useful to not trigger some robot detection system for some engines.

seconds to wait before submission=%random-5-10%

;需要下载profile的页面

just download=1

[STEP2]

;自添加的字段, 过滤掉ID>10000的站点,如果满足条件,则直接跳转到Step3中

modify step=3

;过滤条件

modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&amp;user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&amp;user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&amp;id=[1-9][0-9][0-9][0-9][0-9]*

;This is used to change a found URL to something else.

modify url=%edit_profile_url%

;Try to find a form on the current webpage that has a name like the variable content. If no name is used in the <form> tag, the caption of the submit button is used.

;Example: form name=*Sign Up|*Register

;This tries to find a form there the name or the submit button's name is either matching “*Sign Up” or “*Register”.

form name=userform - Save*|userform -  Sauvegarder|userform*|*Save*

;Try to find a form where the submission URL would match the variable content.

;Example: form url=*/signup.php|*/signup/

;This tries to find a form where the resulting submission URL after pressing the submit button would match “*/signup.php” or “*/signup/”.

form url=*/index.php/component/users/

;This will add a form field if the thing in “add fixed data condition” is found on the webpage. This works only if both variables are defined. Don't worry about this to much as you will probably never use this. It is only required if some sites add data to forms by javascript to prevent automated submissions. You can use “remove fixed data” to just remove that from the posting.

;Example: 

;add fixed data=code=%captcha%;hidden=0

;add fixed data condition=code_sign;hidden_data

;If the webpage has anywhere the word “code_sign” in it we add a new form field called code that will be filled with a captcha. And if the word “hidden_data” ;is found we also add a new form field called “hidden” with the content “0”.

;-----

;页面中出现"description"文字, 则在form的字段中增加为URL的字段, 值为空?

add fixed data=url=%leave%

add fixed data condition=="description"

;If a form field is unknown as we didn't define how to fill it in our engine, we could still fill it by something you define here. The submission aborts if this is not defined and something is unable to get filled. The “set unknown variable condition” doesn't have to be used but if it is, the string on that variable must be present to use this.

set unknown variable=%leave%

;A form is only submitted if certain variables have been used in that form. Example: variable must be used=url,login|email

;The above example requires a form to use the variable “url” or “login” and “email”.\\

variable must be used=url

;修改Name为文章的Title

jform[name]=%article_title%

;使用的email

jform[email1]=%rnd_email%

jform[email2]=%rnd_email%

;默认密码

jform[password1]=%leave%

jform[password2]=%leave%

;性别随机

gender=%random_option%

;描述修改为文章

description=%article%

;Avatar

image=%Profile_Image%

;URL链接

url=%url%

jform[params][editor]=%leave%

jform[params][timezone]=%leave%

jform[params][language]=%leave%

jform[profile][address1]=%columnspinfile-address_data.dat-5%

jform[profile][address2]=%leave%

jform[profile][city]=%columnspinfile-address_data.dat-3%

jform[profile][region]=%columnspinfile-address_data.dat-2%

jform[profile][country]=%columnspinfile-address_data.dat-1%|%random_option%

jform[profile][postal_code]=%columnspinfile-address_data.dat-4%

jform[profile][phone]=%columnspinfile-address_data.dat-6%

jform[profile][website]=%url%

jform[profile][favoritebook]=%leave%

jform[profile][aboutme]=%article%

jform[profile][dob]=%leave%

[STEP3]

just download=1
-------------

PLEASE share me more detailed info about how to get it work, thanks in advance.

Sven

well what do you see in the debug mode?

sashilover

The debug log is attached.

Sven

ok and did he find the correct profile link?

sashilover

There is no data for the variable "edit_profile_url" in "view account data" window... how to verify if profile link is found or not?

sashilover

I write the variable %edit_profile_url% to file, it shows the wrong profile edit URL...
The finial post URL is like:
http://www.xxxx.com/component/k2/itemlist/user/109110332

The edit profile url is:
http://www.xxx.com/component/users/?task=profile.edit&user_id=109110332

Sven

so this might be the problem here then...it has the wrong url to check.

sashilover

the "modify step" doesn't execute no matter what the condition is, I modified the scripts as following:
----

[STEP2]

modify step=3

modify step condition=*user*

modify url =/index.php/component/users/profile
----
It is supposed to jump to [Step3], but it never does.
Could you please help me make it right..
Thanks.

Sven

send me the whole thing with a test url please.

sashilover

PM sent, thanks.

sashilover

----------
[find_alt_url]

type=extract

find url=*/add/?* !image !add/agenda !add/poll !add/offer !add/order !add/album !add/gallery !add/apply-jobs|*?q=add* !image !add/agenda !add/poll !add/offer !add/order !add/album !add/gallery !add/apply-jobs !add/notice
-----------
How is this string " */add/?* !image !add/agenda !add/poll !add/offer !add/order !add/album !add/gallery !add/apply-jobs" being interpreted?
Try to find a URL with "*/add/?*" AND page doesn't have the elements "image" AND "add/agenda ......" ?

What does the BLANK mean? 

Sven

A blank is handled as a separator since a space is not part of a regular url. That ! sign is a negotiator and means that the next part shout NOT be part of the found URL.