Request to update the script manual with the latest variables used in the current engine scripts

sashilover · June 2017

Hi @sven, I tried to read the engine scripts used in the SER, and found it's a bit difficulty to understand because many of variables are not described in the script manual, so could you please update the script manual to describe all variables, this will help us developer custom engines or refine the current enginues...
Here are a few variables not found in the manual:
------
[LOGIN_STEP1]

pass=%password%

name=%login%

mail=%your e-mail%

pass[pass1]=%password%

pass[pass2]=%password%

files[picture_upload]=%leave%

language=%leave%

timezone=%leave%

captcha_response=%captcha%

edit[name]=%login%

edit[pass]=%password%

*mail*=%your e-mail%

*pass*=%password%

mimemail_textonly=0

files[picture_upload]=%leave%

name=%leave%

timezone=%leave%

language=%leave%

adtype_select=%random_option%

adstatus=%leave%

signature[format]=*html*
------

Sven · June 2017

Actually everything is documented as far as i know. That stuff you have here are the names of the forms on the webpage like
<input name="pass" type="password">

sashilover · June 2017

Thanks for your answer, two questions:
1). Does this mean I can add custom field in the script if I find a new field in the form?
2). What does the "*html*|1" mean in the script?
---------------
comment_body[und][0][format]=*html*|1
----------------

Sven · June 2017

1) yes of course
2) It means it will try to match that field first finding *html* in the options and 1 next.

sashilover · June 2017

I am still confused with the second one, could you please give me a sample about how it looks like?

Sven · June 2017

there is a select field on the webpage with....
<select name="comment_body[und][0][format]">
<option value="1">html</option>
<option value="2">plain text</option>
</select>

So SER tries to make the selection based on the mask in that variable, it tries to select the "html" entry and not the "plain text" one.

sashilover · June 2017

Got it, thanks.

sashilover · June 2017

Hi @sven, I have questions about the [*_step1][*_step2]...scenario, let's take a look at the Joomla K2 script:

----------

[LOGIN_STEP1]

find link=Login

find url=%targethost%*component/users/?view=login|%targethost%*view=login*

alternative url=/index.php/component/users/?view=login

;1 = Download the new URL and continue with STEP2 or finish the submission if nothing else should be done.

just download=1

submit failed=Your account has either been blocked or you have not activated it yet.|Username and password do not match or you do not have an account yet.|Invalid Token|<div class="alert alert-warning">|<h4 class="alert-heading">Warning</h4>|data-dismiss="alert">׼/a>

;1 = if a submission is not detected as successful or failed it will still be taken as successful (appearing in log with “unknown submission status”)

verify on unknown status=1

username=%login%

passwd=%password%

remember=1

[LOGIN_STEP2]

modify step=3

modify url=/index.php?option=com_user&view=login

just download=1

[LOGIN_STEP3]

form name=login - Login|FORM 1 - Log in|*Log in

form url=*/component/users/?task=user.login|*option=com_user*

form id=login

seconds to wait before submission=%random-5-10%

set unknown variable=%leave%
---------------------

What exactly does [Login_step1][Login_step2][Login_step2] do? why do we need 3 steps?
My understanding is:
1). Login_step1 find the login URL like: /index.php/component/users/?view=login? then download it? fill the fields with specified value, then submit the form? then match the message to check if it's submitted successfully or fail?

2). Why do we need the [Login_step2] and [Login_step3]? will the login_step2 and login_step3 execute no matter successful or failed of [login_step1]?

Please advise, thanks.

Sven · June 2017

pay attention to the "modify step" and "modify step condition" statements.
LOGIN_STEP1 is performed and then LOGIN_STEP2. On that second step the condition is checked if we are really on the login page, if so, we move to the last step and perform he actual submission, else we try to go to another login page.

This is usually not needed but for K2 you usually have no login link or form. It's hidden and you have to try several login-urls till you find the right one.

sashilover · June 2017

I understand the "modify step" and "modify step condition", what I don't understand is when the URL and form found in [login_step1], I suppose the submission will proceed?
In this case, will the [login_step2] be executed?

Sven · June 2017

login_step1 - try to dl one possible login-url
login_Step2 - check if thats correct, if so...jump to step3 and submit else try another url
login_step3 - fill form and submit

sashilover · June 2017

I think I understand it, it works like this:
[Login_step1] --- Download the link from the "login_step1".
[Login_step2] --- Check condition on the page from "login_step1", if yes, go to "login_step3", fill form and submit, if the condition doesn't match, download the page in the "login_step2", then go to "login_step3" fill form and submit.
Please correct if I miss something, thanks.

Sven · June 2017

yes, thats correct.

sashilover · June 2017

Hi @Sven, I have a question for the K2 engine, I want to ignore the post step if the %user_id% large than certain number let's say 10000, how to modify the script?
----

[Step1]

modify step=2

modify step condition=%user_id%>10000

[Step2]
Abort post article...

.....
------
Please advise.

Sven · June 2017

you can do that by:
modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&id=[1-9][0-9][0-9][0-9][0-9]*

sashilover · June 2017

Hi @Sven, it doesn't seem to work, I add the snippet in the [Step1] as following:
----

[STEP1]

modify step=2

modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&id=[1-9][0-9][0-9][0-9][0-9]*

.....
[STEP2]
just download=1
------

I tested a few sites and checked the debug log, the [Step2] never executed no matter whether the condition meets or not...

Please advise.
Thanks.

Sven · June 2017

well you have to give step1 something to download...then the modify step is applies based on the condition. you might have to add an extra step here.

sashilover · June 2017

---

[STEP1]

modify step=3

modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&id=[1-9][0-9][0-9][0-9][0-9]*

modify url=%edit_profile_url%

just download=1

[STEP2]

;Defines the type of backlink created. Can be anything you want but you might want to use the types already used in other scripts.

link type=Article

;This is used to change a found URL to something else.

modify url=%edit_profile_url%

submit failed=Benutzername und Passwort falsch oder das Benutzerkonto existiert noch nicht!|Login denied!|Your account has either been blocked or you have not activated it yet.|<div class="alert alert-warning">|<h4 class="alert-heading">Warning</h4>|data-dismiss="alert">׼/a>

;After a successful submission the URL is put to the verification list and checked in intervals if the link appears there.

verify submission=1

;url = downloads the URL in ”verify url” and checks for the present of “verify search for” which is by default your URL

verify by=url

verify url=/?option=com_k2&view=itemlist&task=user&id=%user_id%|/component/k2/itemlist/user/%user_id%|/index.php/component/k2/itemlist/user/%user_id%

first verify=0

verify interval=20

verify timeout=30

;1 = if a submission is not detected as successful or failed it will still be taken as successful (appearing in log with “unknown submission status”)

verify on unknown status=1

;???? Don't understand

verify search detail url=0

;Will wait a given number of seconds before the form is actually submitted. This is useful to not trigger some robot detection system for some engines.

seconds to wait before submission=%random-5-10%

;Try to find a form on the current webpage that has a name like the variable content. If no name is used in the <form> tag, the caption of the submit button is used.

;Example: form name=*Sign Up|*Register

;This tries to find a form there the name or the submit button's name is either matching “*Sign Up” or “*Register”.

form name=userform - Save*|userform - Sauvegarder|userform*|*Save*

;Try to find a form where the submission URL would match the variable content.

;Example: form url=*/signup.php|*/signup/

;This tries to find a form where the resulting submission URL after pressing the submit button would match “*/signup.php” or “*/signup/”.

form url=*/index.php/component/users/

;This will add a form field if the thing in “add fixed data condition” is found on the webpage. This works only if both variables are defined. Don't worry about this to much as you will probably never use this. It is only required if some sites add data to forms by javascript to prevent automated submissions. You can use “remove fixed data” to just remove that from the posting.

;Example:

;add fixed data=code=%captcha%;hidden=0

;add fixed data condition=code_sign;hidden_data

;If the webpage has anywhere the word “code_sign” in it we add a new form field called code that will be filled with a captcha. And if the word “hidden_data” ;is found we also add a new form field called “hidden” with the content “0”.

;-----

add fixed data=url=%leave%

add fixed data condition=="description"

;If a form field is unknown as we didn't define how to fill it in our engine, we could still fill it by something you define here. The submission aborts if this is not defined and something is unable to get filled. The “set unknown variable condition” doesn't have to be used but if it is, the string on that variable must be present to use this.

set unknown variable=%leave%

;A form is only submitted if certain variables have been used in that form. Example: variable must be used=url,login|email

;The above example requires a form to use the variable “url” or “login” and “email”.\\

variable must be used=url

jform[name]=%article_title%

jform[email1]=%rnd_email%

jform[email2]=%rnd_email%

jform[password1]=%leave%

jform[password2]=%leave%

;性别随机

gender=%random_option%

;描述修改为文章

description=%article%

;Avatar

image=%Profile_Image%

;URL链接

url=%url%

jform[params][editor]=%leave%

jform[params][timezone]=%leave%

jform[params][language]=%leave%

jform[profile][address1]=%columnspinfile-address_data.dat-5%

jform[profile][address2]=%leave%

jform[profile][city]=%columnspinfile-address_data.dat-3%

jform[profile][region]=%columnspinfile-address_data.dat-2%

jform[profile][country]=%columnspinfile-address_data.dat-1%|%random_option%

jform[profile][postal_code]=%columnspinfile-address_data.dat-4%

jform[profile][phone]=%columnspinfile-address_data.dat-6%

jform[profile][website]=%url%

jform[profile][favoritebook]=%leave%

jform[profile][aboutme]=%article%

jform[profile][dob]=%leave%

[STEP3]

just download=1
---------

THis is the script I modified, it still doesn't work, NO posts are submitted at all, I don't understand the concepts of [Extra_step*], what's it, when would be used? why don't just use [step1,2,3....x] instead?

Please advise.

Sven · June 2017

you have to keep the "submission success/failed" and "verify *" stuff in step1...always.

sashilover · June 2017

Hi @sven, I appreciate your help, the scripts blow still doesn't work, it posts no matter the condition meets or not..

-----

[STEP1]

;Defines the type of backlink created. Can be anything you want but you might want to use the types already used in other scripts.

link type=Article

;This is used to change a found URL to something else.

modify url=%edit_profile_url%

submit failed=Benutzername und Passwort falsch oder das Benutzerkonto existiert noch nicht!|Login denied!|Your account has either been blocked or you have not activated it yet.|<div class="alert alert-warning">|<h4 class="alert-heading">Warning</h4>|data-dismiss="alert">׼/a>

;After a successful submission the URL is put to the verification list and checked in intervals if the link appears there.

verify submission=1

;url = downloads the URL in ”verify url” and checks for the present of “verify search for” which is by default your URL

verify by=url

verify url=/?option=com_k2&view=itemlist&task=user&id=%user_id%|/component/k2/itemlist/user/%user_id%|/index.php/component/k2/itemlist/user/%user_id%

first verify=0

verify interval=20

verify timeout=30

;1 = if a submission is not detected as successful or failed it will still be taken as successful (appearing in log with “unknown submission status”)

verify on unknown status=1

;???? Don't understand

verify search detail url=0

;Will wait a given number of seconds before the form is actually submitted. This is useful to not trigger some robot detection system for some engines.

seconds to wait before submission=%random-5-10%

;需要下载profile的页面

just download=1

[STEP2]

;自添加的字段, 过滤掉ID>10000的站点,如果满足条件,则直接跳转到Step3中

modify step=3

;过滤条件

modify step condition=*jform[id]" value="[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=profile.edit&user_id=[1-9][0-9][0-9][0-9][0-9]*|*view=profile&user_id=*|*view=profile&user_id=[1-9][0-9][0-9][0-9][0-9]*|*task=user&id=[1-9][0-9][0-9][0-9][0-9]*

;This is used to change a found URL to something else.

modify url=%edit_profile_url%

;Try to find a form on the current webpage that has a name like the variable content. If no name is used in the <form> tag, the caption of the submit button is used.

;Example: form name=*Sign Up|*Register

;This tries to find a form there the name or the submit button's name is either matching “*Sign Up” or “*Register”.

form name=userform - Save*|userform - Sauvegarder|userform*|*Save*

;Try to find a form where the submission URL would match the variable content.

;Example: form url=*/signup.php|*/signup/

;This tries to find a form where the resulting submission URL after pressing the submit button would match “*/signup.php” or “*/signup/”.

form url=*/index.php/component/users/

;This will add a form field if the thing in “add fixed data condition” is found on the webpage. This works only if both variables are defined. Don't worry about this to much as you will probably never use this. It is only required if some sites add data to forms by javascript to prevent automated submissions. You can use “remove fixed data” to just remove that from the posting.

;Example:

;add fixed data=code=%captcha%;hidden=0

;add fixed data condition=code_sign;hidden_data

;If the webpage has anywhere the word “code_sign” in it we add a new form field called code that will be filled with a captcha. And if the word “hidden_data” ;is found we also add a new form field called “hidden” with the content “0”.

;-----

;页面中出现"description"文字, 则在form的字段中增加为URL的字段, 值为空?

add fixed data=url=%leave%

add fixed data condition=="description"

;If a form field is unknown as we didn't define how to fill it in our engine, we could still fill it by something you define here. The submission aborts if this is not defined and something is unable to get filled. The “set unknown variable condition” doesn't have to be used but if it is, the string on that variable must be present to use this.

set unknown variable=%leave%

;A form is only submitted if certain variables have been used in that form. Example: variable must be used=url,login|email

;The above example requires a form to use the variable “url” or “login” and “email”.\\

variable must be used=url

;修改Name为文章的Title

jform[name]=%article_title%

;使用的email

jform[email1]=%rnd_email%

jform[email2]=%rnd_email%

;默认密码

jform[password1]=%leave%

jform[password2]=%leave%

;性别随机

gender=%random_option%

;描述修改为文章

description=%article%

;Avatar

image=%Profile_Image%

;URL链接

url=%url%

jform[params][editor]=%leave%

jform[params][timezone]=%leave%

jform[params][language]=%leave%

jform[profile][address1]=%columnspinfile-address_data.dat-5%

jform[profile][address2]=%leave%

jform[profile][city]=%columnspinfile-address_data.dat-3%

jform[profile][region]=%columnspinfile-address_data.dat-2%

jform[profile][country]=%columnspinfile-address_data.dat-1%|%random_option%

jform[profile][postal_code]=%columnspinfile-address_data.dat-4%

jform[profile][phone]=%columnspinfile-address_data.dat-6%

jform[profile][website]=%url%

jform[profile][favoritebook]=%leave%

jform[profile][aboutme]=%article%

jform[profile][dob]=%leave%

[STEP3]

just download=1
-------------

PLEASE share me more detailed info about how to get it work, thanks in advance.

Sven · June 2017

well what do you see in the debug mode?

sashilover · June 2017

The debug log is attached.

Image: https://forum.gsa-online.de/uploads/editor/8w/2xku4mxyww3c.png

Sven · June 2017

ok and did he find the correct profile link?

sashilover · June 2017

There is no data for the variable "edit_profile_url" in "view account data" window... how to verify if profile link is found or not?

sashilover · June 2017

I write the variable %edit_profile_url% to file, it shows the wrong profile edit URL...
The finial post URL is like:
http://www.xxxx.com/component/k2/itemlist/user/109110332

The edit profile url is:
http://www.xxx.com/component/users/?task=profile.edit&user_id=109110332

Sven · June 2017

so this might be the problem here then...it has the wrong url to check.

sashilover · June 2017

the "modify step" doesn't execute no matter what the condition is, I modified the scripts as following:
----

[STEP2]

modify step=3

modify step condition=*user*

modify url =/index.php/component/users/profile
----
It is supposed to jump to [Step3], but it never does.
Could you please help me make it right..
Thanks.

Sven · June 2017

send me the whole thing with a test url please.

sashilover · June 2017

PM sent, thanks.

sashilover · June 2017

----------
[find_alt_url]

type=extract

find url=*/add/?* !image !add/agenda !add/poll !add/offer !add/order !add/album !add/gallery !add/apply-jobs|*?q=add* !image !add/agenda !add/poll !add/offer !add/order !add/album !add/gallery !add/apply-jobs !add/notice
-----------
How is this string " */add/?* !image !add/agenda !add/poll !add/offer !add/order !add/album !add/gallery !add/apply-jobs" being interpreted?
Try to find a URL with "*/add/?*" AND page doesn't have the elements "image" AND "add/agenda ......" ?

What does the BLANK mean?

Sven · June 2017

A blank is handled as a separator since a space is not part of a regular url. That ! sign is a negotiator and means that the next part shout NOT be part of the found URL.

sashilover · June 2017

Thanks, make sense to me.

sashilover · June 2017

The [login_step*] seems to execute without waiting for the verification of registration being done, which causes a lot of "login failed/awaiting account verification" errors in Drupal engine, is there a specific variable to control when the [login_step*] being executed?

Sven · June 2017

yes, it's all in toe docu actually

"try to continue without verification"

sashilover · June 2017

Hi Sven, I have a problem on the registration on a Durpal site, the value of a hidden field is not specified successfully, please see the PM message

sashilover · July 2017

Hi @Sven, it seems that the value of "front","back" variable doesn't accept variable...
I want to extract first folder from a given URL as following:
----
Original url: http://www.example.com/subfolder/page-name.hmtl
I need to exact the first folder's name "subfolder" in this case.

[sub_folder]

type=extract

url=%original_targeturl%

front=%targethost%/

back=/

The script doesn't work, please advise, thanks.

Sven · July 2017

use this...

[sub_folder]
type=extract
input=%original_targeturl%

front1=://
front2=/
back=/

sashilover · July 2017

Thanks for your quick reply, it worked, however, I am a little confused about the snippet:
1). What exactly does the "input" variable mean? I can't find any info in the script document.
2). Are the "front1", "front2".... similar with the "AND" operator, which mean ALL the conditions that specified in "front1", "front2"... should meet during the extraction?

Thanks again.

Sven · July 2017

1) by default "input" is the content of the last downloaded/submitted webpage or if you specified url=... that content. If you specify something else with "input" it is using that.

---

docu was updated as well.

Sven · July 2017

2. front1 must be found and then front2, front3 and so on. If one does not match, it will fail. If you want to match with OR, you can use front1=variation1|variation2|...

sashilover · July 2017

Make sense to me, thanks.

sashilover · July 2017

Hi Sven, is it possible to add a particular step to handle the situation of un-successful submission? it will be very helpful to log those sites and check them manually... for example, a step like:
--------
[Unsuccessful_Submission_step1]
write file=c:\failed_sitelist.txt
--------
This step will be executed when the submission was detected as either failure or unknow status.

Alternatively, if there is a way to save the sites with failure or unknow submission status, that will be fine.
Please advise.

Sven · July 2017

you can do that already by using "modify step condition"

sashilover · July 2017

Thanks for your tips, I get it work partly, the only problem is "modify step condition" doesn't seem to accept the regular expression, which means I can't use "!" symbol to exclude the conditions, for example:
-----
submit success=Regardez dans votre courrier pour activer votre compte|Inregistrare completa!|Sign Up Complete!|You have successfully created your account!|To begin using this site you will need to activate your account via the email we have just sent to your address.|/members/%login%/|/members-2/%login%|>Log Out<|/my-profile/">My Profile<|egistro completado!|Rejestracja (krok 2 z 3)|Registration complete. You may now log in.|Comprueba tu correo para activar tu cuenta|Hai creato con successo il tuo account
-------
Those are the string for a successful submission, how to exclude them in the "modify step condition".
I can't seem to use:
--------
modify step condition1=!Regardez dans votre courrier pour activer votre compte
modify step condition2=!Inregistrare completa!
....
--------

Please advise.
Thanks.

Sven · July 2017

hmm there is no condition1,2...just one string "modify step condition" and you can e.g. switch it's behaviour when you modify the step once it matches and then jumps to the part where to continue the submission, else write error and jump to somewhere else

sashilover · July 2017

I know how the modify step works, the problem is the condition, how to use a condition like the page DOES NOT have the string "Regardez dans votre courrier pour activer votre compte"?

Sven · July 2017

that is not part of the syntax here, but you can switch the logic and jump if it matches no?

sashilover · July 2017

It's impossible to enumerate all the possibilities since I want to track all the unsuccessful submissions including unknown status submission...
Is it possible to add the regular expression supporting for the "modify step condition" in the near future? that will be really helpful.
Thanks.

Sven · July 2017

i will try

sashilover · July 2017

Hi Sven, how to import/export the account data with PR field? currently it seems the PR value is not able to save into the account data, I will need the format like:
--------
http://xxx.com:userID:password|PR
-------
Then I can resuse those accounts in different tier projects with PR filter enabled.
Please advise how can i do this.
Thanks.

Sven · July 2017

next update allows you to import/export with PR attached.
the format would be

URL|PR:login:password

sashilover · July 2017

thanks a lot.

sashilover · July 2017

Hi Sven, how to deal with the Buddypress site with Recaptcha/text captcha enabled in its login page? I think we need to modify the script in order to make it work(2cpatcha is enabled)? if so, how exactly to do that?
Please advise, I attached a sample site here.

sashilover · July 2017

file attached.

Sven · July 2017

should work on next update

sashilover · July 2017

Thankkkkkkkkkkkkkkkkks, Sven

sashilover · July 2017

Hi Sven, how to skip the entire [Login_Step*] after registration submission was successful, I found some sites don't need to verify or login in order to post an article after I registered an account, I tried the "submit success skip verify=> My Profile<", and "try to continue without verification=1", SER still goes to the [Login_step1]
Any tips?

Sven · July 2017

you would just put some message in the login step as "submit success" and SER will recognize that this is already successful.

sashilover · July 2017

What does this variable mean in the buddypress script, I just can't figure out it.
--

[gnr]

type=extract

front=/groups

back1=/

back2='

back3="

back4=>

default=""
---
Is it possible show me a sample of URL/site which the value of variable is not empty?

sashilover · July 2017

Hi Sven, another question for you..
1). How to handle a page with different forms that matches the same "form url" condition, the two forms have different "form name" and "form ID", but It seems SER use the "form URL" to search the forms, and use the first found one...
Is it possible to assign the priority for the "form name/form ID/form URL" when searching a form?
Ideally, SER should search "form name" first, then "form ID", then "form URL" to locate a form.
Please advise.

Sven · August 2017

gnr: it means that this variable is filled by searching on the downloaded page for "front" and it cuts all off defined in "back". If nothing is found, it is set to empty.

1) SER uses all of it, "form url", "form id" "form name" and even "form class". If all match, it is best, but not everything has to match. You can define multiple masks on it using "|". The first gets you a better ranking than later masks.

Request to update the script manual with the latest variables used in the current engine scripts

Comments