Skip to content

GSA SER still works amazingly in 2023. But pay heed to the following: - Update July 2023

Note before reading: This is not an attempt to take a cheap shot at "experts" not is it meant to be put down anyone. Everyone has their own experiences. These opinions are just my own, and I wish to share them with any people so they don't have the same doubts that I did and so that they can save their time.

Search engine marketing forums (black hat, grey hat, white hat etc) are full of alleged experts insisting that you cannot use automated tools to build your money site. I have learned this is nonsense and I wish to share something I have learned.

GSA is amazingly powerful if you know how to use it right. Even while not going nuclear, I am getting 70+ LPM a minute while targeting high-quality sites and the links are getting indexed and staying live. One myth that is common is that its easy to get google slapped for having some bad links.
It is not, otherwise think about how lucrative the market would be for taking out all the competition if it was so easy to get them banned from the search engines. It may be true that low-quality links give you zero benefit, but they are unlikely to de-index you. There is a difference between getting zero benefits from something and getting penalized for it. Get as many backlinks to your site indexed as possible and in general, the risks to your rank are to the upside.

So how can you make these backlinks get indexed and give you juice? Three mistakes people make:

1.) Emails.
2.) Content.
3.) Indexing.

Do not use generic throw-away emails. Your accounts will get banned and then all your work has been in vain. This is the most crucial part before going anywhere else. I have learned the following methods are very effective:

1.) Buying gmail accounts.
2.) Buying cheap domains and set up your own catch-all. (This is the most economical and works)
3.) Buying old yahoo accounts.

Not only do the accounts get registered more smoothly, but they do not get banned and hence you can reuse the article and social network and bookmark ones over and over. This not only saves you money on captchas but also significantly increases your LPM. Give it a try. You will see a 10X increase once you have your accounts all made.

Use at least decent quality content and you can get it indexed. It is that simple. It may be more expensive but it works. The most important thing that I have realized is the use of spinners. Spinrewriter and Word AI work wonders and have a far higher index rate than generic built-in junk which will not get indexed.

Use the filters that GSA offers to fine-tune your verified list. If you are targeting UK traffic, build your links onto UK-hosted sites. GSA gives you the option to use it. I know this sounds petty, but trust me, it makes a difference. There is a dating term that has 10x more UK traffic than the US and I have noticed a significant gain already in google.co.uk just by making this tuning. I'm sitting just outside of the top 10 and hope to break the first page for this 20k+ searched term in the next month.

Now the final part how do you Index? A link that isn't indexed has no value, we all know this song and dance. However, Indexing is harder now than in was in 2015. However, in all cases, Indexing starts with getting your link in front of the web crawlers. That is the goal of this next part.

Well, I have a really neat and almost free trick(it will cost if you use paid hosting but still be a fraction of the so-called indexing services, which do not seem to work anymore) that I have been trying for the last two weeks and it has worked more than any of the indexing services I have used. (Yes, I was bored enough and had enough free time to do multiple control testing).

Here is all you need to do:
1.) Get some hosting (it can even be free with a free subdomain).
2.) Install the free Yourls PHP script (it is a URL shortenter and redirected).
3.) Batch upload your created links into it using a custom PHP script. I'm happy to post the code I use here, if the admins allow it.
4.) Generate a sitemap from this script.
5.) Use IndexNow (which takes two minutes to set up) to batch submit the URLs. (I have noticed there is a 10k limit per submission, which you can get around by using a loop to do multiple submissions).
6.) You will see the bots crawling your URLs within minutes and following the redirects. Note, despite what people say, google always does this.


Sit back and enjoy the results. If the link created is decent enough (content is what matters more so that the domain in 95% of cases), it will be indexed within a few days.

I hope you all benefit from this post as I feel it adds way more value than all the naysayers on BHW who I am convinced now have no idea about what they are talking about in 99% of all cases.

GSA does not get enough credit, it is amazing and has:
  1. The friendly community.
  2. The constant updates.
  3. The one-off price for the software.
  4. The ease of use. It literally is like a well-oiled machine and can just keep going.

Disclaimer: I have zero affiliation with any of the products mentioned and am simply a customer/user. I am sharing this as a way of giving back to this community that has helped me a lot in the past.

Disclaimer 2: As I said, I'm happy to post open source code here for the above, but won't do so without the green light from the forum owners. I want to respect protocol.

Comments

  • edited August 2022
    Thanks for the share. I totally agree with you on all fronts, esp about the (laughable) gurus on BHW. Every single person that says SER dOeSnT wOrK has a pbn backlink service in their signature, OR is a marketplace seller. If everyone still used SER then the marketplace wouldn’t generate as much money. That’s why tools like this are frowned upon because they have turned link building into a PayPal money generator (the marketplace)

    Post the code please!
  • edited July 2023
    CODE DELETED
  • edited July 2023
    CODE DELETED
  • edited July 2023
    CODE DELETED
  • Thanks for the share. I totally agree with you on all fronts, esp about the (laughable) gurus on BHW. Every single person that says SER dOeSnT wOrK has a pbn backlink service in their signature, OR is a marketplace seller. If everyone still used SER then the marketplace wouldn’t generate as much money. That’s why tools like this are frowned upon because they have turned link building into a PayPal money generator (the marketplace)

    Post the code please!
    Gotten crawled yet?
  • Momo said:
    Gotten crawled yet?
    I have not implemented this yet. I am programming an algorithm today. I will implement this as soon as I have time. I will post when I do. Thank you for this share. It will save $!
  • I have not implemented this yet. I am programming an algorithm today. I will implement this as soon as I have time. I will post when I do. Thank you for this share. It will save $!
    If ever you wish to collaborate on any projects, please let me know. Im enjoying being back in the space and trying new things.
    Thanked by 1the_other_dude
  • In my opinion, Bing's IndexNow does not work. It can be done automatically on your own site with a plugin or Cloudflare.
    I am not able to index my sites on Bing with 1,5k words unique and well-formatted content on its homepages.
    Yes, it sends a bot, but simply refuses to index sites for weeks with weird errors such as -
    Bing Index:
    Live URL:
    Also, I noticed that Bing deindexed my old sites having backlinks and traffic when I started to make 10-20 contextual backlinks (referring domains) daily with GSA.



  • Smeklinis said:
    In my opinion, Bing's IndexNow does not work. It can be done automatically on your own site with a plugin or Cloudflare.
    I am not able to index my sites on Bing with 1,5k words unique and well-formatted content on its homepages.
    Yes, it sends a bot, but simply refuses to index sites for weeks with weird errors such as -
    Bing Index:
    Live URL:
    Also, I noticed that Bing deindexed my old sites having backlinks and traffic when I started to make 10-20 contextual backlinks (referring domains) daily with GSA.



    From the report above it seems their crawler cannot access your site. I'm sure it is not a block by robots.txt or something that silly as I've read some of your comments, and you do know what you are doing. What server setup are you using? I once had an issue with the fail2ban plugin on Plesk blocking search engine crawlers.
  • Momo said:
    From the report above it seems their crawler cannot access your site. I'm sure it is not a block by robots.txt or something that silly as I've read some of your comments, and you do know what you are doing. What server setup are you using? I once had an issue with the fail2ban plugin on Plesk blocking search engine crawlers.
    Server is fine and I am getting Bing bot visits everyday with 200 OK in the server's log file on the current domain. I am not blocking Bing. As I heard Bing something f*cked up from their side about 3 months ago.
    In the Site Explorer I can see this:
    The date of Last crawled say that site was even not launched on that time.
    But as I said before, Bing bot visits the site everyday.

    Saw on BHW many threads that after some changes on Bing (3 months ago) everyone has trouble indexing on it.

  • Smeklinis said:
    Server is fine and I am getting Bing bot visits everyday with 200 OK in the server's log file on the current domain. I am not blocking Bing. As I heard Bing something f*cked up from their side about 3 months ago.
    In the Site Explorer I can see this:
    The date of Last crawled say that site was even not launched on that time.
    But as I said before, Bing bot visits the site everyday.

    Saw on BHW many threads that after some changes on Bing (3 months ago) everyone has trouble indexing on it.

    That is strange. I have been indexing new pages with them all week long, even with a domain that was registered on 21 July 2022.
  • Momo said:
    If ever you wish to collaborate on any projects, please let me know. Im enjoying being back in the space and trying new things.
    Glad to hear you’re doing things that you enjoy. If I can think of anything in the future I’ll be sure to drop a line about collab.
  • Thanks for this encouraging thread 
    i'm back to GSA since a while 
    i just followed your steps 
    I hope it work 

    for the bulk upload 
    i just found a plugin you can add to your Yourls website : 
    https://github.com/vaughany/yourls-bulk-import-and-shorten

    Just to confirm you're building Tier 1 with GSA to your money site directly ? 
    Did you use any verified list service or you are crawling your own ? 

    Thanks 
  • "Just to confirm you're building Tier 1 with GSA to your money site directly ? "

    I am.

    "Did you use any verified list service or you are crawling your own ?"

    Combination of both (most bought) as I found scraping to not be worth it. I then import lists into each project tier. I break down which sites I choose after using Scrapebox's page authority checker. 
    Thanked by 1hardcorenuker
  • edited August 2022
    @Momo

    Thanks for openly sharing your experience !

    From what I see (in indexnow.org) Google is not supporting indexnow ?

  • coral99 said:
    @Momo

    Thanks for openly sharing your experience !

    From what I see (in indexnow.org) Google is not supporting indexnow ?

    As of November 2021, rumours started that they were experimenting with it. I also see their crawlers (mobile versions) crawling links that I have submitted in such a way. But none of that matters.

     Most importantly, I see the pages submitted in such a way indexed by google and that's the most important thing. Costs literally nothing and is 1,580% better, as per my tests, than services that charge people hundreds to thousands of dollars per yet for junk that hasn't worked for years. Naturally none of them will admit it, they earn their bread and butter of such ignorance, like the same people on BHW who say GSA SER does not work, yet sell services for hundreds of dollars per month by using their GSA SER programs to build links.....
    Thanked by 1coral99
  • Momo said:
    As of November 2021, rumours started that they were experimenting with it. I also see their crawlers (mobile versions) crawling links that I have submitted in such a way. But none of that matters.

     Most importantly, I see the pages submitted in such a way indexed by google and that's the most important thing. Costs literally nothing and is 1,580% better, as per my tests, than services that charge people hundreds to thousands of dollars per yet for junk that hasn't worked for years. Naturally none of them will admit it, they earn their bread and butter of such ignorance, like the same people on BHW who say GSA SER does not work, yet sell services for hundreds of dollars per month by using their GSA SER programs to build links.....
    Thanks, I'll check this out and report
  • Momo said:
    "Just to confirm you're building Tier 1 with GSA to your money site directly ? "

    I am.

    "Did you use any verified list service or you are crawling your own ?"

    Combination of both (most bought) as I found scraping to not be worth it. I then import lists into each project tier. I break down which sites I choose after using Scrapebox's page authority checker. 
    Do you mind sharing the engine types you use for Tier 1 2 and 3? :)
  • Do you have any feedback on the indexing after your test @the_other_dude
  • May i know which hosting company is the best (in terms of value for money) to catch our own catchall email accounts?
  • May i know which hosting company is the best (in terms of value for money) to catch our own catchall email accounts?
    You don't need a powerful VPS for such, a small and cheap machine will do. I am running several own and customer catchall accounts on a small dedicated VPS and it is stable and performant.
  • You don't need a powerful VPS for such, a small and cheap machine will do. I am running several own and customer catchall accounts on a small dedicated VPS and it is stable and performant.
    what software do you run off a vps for the email server?
  • what software do you run off a vps for the email server?
    I use cheap cpanel shared hosting from BHW for catchall. It’s about $10 a year, unlimited domains :)

  • Thanks for sharing can you give us what is the service at 10 usd per year please? @the_other_dude
  • googlealchemistgooglealchemist Anywhere I want
    Momo said:
    Now that we finally have the links in the database with the redirects created and the sitemap made, we can summon the crawlers. For this, you first need to go here: https://www.bing.com/indexnow#implementation
    Generate an API, download the text file and upload it to your hosting. Make a note of the name.
    Now you need to create and run the following in python (.py file). (if you don't have it, install pycharm community edition which is free). Not this is a basic script for one sitemap but you can easily use it to loop through and submit multiple sitemaps. If there is enough interest, ill make such a script and release it.
    import requests
    import advertools as adv
    
    
    key = "12367346914527865247865429" #this is the key you generated
    keyLocation = "https://mysite.com/12367346914527865247865429.txt" #this is where you uploaded it
    
    list = ["https://mysite.com/sitemap.php"] #this is where your sitemap is
    
    for sitemap in list:
        sitemap_urls = adv.sitemap_to_df(sitemap)
        urls = sitemap_urls["loc"].to_list()
        #urls = sitemap_urls["URL"].to_list()
    
        host = 'https://www.bing.com/indexnow'
        website = 'https://mysite.com'
    
        headers = {"Content-type": "application/json", "charset": "utf-8"}
        url = host
        myobj = {
                "host": website,
                "key": key,
                "keyLocation": keyLocation,
                "urlList": urls
            }
        x = requests.post(url, json=myobj, headers=headers)
        print(x.status_code)

    If done correctly, the code will return 200 Status and you will see the search engine bots (Bing, Yandex, Google) visiting your robots.txt files and submitted URLs in less than a minute. Hope you all benefited from this. Please reach out if you have any questions.
    thanks for the post

    i get lost with any sort of coding/db stuff

    whats the point of putting all the links into the redirect thing vs just posting them all in a blog post to get that post into the sitemap and using that sitemap to submit to indexnow?
  • Momo said:
    Now that we finally have the links in the database with the redirects created and the sitemap made, we can summon the crawlers. For this, you first need to go here: https://www.bing.com/indexnow#implementation
    Generate an API, download the text file and upload it to your hosting. Make a note of the name.
    Now you need to create and run the following in python (.py file). (if you don't have it, install pycharm community edition which is free). Not this is a basic script for one sitemap but you can easily use it to loop through and submit multiple sitemaps. If there is enough interest, ill make such a script and release it.
    import requests
    import advertools as adv
    
    
    key = "12367346914527865247865429" #this is the key you generated
    keyLocation = "https://mysite.com/12367346914527865247865429.txt" #this is where you uploaded it
    
    list = ["https://mysite.com/sitemap.php"] #this is where your sitemap is
    
    for sitemap in list:
        sitemap_urls = adv.sitemap_to_df(sitemap)
        urls = sitemap_urls["loc"].to_list()
        #urls = sitemap_urls["URL"].to_list()
    
        host = 'https://www.bing.com/indexnow'
        website = 'https://mysite.com'
    
        headers = {"Content-type": "application/json", "charset": "utf-8"}
        url = host
        myobj = {
                "host": website,
                "key": key,
                "keyLocation": keyLocation,
                "urlList": urls
            }
        x = requests.post(url, json=myobj, headers=headers)
        print(x.status_code)

    If done correctly, the code will return 200 Status and you will see the search engine bots (Bing, Yandex, Google) visiting your robots.txt files and submitted URLs in less than a minute. Hope you all benefited from this. Please reach out if you have any questions.

    Thank you so much for sharing! Loop code would be very helpful
  • about
    "DISCOVERED BUT NOT CRAWLED"

    i didnt read all of the other information that you had to go with this on purpose, because with discovered but not crawled the info bing & google gives is sometimes contradictory, so you will be tearing your hair out trying to resolve it based on what you see in front of you, when it is not always accurate, but i have had "DISCOVERED BUT NOT CRAWLED" & it was because of semi duplicate content, i kept taking out some of the duplicate content but it was not indexing the pages, it was only when i took out all of the duplicate content that this changed & got the rest of the pages that werent duplicate content indexed.
    in google search console it will give you various data/messages etc, but some of these are not accurate at all, so if you have the "DISCOVERED BUT NOT CRAWLED" message, see if you think that google/bing might be seeing some (ANY) pages might be seen as duplicate content or not good enough in googles eyes, because it wont just stop it indexing those pages, it might stop google/bing indexing the rest of the pages, i dont know if this helps, but any other messages you see might not be accurate/relevant, i had that (contradictory messages) but when i removed all of the duplicate pages it started to index the non duplicate pages as well.

    another thing, i think it depends what market you are in & how big the market is, smaller, less competitive markets you can get away with more, such as local markets, but if you apply the same things to national/international sites sometimes google does not index the pages.


  • Momo said:
    Now that we finally have the links in the database with the redirects created and the sitemap made, we can summon the crawlers. For this, you first need to go here: https://www.bing.com/indexnow#implementation
    Generate an API, download the text file and upload it to your hosting. Make a note of the name.
    Now you need to create and run the following in python (.py file). (if you don't have it, install pycharm community edition which is free). Not this is a basic script for one sitemap but you can easily use it to loop through and submit multiple sitemaps. If there is enough interest, ill make such a script and release it.

    If done correctly, the code will return 200 Status and you will see the search engine bots (Bing, Yandex, Google) visiting your robots.txt files and submitted URLs in less than a minute. Hope you all benefited from this. Please reach out if you have any questions.
    @momo Are you actually seeing any evidence from Bing and/or Google?

    As far as my installation is concerned, only Yandex has crawled in the last several days.
    There's been no Bing or Google - and from what research shows Google is not a participant in Index now - only Microsoft Bing, Yandex, and Seznam.
  • KaineKaine thebestindexer.com
    edited December 2022
    Interesting thread, if you are interested we index on Google, Bing and Yandex simultaneously and our formulas are unlimited.
    We are going to open a thread with 1 day trial offered on TheBestIndexer.com, free for all members of the GSA forum to celebrate the happy new year.
    This will be effective today or tomorrow, stay around to enjoy it!
  • Kaine said:
    Interesting thread, if you are interested we index on Google, Bing and Yandex simultaneously and our formulas are unlimited.
    We are going to open a thread with 1 day trial offered on TheBestIndexer.com, free for all members of the GSA forum to celebrate the happy new year.
    This will be effective today or tomorrow, stay around to enjoy it!
    That is great, yet another thread hijack attempt to promote your services or lists, when people are discussing their own solutions. You are really adding value..... Is anyone else getting bored of this type of behaviour?

    rastarr said:
    @momo Are you actually seeing any evidence from Bing and/or Google?

    As far as my installation is concerned, only Yandex has crawled in the last several days.
    There's been no Bing or Google - and from what research shows Google is not a participant in Index now - only Microsoft Bing, Yandex, and Seznam.
    Would like to show you something:



    Here is a site, I started juicing with GSA and using this 301 redirection sitemap method in the second week of May. Already the Googlebot has discovered 140k+ links and has crawled them as directed by the 301. Now some may ask why does it not index on my domain, well the reason is simple:

    The sitemap is there to feed Tier links to the google bot. It then crawls them and indexes them. Magic, free and easy, as you can see below.



    Using this method you can literally throw more new links in front of the SERPS than the competition. Already have some top 10s in 100k monthly volume competitive niches, competing with the "Kings" who have been there 20+ years, and all in a few months.

    I hope to soon combine this with a Scraper I am building that will target platforms only used by GSA SER and then rank them based on social signals. Picking the best of these, I hope to throw out millions of links per month in lots of new niches and see how far I can go.

    Core message: Don't listen to people who spread fear and doubt. GSA works very well and all you need is GSA SER, A good list to link build on and then to use the above free script I made to get those links in front of the search engine bots to be indexed. It really is that simple (assuming you actually have content on your site). Don't be discouraged by the average black hat expert on certain forums, they literally have no idea what they are talking about and earn bread and milk money hustling fiver gigs with scrapebox and senuke spam. Took me ages to accept that fact. Once I did, I'm getting results.
  • I am new on this forum since today.   I really want to try this.   But how many links you can build daily to you're moneysite?

    I don't have a paid version of GSA, is it still working in 2023?
  • rastarrrastarr Thailand
    Momo said:

    rastarr said:
    Momo said:
    Now that we finally have the links in the database with the redirects created and the sitemap made, we can summon the crawlers. For this, you first need to go here: https://www.bing.com/indexnow#implementation
    Generate an API, download the text file and upload it to your hosting. Make a note of the name.
    Now you need to create and run the following in python (.py file). (if you don't have it, install pycharm community edition which is free). Not this is a basic script for one sitemap but you can easily use it to loop through and submit multiple sitemaps. If there is enough interest, ill make such a script and release it.

    If done correctly, the code will return 200 Status and you will see the search engine bots (Bing, Yandex, Google) visiting your robots.txt files and submitted URLs in less than a minute. Hope you all benefited from this. Please reach out if you have any questions.
    @momo Are you actually seeing any evidence from Bing and/or Google?

    As far as my installation is concerned, only Yandex has crawled in the last several days.
    There's been no Bing or Google - and from what research shows Google is not a participant in Index now - only Microsoft Bing, Yandex, and Seznam.
    Would like to show you something:

    Here is a site, I started juicing with GSA and using this 301 redirection sitemap method in the second week of May. Already the Googlebot has discovered 140k+ links and has crawled them as directed by the 301. Now some may ask why does it not index on my domain, well the reason is simple:

    The sitemap is there to feed Tier links to the google bot. It then crawls them and indexes them. Magic, free and easy, as you can see below.



    Using this method you can literally throw more new links in front of the SERPS than the competition. Already have some top 10s in 100k monthly volume competitive niches, competing with the "Kings" who have been there 20+ years, and all in a few months.

    I hope to soon combine this with a Scraper I am building that will target platforms only used by GSA SER and then rank them based on social signals. Picking the best of these, I hope to throw out millions of links per month in lots of new niches and see how far I can go.

    Core message: Don't listen to people who spread fear and doubt. GSA works very well and all you need is GSA SER, A good list to link build on and then to use the above free script I made to get those links in front of the search engine bots to be indexed. It really is that simple (assuming you actually have content on your site). Don't be discouraged by the average black hat expert on certain forums, they literally have no idea what they are talking about and earn bread and milk money hustling fiver gigs with scrapebox and senuke spam. Took me ages to accept that fact. Once I did, I'm getting results.
    Excellent and thanks for the update. great to see this is working for you.
    My installation is still running.
    There's very very little Googlebot activity on my installation.

    [1] Have you made any changes to your original script though?
    [2] How are you getting Google to crawl your Yourls installation so well?
    [3] Have you changed the script to include multiple sitemaps? I see mention of multiples in your screenshot, is all.

    I'd love to get these same results. It sounds like you've made changes since your original post, a year ago though.
  • I've just tried to run the scripts but both and are giving errors:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    Anyone having success and/or suggestions?
  • I've just tried to run the scripts but both and are giving errors:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    Anyone having success and/or suggestions?
    Regarding:

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    You didn't add the row column to the SQL database as instructed above. Please see the very fist sentence written:

    "The first thing you will need to do is edit the database table for Yourls, this one 'yourls_url' so it has the following structure"

    Regarding:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    Thanks for pointing out. It is missing a comma. Have updated.

    Good luck!
  • Momo said:
    I've just tried to run the scripts but both and are giving errors:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    Anyone having success and/or suggestions?
    Regarding:

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    You didn't add the row column to the SQL database as instructed above. Please see the very fist sentence written:

    "The first thing you will need to do is edit the database table for Yourls, this one 'yourls_url' so it has the following structure"

    Regarding:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    Thanks for pointing out. It is missing a comma. Have updated.

    Good luck!
    Thanks. I also spotted some differences in the DB settings (dbpassword / password).

    Sitemap has been submitted, so I am waiting for the crawlers now :)
  • Momo said:
    I've just tried to run the scripts but both and are giving errors:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    Anyone having success and/or suggestions?
    Regarding:

    [Tue Jul 04 06:41:47.234973 2023] [php:error] [pid 5548] [client 95.91.221.5:7128] PHP Fatal error:  Uncaught mysqli_sql_exception: Unknown column 'row' in 'order clause' in /var/www/html/sitemap.php:13\nStack trace:\n#0 /var/www/html/sitemap.php(13): mysqli->query()\n#1 {main}\n  thrown in /var/www/html/sitemap.php on line 13

    You didn't add the row column to the SQL database as instructed above. Please see the very fist sentence written:

    "The first thing you will need to do is edit the database table for Yourls, this one 'yourls_url' so it has the following structure"

    Regarding:

    [Tue Jul 04 06:41:41.039095 2023] [php:error] [pid 5550] [client 95.91.221.5:7164] PHP Parse error:  syntax error, unexpected variable "$line", expecting ")" in /var/www/html/crawlme.php on line 47

    Thanks for pointing out. It is missing a comma. Have updated.

    Good luck!
    Thanks. I also spotted some differences in the DB settings (dbpassword / password).

    Sitemap has been submitted, so I am waiting for the crawlers now :)
    That is great. Glad you got it working. Good luck!
  • hi,
    thanks for all the work. I do have a couple of questions if you could help.
    do you submit every link  in the database or just the new additions when using the python script to send new links to indexnow
    right now i set it up to only send the new entries by adding a where row > x in the dynamic sitemap generator php script
    i also made a robots.txt file that refers to a full sitemap.xml holding every entry in the database
    python doesnt work on my hosting so i run the php scripts on the hosting but run the python script on another computer.
    and i am getting the 200 success response.
    does what i am doing sound reasonable,
    thanks again,
    george
  • hi,
    one other question, do you open a google webmaster account on the site hosting the redirects and then submit that full sitemap?
    thanks again,
    george
  • hi,
    thanks for all the work. I do have a couple of questions if you could help.
    do you submit every link  in the database or just the new additions when using the python script to send new links to indexnow
    right now i set it up to only send the new entries by adding a where row > x in the dynamic sitemap generator php script
    i also made a robots.txt file that refers to a full sitemap.xml holding every entry in the database
    python doesnt work on my hosting so i run the php scripts on the hosting but run the python script on another computer.
    and i am getting the 200 success response.
    does what i am doing sound reasonable,
    thanks again,
    george
    Hello George. I just submit the sitemaps. I found it works and it saves AGES of time.

    Regarding a full stiemap.xml bear in mind that search engines bots stop reading a sitemap after a certain size. So you are better off dividing into multiple smaller ones when you have tens of millions of links. I usually go with 5000 mapped to another 5000 and mapped to another 5000. I found that this works.

    If you are getting the 200 success response, it means that the submission worked well. The best way to verify would be via your logs though. Look for evidence of the crawlers coming.
  • edited July 2023
    CODE DELETED
  • thank you so much,
    i will drop the webmaster account,
    i really appreciate your response,
    thanks,
    george
  • rastarrrastarr Thailand
    @Momo said:
    Hello George. I just submit the sitemaps. I found it works and it saves AGES of time.

    Regarding a full stiemap.xml bear in mind that search engines bots stop reading a sitemap after a certain size. So you are better off dividing into multiple smaller ones when you have tens of millions of links. I usually go with 5000 mapped to another 5000 and mapped to another 5000. I found that this works.

    If you are getting the 200 success response, it means that the submission worked well. The best way to verify would be via your logs though. Look for evidence of the crawlers coming.
    Would it be possible for you to re-post your Yourls script, as it is today? While I have some rudimentary Python skills ( https://forum.gsa-online.de/discussion/30601/free-recaptcha-v2-and-v3-breaker#latest ), getting your working multiple 5000 entry sitemap stuff would be a great help since I found a few bugs in your original stuff.
  • edited July 2023
    CODE DELETED
  • rastarrrastarr Thailand
    Momo said:

    I don't do it in Yourls, thats purely server side. I use a different python script for it locally. For example, the following method will suffice if you have 25m links. (5000*5000). Then you can zip the files. Upload them to a directory on your server (e.g. httpsdocs/sitemaps1/) and unzip them. This will save your database from constantly being smashed every time the bots crawl (and they crawl A LOT). It also means you save costs on hosting as you wont need a beast SQL server. The method is very simple.


    #here you call the method with the textfile (list of links to index) and domain (e.g. example.com)
    def createXMLSitemap(textfile, domain):
        urls = []
        with open(textfile) as f:
            for line in f:
                line = line.strip('\n')
                urls.append(line)
        numberToDo = len(urls)/5000
        print("Sitemaps to make: " + str(row_count))
        print("Sitemaps to make: " + str(numberToDo))
        x = 1
        while x - 1 < numberToDo:
            with open("sitemap_" + str(x) + ".xml", 'w') as fp:
                fp.writelines('<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">')
                for link in urls:
                    fp.writelines('<url>')
                    fp.writelines('<loc>'+ link + '</loc>')
                    fp.writelines('<lastmod>2023-07-08T15:15:27+00:00</lastmod>')
                    fp.writelines('<changefreq>Daily</changefreq>')
                    fp.writelines('<priority>0.9</priority>')
                    fp.writelines('</url>')
                fp.writelines('</urlset>')
            x = x + 1
        x = 1
        with open("sitemap_index.xml", 'w') as fp:
            fp.writelines('<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">')
            while x - 1 < numberToDo:
                fp.writelines('<url>')
                fp.writelines('<loc>https://' + domain  + '/sitemaps_1/sitemap_' + str(x) + '.xml</loc>')
                fp.writelines('<lastmod>2023-07-08T15:15:27+00:00</lastmod>')
                fp.writelines('<changefreq>Daily</changefreq>')
                fp.writelines('<priority>0.9</priority>')
                fp.writelines('</url>')
                x = x + 1
            fp.writelines('</urlset>')


    Calling the method as an example:
    textfile = "myT2urls.txt"
    domain= "example.com"
    createXMLSitemap(textfile, domain)

    depending on your PC specs, you may get memory errors if the size of the list is too big. If that is the case, split the list up and call the method a few times, adjusting x as appropriate.
    So you don't actually shorten the URLs, if I'm reading this new code correctly. Right?
    You are just filling the sitemap with links from another domain.
    Won't they get ignored by Google if the sitemap contains non-domain links?
  • No. I shorten them with Yourls. This is how I generate the sitemaps. Hence why you need to enter your domain and the shortened urls created.
  • I think it would be better for everyone if I just release a python package for this. That way, everyone can benefit from it with minimal confusion. Will write one later today/tomorrow when I have an hour to spare.
  • One suggestion I have is if you are needing help with code to ask chatgpt to make it for you. Or show it the code you’re having problems with and give it the error. I’ve generated a bunch of code like this I wouldn’t have been able to make myself.
  • Ok. Released a python package to literally make it easy for even newbies to understand and use. Very simple to use.

    Requirements:
    Python.

    Installation:
    pip install momositemaps

    Code:

    import momositemaps as sitemapmaker

    host = ""  # Database Host
    user = ""  # Database Username
    password = ""  # Database Username
    database = ""  # Database Username
    domain = ""  # Domain hosting the sitemaps
    directory = ""  # Directory of maps on domain (folder where you will upload the sitemaps)
    links = "links.txt"  # List of links to be submitted then made into sitemaps

    # This method will add all the links to the database and then recreate all sitemaps
    sitemapmaker.submitAndBuildSitemaps(host, user, password, database, domain, directory, links)


    Enjoy everyone.


  • Thank you for contributing man. I was reading forum for few years and just registered to say thank you! Best post here for last few years for sure.
  • Jsilva said:
    Thank you for contributing man. I was reading forum for few years and just registered to say thank you! Best post here for last few years for sure.
    Thank you. Really happy to hear this. You made my day. I wish you success sir!
  • Hey Momo, thanks for updating this with more valuable information. I am setting up catch all email domains now and getting ready to use SER after a long break.
    For some reason I though that google has made it so we cant use SER with gmail a year or so ago..? Gmails are working again?
  • I haven't had any issues with them, but actually started just using my own hosted catchalls a while back, it was more economic.
    It could be possible that google did something for newer accounts or accounts which aren't verified by mobile. I always buy aged accounts for things like twitter, gmail etc. I find they raise less flags.
  • Momo said:
    I haven't had any issues with them, but actually started just using my own hosted catchalls a while back, it was more economic.
    It could be possible that google did something for newer accounts or accounts which aren't verified by mobile. I always buy aged accounts for things like twitter, gmail etc. I find they raise less flags.
    I agree with catchalls being more economical. That’s all I’ve used for years is my own catchalls. I’ll just stick with that since my google account are expensive and I don’t want to get them banned or something. Thanks.
  • Thanks for this thread
  • Have deleted the repository and code. This has been abused by unsavory people on BHW to sell services to others. That is 100% against my intentions when I made this freely available and I want to stop it getting any worse. I hope you all understand my wishes in this regard.
  • I hate people they ruin it for everyone lol. 

    I read this post a few days ago and sat tonight to get this sorted as indexing was a problemo for me. 

    Would appreciate a PM witht he code @Momo, in advanced thanks for all the knowledge sharing!

  • rastarrrastarr Thailand
    Momo said:
    Have deleted the repository and code. This has been abused by unsavory people on BHW to sell services to others. That is 100% against my intentions when I made this freely available and I want to stop it getting any worse. I hope you all understand my wishes in this regard.
    wait, what? I'm so glad I got this working yesterday, after some code modification to get it working with Yourls v1.91.
    There will always be some people who spoil it for others.
    I've certainly appreciated your contributions @Momo
    btw if you want the modifications I made to your code, please ask. Happy to share those back to you
  • cherubcherub SERnuke.com
    Which service was it? So I know not to give them any business (I've probably tried every single indexing service out there)
  • thanks again, i have been trying to crack the indexing problem all year and you started me off on the right track. great post! btw, google cloud has an indexing api meant for job posting that ties into the webmaster console. eg. if the yourls shortner is on a site and that site is coupled into the webmaster console you can submit links that you dont own which are shortened, but this adds another level of complication.
    thank you again!
  • This is pretty great, I have taken @Momo 's path and successfully got this working....I am planning on adding the google indexing API with this and have it run continuously and submit 200 links a day. 



Sign In or Register to comment.