Skip to content

Check if email verification is working

bradmcabradmca
in GSA Search Engine Ranker
Hi,

What's the nice quick way to be sure that my catchall gmail account is being logged into and confirmation links are being processed?

I seem to have a lower verification rate and a large inbox with emails going beyond by "delete message if older than" duration.
Thanked by 2Micheleappar CherylPaits

Comments

  • SvenSven www.GSA-Online.de
    edited May 2014
    turn the project to "Active (e-mail verify only)" and check the log about what emails get parsed and what urls get found.
  • bradmcabradmca
    Did that, and as feared although the log went through looking fine, I still have a couple of thousand emails in the inbox with confirmation links in them unread...

    They all have the structure %spinfile-names.dat%{.|-|_}%spinfile-lnames.dat%{%random-1-100%|}@domainB.com

    I login to my gmail catchall account which is brad@domainA.com and see an inbox with 3364 unread emails, lets say 10% are junk, there are still around 60% with emails that when I click the approve link it approves me for the first time (I know some would say the account has already been approved but none of these do)
  • bradmcabradmca
    edited May 2014
    The times of the messages matches the times these projects are run, an example of an email is as below.


    To: stan-mcclelland27@domainB.com

    Benvenuto su Trisciuoglio.it!

    Benvenuto su Trisciuoglio.it! Ti sei registrato comunicando questa casella di posta: (stan-mcclelland27@domainB.com) per registrare un account su Trisciuoglio.it.

     Per rendere effettiva la registrazione devi obbligatoriamente cliccare sul seguente link ENTRO 24 ORE!

     http://www.trisciuoglio.it/modules.php?name=Your_Account&op=activate&username=SZ65&check_num=6e28719bc07addba6e6612c1d8f256a3

     Seguono le informazioni utente:

    -Nickname: SZ65
    -Password: FxOrP6Db
  • SvenSven www.GSA-Online.de
    is the log saying anything about found urls?
  • bradmcabradmca
    Sample log output

    20:42:58: Setting up project
    20:42:58: Setting up project
    20:42:59: Starting project
    20:42:59: Starting project
    20:42:59: [ ] Verifying links...
    20:42:59: [ ] Verifying links...
    20:43:01: [ ] Parsing 2 E-Mails...
    20:43:01: [ ] Parsing 2 E-Mails...
    20:43:01: [ ] E-Mails parsed with 11 URLs
    20:43:01: [ ] E-Mails parsed with 11 URLs
    20:43:02: [-] Verification (round 1) of DokuWiki not successful->removed - http://andrewpuschak.com/dokuwiki/doku.php?do=register
    20:43:02: [-] Verification (round 1) of Laconica not successful->removed - http://datnewcudi.us6.list-manage1.com/subscribe/
    20:43:02: [-] Verification (round 1) of WebProNews not successful->removed - http://directory.managernewz.com/landing.html
    20:43:02: [-] Verification (round 1) of FireBoard not successful->removed - http://ksap.sp.cs.cmu.edu/html/Joomla/index.php?option=com_fireboard&Itemid=6&func=showcat&catid=44
    20:43:02: [-] Verification (round 1) of XenForo not successful->removed - http://pokemon-online.eu/login/
    20:43:02: [-] Verification (round 1) of FireBoard not successful->removed - http://thevillagescrapper.co.za/index.php?option=com_fireboard&Itemid=5&id=1068&catid=7&func=fb_pdf
    20:43:02: [-] Verification (round 1) of MediaWiki not successful->removed - http://wiki.bibalex.org/DAFWiki/index.php?title=Special:Userlogin&action=submitlogin&type=signup&returnto=Forum
    20:43:02: [-] Verification (round 1) of WikkaWiki not successful->removed - http://wiki.macbre.net/UserSettings
    20:43:02: [-] Verification (round 1) of DokuWiki not successful->removed - http://wiki.piumalinux.org/doku.php?id=art&do=register
    20:43:02: [-] Verification (round 1) of Vanilla not successful->removed - http://www.bluecafe.com/index.php?p=/entry/register
    20:43:02: [-] Verification (round 1) of GeekLog not successful->removed - http://www.russianstove.com/brickyard/users.php?mode=new
    20:43:02: [-] Verification (round 1) of Coppermine Photo Gallery-login not successful->removed - http://www.snowmobileguys.com/sggallery/displayimage.php?pos=-616
    20:43:02: [-] Verification (round 1) of phpLinks not successful->removed - http://eheli.at/phplinks/index.php?show=new
    20:43:02: [ ] E-Mail Verification finished.
    20:43:02: [-] Verification (round 1) of FireBoard not successful->removed - http://www.agapeatp.com/hk/index.php?option=com_fireboard
    20:43:02: [-] Verification (round 1) of Freeglobes not successful->removed - http://www.annu-blogs.fr/nouveautes.html
    20:43:02: [-] Verification (round 1) of eSyndiCat not successful->removed - http://www.bowhuntinglinks.com/suggest-listing.php?id=0
    20:43:02: [-] Verification (round 1) of FireBoard not successful->removed - http://www.ptsm.szczecin.pl/index.php?option=com_fireboard&Itemid=50
    20:43:02: [-] Verification (round 1) of LINKER not successful->removed - http://www.wrldsites.com/add.php?sid=
    20:43:02: [ ] E-Mail Verification finished.
    20:43:02: [ ] Verifying finished (52 checks done)
    20:43:02: [ ] Verifying finished (58 checks done)
  • bradmcabradmca
    solved... 
  • SvenSven www.GSA-Online.de
    let us know how? ;)
  • bradmcabradmca
    edited May 2014
    Well it was sort of solved... now not so I think

    First test, make a pop client with OpenPop.NET and see that although POP was logging in, no messages were downloaded (inbox count = 0)

    So I had the gmail pop settings stating that "it's on for all pop since years ago", but no radio buttons checked so I put it to access all even ones already downloaded, suddenly 289 messages were downloaded (although there were 1500ish in the inbox but whatever... I had clicked 500 links manually by then and wanted a way out..)

    Today though with another 800 emails in the inbox, for GSA is as it was at first, and my test pop client has an inbox count of 0 again :(

    So will try make an IMAP client this morning to just loop through mails (if viewable...) and just click to every link found in the message body and disregard the response...

    QUESTION: will me having a separate email parse utility cause the links to be verified in GSA still? - that's what I am after is a high # of verified per day..
  • SvenSven www.GSA-Online.de
    where just an email link is to be clicked yes, for those where login/password has to be extracted not.
  • bradmcabradmca
    hmmm could cause some problems then, but I guess I would rather have some download + "click" activity on that catchall mailbox than none.

    any idea why gmail with an inbox of 800+ would report 0 to a pop client?

    Outlook + thunderbird can view the mailbox messages
    GSA + openpop see an empty mailbox

    Must be a logical reason.
  • SvenSven www.GSA-Online.de
    hmm no idea really. but nice you can debug it that way. I would be interested if you find out more. Maybe the emails are in some spam folder thats not accessable by pop3?
  • bradmcabradmca
    nope, straightforward inbox label.

    need to get a process working today, if i discover any gotchas willl add them here.
  • bradmcabradmca
    Same pop client on my non catchall account... sees emails... but for catchall account, no.

    Question: what does the dialogue box on SER "Is this a Catch-All Email?" Y/N do, if anything?
  • SvenSven www.GSA-Online.de
    it just changes the email with spins so it can use them for webforms and have a random one.
  • bradmcabradmca
    If I use an IMAP client, I get more emails than the web gmail reports but that might be due to conversations or whatever... either way they are downloadable from the catchall account

    I have a console app now with MailKit library and will just put in Html Agility Pack to parse the links, should save me some eyeballs and fingers.

    One observation is that where the inbox count for imap = > 0, the "recent" count = 0....
  • bradmcabradmca
    This eases my email verification pain

    using System;
    using System.Linq;
    using System.Net;
    using System.Text.RegularExpressions;
    using System.Threading;
    using MailKit;
    using MailKit.Net.Imap;
    using MailKit.Search;
    using MimeKit;

    namespace GmailImapClient
    {
        class Program
        {
            public static void Main(string[] args)
            {
                using (var client = new ImapClient())
                {
                    var credentials = new NetworkCredential("login@domain.com", "mailpassword");
                    var uri = new Uri("imaps://imap.gmail.com");

                    using (var cancel = new CancellationTokenSource())
                    {
                        client.Connect(uri, cancel.Token);

                        // Note: since we don't have an OAuth2 token, disable
                        // the XOAUTH2 authentication mechanism.
                        client.AuthenticationMechanisms.Remove("XOAUTH");

                        client.Authenticate(credentials, cancel.Token);

                        // The Inbox folder is always available on all IMAP servers...
                        var inbox = client.Inbox;
                        var trash = client.GetFolder(SpecialFolder.Trash);

                        inbox.Open(FolderAccess.ReadWrite, cancel.Token);

                        Console.WriteLine("Total messages: {0}", inbox.Count);

                        var query = SearchQuery.DeliveredAfter(DateTime.Parse("2013-01-12"));

                        foreach (var uid in inbox.Search(query, cancel.Token))
                        {
                            var delete = false;
                            var message = inbox.GetMessage(uid, cancel.Token);
                            Console.WriteLine("[match] {0}: {1} : {2}", uid, message.Subject, message.Date);
                            
                            var body = message.BodyParts.OfType<TextPart>().FirstOrDefault();

                            if (body != null)
                            {
                                if (body.Text != "")
                                {

                                    //Find suitable confirmation links and "click" them
                                    Regex linkParser = new Regex(@\b(?:https?://|www\.)\S+\b, RegexOptions.Compiled | RegexOptions.IgnoreCase);
                                    foreach (Match m in linkParser.Matches(body.Text))
                                    {
                                        if ((m.Value.Contains("onfirm")) || (m.Value.Contains("ctivate")))
                                        {
                                            Console.WriteLine(m.Value);
                                            GetPage(m.Value);
                                            delete = true;
                                        }
                                    }

                                }
                            }

                            if (delete == true)
                            {

                                //inbox.MoveTo(new UniqueId[] { uid }, trash);
                                inbox.AddFlags(new UniqueId[]{uid}, MessageFlags.Deleted, true, cancel.Token);
                                inbox.Expunge(cancel.Token);
                            }
                            //Console.ReadLine();
                        }



                        client.Disconnect(true, cancel.Token);
                    }
                }
            }

            static void GetPage(string url)
            {
                // Create web client simulating IE6.
                using (WebClient client = new WebClient())
                {
                    client.Headers["User-Agent"] =
                    "Mozilla/4.0 (Compatible; Windows NT 5.1; MSIE 6.0) " +
                    "(compatible; MSIE 6.0; Windows NT 5.1; " +
                    ".NET CLR 1.1.4322; .NET CLR 2.0.50727)";

                    // Download data.
                    try
                    {
                        byte[] arr = client.DownloadData(url);
                        Console.WriteLine(arr.Length);
                    }
                    catch (Exception exception)
                    {
                        Console.WriteLine(exception.Message);
                    }
                    
                }
            }

        }
    }
  • bradmcabradmca
    My final solution if anyone else wants to clear their catchall accounts better



    Thanked by 1Digitalw0
Sign In or Register to comment.