Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

PHP Codes to removing duplicate url from one list compared to other list

boilerboiler Indonesia
edited December 2020 in GSA Search Engine Ranker
First i was trying to removing same urls from 2 different urls list and can't found it so I'm deciding to make it using php to removing urls from List 1 if found on List 2 on the same filename.
With this codes you can make sure that there is no duplicated urls on the other urls list if you need to run multiple urls list on your GSA SER

To make this codes work, you need to identifying first the raw urls using GSA Platform Identified or GSA SER and then putting them to 2 different folder then setup the path

Feel free to modifying or improving this codes

<?php
$files = glob('D:\GSA\List\GlobalList\*.txt');
$comparepath = "D:\GSA\List\IndonesiaList";
//$comparefile = "D:\htdocs\serpresults\betresultsEN.txt";
foreach ($files as $key => $value)
{
    clearstatcache();
    $i = 0;
    $file_path = pathinfo($value);
    $comparefile = $comparepath."/".$file_path['basename'];
    if (file_exists($comparefile))
    {
        echo "Removing duplicate from ".$value." and ".$comparefile." ";
        $contenta = file($value);
        $contentb = file($comparefile);
        if (!empty($contenta) && !empty($contentb))
        {
            foreach ($contenta as $keyc => $cvalue)
            {
                $grhost = parse_url($cvalue, PHP_URL_HOST);
                if (!empty($grhost))
                {
                    if (empty(preg_grep("/\b$grhost\b/i", $contentb)))
                    {
                        //$result = $grscheme."://".$grhost;
                        $cleanurl[] = $cvalue.PHP_EOL;
                    }
                    else
                    {
                        $i++;
                    }
                }
            }
            //$diff = array_diff($contentb, $contenta);
            $cleanfile = fopen($value, 'r+');
            $cleanarr = implode($cleanurl);
            ftruncate($cleanfile, 0);
            rewind($cleanfile);
            fwrite($cleanfile, $cleanarr);
            fclose($cleanfile);
            unset($contenta);
            unset($contentb);
            unset($cleanarr);
        }
    }
    echo "Found ".$i." duplicate on ".$value."\n";
    unset($i);
}
?>

Comments

  • SvenSven www.GSA-Online.de
    Thanks for the code. There is also one other way to get this done quickly. GSA Search engine Ranker has this included. Go to options->Advanced->tools->Remove duplicates from file
  • boilerboiler Indonesia
    edited December 2020
    Sven said:
    Thanks for the code. There is also one other way to get this done quickly. GSA Search engine Ranker has this included. Go to options->Advanced->tools->Remove duplicates from file
    it's only worked for 1 file i think
    the code i wrote will comparing the content from each file and removing it from the first list if url found on second list and since I'm using 3 list on my GSA, with this code i'm really sure that there is no same domain for each list :)

    for example domain abc.com will only found on list A but can't be find on List B or List C

    Removing duplicate from D:\GSA\List\GlobalList\sitelist_Article-BuddyPress.txt and D:\GSA\List\IndonesiaList/sitelist_Article-BuddyPress.txt Found 0 duplicate on D:\GSA\List\GlobalList\sitelist_Article-BuddyPress.txt
    Removing duplicate from D:\GSA\List\GlobalList\sitelist_Article-Catalyst Web CMS.txt and D:\GSA\List\IndonesiaList/sitelist_Article-Catalyst Web CMS.txt Found 0 duplicate on D:\GSA\List\GlobalList\sitelist_Article-Catalyst Web CMS.txt
    Removing duplicate from D:\GSA\List\GlobalList\sitelist_Article-ClassiPress.txt and D:\GSA\List\IndonesiaList/sitelist_Article-ClassiPress.txt Found 0 duplicate on D:\GSA\List\GlobalList\sitelist_Article-ClassiPress.txt
    Removing duplicate from D:\GSA\List\GlobalList\sitelist_Article-Drupal - Blog.txt and D:\GSA\List\IndonesiaList/sitelist_Article-Drupal - Blog.txt Found 0 duplicate on D:\GSA\List\GlobalList\sitelist_Article-Drupal - Blog.txt
Sign In or Register to comment.