FEATURE REQUEST: Global Blacklist

AlexR

Hi

I would like to have 1 global blacklist across a few VPS's. To do this I'd like to be able to specify a txt file that it checks.
(like the #file macro)

It would also make it much easier adding domains to it. (i.e. if you're checking your inbox and want to remove someone you just update the TXT file rather than having to login to the VPS & import it)

wolfvanween

There's another point where "check a folder" would be incredibly useful: 
In Article Manager "Add Folder". If I'm not mistaken, this is a one-time import now, but it would be great if it would check for new articles every time a project is set to active.

mrlinks

Anyone got a global blacklist yet?

coneh34d

@walfvanween - Check out the %spinfolder% macro. It will pull a random article from the folder specified.

Syntax - %spinfolder-c:\my_articles\%

AlexR

1) Can it be used in the global blacklist?

coneh34d

For your Global Blacklist across a few VPS's first I'd create a folder on one of my webservers. Next I would place the global-blacklist.txt in that folder. Finally I'd go to Options > Filter and add the Blacklist URL there. Do this on all your servers. Update the blacklist text file periodically in hxxp://mywebserver/folder/global-blacklist.txt. Use FTP for automating this update task from your local global-blacklist.txt to the server copy.

loopline

I have a blacklist I use across all my servers.  I just have a dropbox folder.   That said I do blacklist filtering before loading the urls into website contact.  My blacklist is nearing 50K and there is no point in having website contact sweep every url it tries to post to in ever project thru 50K urls every time, its massively redundant and waste of resources.  none the less dropbox works for me.

AlexR

@Loopline - thanks. That makes sense if you're importing URLS. 

What if you're using Contact Submitter to scrape & submit? Surely it needs to check the blacklist first?

loopline

@AlexR

Yes good point.  I just "assumed" that most people were importing urls, my fault.  Yes you would obviously need a blacklist if you are scraping with website contact. 

I also use the blacklist in website contact, but only as a backup for really complaining people.  lol

AlexR

@loopline
"That said I do blacklist filtering before loading the urls into website contact. "

What's the best way to do that?

Sven

to add something to this discussion:

The latest update has a new option that will check if a message was sent before by maybe another project to that same site and skip that (optional).

AlexR

Where is this feature? I saw it in the changelog but can't find the setting.

Sven

global options under the blacklist.

AlexR

Awesome!

I got a few contact form submitted with:
System 1:
'click here to never be contacted again' and had an URL which I assume added you to a global blacklist

This would be much easier than having to check an inbox and add people to a black list.

System 2:
reply with 'stop' to not be contacted again. Again I assume they have some filter that detects this and auto removes them.

Would something like this be possible?

Sven

The problem is that you don't know what email is behind a website. How do you estimate what email belongs to what website? Obviously only by adding the domain to the message or parameter on the "unsubscribe url".

AlexR

Obviously only by adding the domain to the message or parameter on the "unsubscribe url". 

-> the goal would be just blacklist domains with an easy stop contacting me link.

loopline

Yeah, but then you have to have a stop contacting me link that winds up redirecting to the ip address that Website contact is on. 

Then they need only fabricate a complaint to your server company and your server is taken offline.  Pain in the rear end.  Largely Defeats the purpose of using proxies in the first place.  

I never want anyone to know the ip of my server. 

Ill pm you the program I use, as its not a competiting program but I respect Sven and his hard work and I don't want to discuss non GSA programs here on this form in public.  Just out of respect because Im not trying to deter anyone from buying GSA products. 

I mean I have 7 licenses for website contact nearly a dozen for SER and other GSA product licenses as well, its a great product Suite that saves me tons of time and makes me lots of $$$   

Thanks Sven! 

I use opt out links though, I redirect them to a wordpress site with a contact form and I do as Sven noted and put the domain its sent to in the opt out link.  Then I had my full stack dev write a wordpress plugin that auto captures that and adds it to a database on the site.  Then it gives them a nice clean "X domain has been opted out" but also provides them an option to submit more domains, leave comments and the icing on the cake is I provide them with a fancy box to check that says "report this as spam" 

Because people want to "Do their duty" and report everything under the sun as spam, frankly I think some of these people need to be reported as spam themselves, but thats another story. 

So if it makes them happy, they can report it to me as spam and then they don't feel the need to go report it somewhere else. 

Then once a week I download all my opt outs and add scrub them (people submit dumb stuff in the extra domains field) and add them to my opt out list.  Its maybe a 20 min process once a week, used to take longer but I fine tuned it. 

Then I pop that in dropbox and every time I load up a chunk of urls into a campaign I first scrub it against the current opt out list and boom Im set. 

Then if I get really grouchy people or if one of my customers gets a bit complainer (had a few over the months) I just pop their domain into the blacklist just to make sure its immediately opted out until my next round of processing.

Plus even if Sven had it where you could do 1 click opt outs and it was fully automatic, I would not do it.

Because my contact form gives me valueable data.  Like for 1 if they have more then 1 domain, tons of people have more then 1 domain and they dont understand its contact form marketing, they think you are emailing them.  So by the time they are on domain 3 clicking opt out they think its not working, get ticked off and are not clicking on any more opt out links.  Ive had people with 500 domains. 

Second is the realitors.  They leave valueable feedback in the comments section like "your contacting us on every new listing!" 

Well some realitors create new domains for every single home they list.  Thats endless and they will never keep clicking opt outs, they will go elsewhere or go report you or worse.

So then I just find their name or some piece of info that shows up on every single one of their sites and add it into the filter section in website contact for all my projects. 

So really its not a simple cut and dry game.  If it was emailing, blacklist the email and your set, but with contact form marketing, its low hanging fruit that scales the submissions inexpensively, but its essentially an email forwarder or forwarder domain, because the message still gets sent to someones email and you can't see that.  and if they have 500 domains and you send them 500 messages to 1 inbox over time they are ticked and you have no way of stopping it unless you can opt out all their domains.

I even have an opt out shell company so I can mail folks as needed.  Opt outs has been one of  my largest hurdles, fine tuning it all into a well oiled machine.  But lots of trial and error (and people you have to deal with that have no perspective on life) along the way. 

Im also working with 10million+ domains so if your going much smaller scale or more targeted, many of these things may be no issue, but I need something that works at scale. 

Sven

Thanks @loopline for your feedback on this. If there is anything I can make it easier like right click on project->filter out by domains in file or alike, let me know.

AlexR

Wow. @loopline You crushed it with that answer. :-)

I guess that opt out website is on a different server IP! :-)

Time to get cracking and take action on your great advice. 

I felt that opt outs was a PITA but hadn't built a process on it yet.

loopline

@Sven

Thanks, Ill keep that in mind.  Ive run into a few struggles, so Ill mull it over and let you know if something really makes sense. 

@AlexR

Glad to be of a help.     Opt outs are kind of just annoying, but Ive got it down to a mostly science at this point so they are passed the PITA stage for me at least. 

Also yes, you are correct, my opt out page is on a different ip all together.  I actually have everything setup for redundancy incase of issues.  I have multiple hosting providers for websites, a completely different hosting provider that I only use for redirects, privacy guard on all domains, and then the opt out company for mailing folks (is nice because I can say "we don't do marketing just handle the opt outs" when I mail the occasional person).

I have multiple physical mailing addresses as well.  Also just bear in mind people look up that physical mail address. You would think that if you had an opt out link, they would just click it, but no, they look up the mailing address and try and complain. 

So I have a site with the primary mailing address and it just says some basic info that we are privacy company and it has a contact form (people feel, on the whole, that clicking opt out won't opt them out, they want to contact someone, some way, somehow)

Then I have another shell/dba company that is just a marketing company for my client that wants to distance his entire company name, so at the bottom of messages sent, with the address, above the opt out link, it flat says that its sent by xyz marketing company, which has the mailing address. 

To be frank, my opt out structure, servers and campaigns is more complex then the sending.  Its still really great though, contact form marketing is massively low hanging fruit compared to cold mail marketing. 

Its a minute to set it all up, but then it runs and its worth it. 

AlexR

Is it possible to have 

%url_domain%

%url_path%
To have these working? (Then we can use them in the unsubscribe link to pass the correct variable out for the project.

Sven

you have that already...it's %domain% and %subdomain% ... you don't need the path.

loopline

I use %domain%

AlexR

%domain% - takes it from the target domain. I use that in the content.
But for the unsubscribe I want to tag in the domain it used to submit in the website field for the contact form. (You see it grabs a random one from a list of domains. e.g. website.xyz, website.abc, - I never use my real one and each project has multiple domains it uses that redirect to the main website so I can keep my main website clean)
%url_domain% - I thought this would do that. 

Sven

@AlexR, are we talking about the same program here?

AlexR

Website Contact Form Submitter. 
- I'm trying to append the website it used in the submissions to an unsub url. (e.g /unsub/?project=mywebsite.com, /unsub/?project=mywebsite.xyz, etc)

Sven

then %domain% is exactly what you need.

AlexR

Does %domain% - use the target domain or use the domain I'm using to submit? 
If %domain% is the domain I'm using to submit what is the variable for the domain of the site it's filling out the contact form on?

Sven

its the target domains name.

AlexR

That's what I thought!
What's the variable for the domain name used in the website field?

Sven

there is no macro for that.