Are you using PI Blacklist feature? Whats the size of your Blacklist database?

mazs91

Hi Everyone,

I have already discussed this issue with @s4nt0s, he has no immediate plan to improve the functionality, so here I am trying to figure out how other users handling this issue.

The Problem is my PI Blacklist file (bldatabase.sqlite) size is now 12 GB (started using PI 25 days back). I have a VPS with 50GB SSD and it is soon going to be out of space as the size of this file is increasing quite fast. I am assuming the PI is storing the complete url instead of the hash value. I have processed 10 times more urls with SER and still the size of the history ((hosts_done & urls_done) is not more than 2mb.

Are you people experiencing the same issue or is it just me? Is there any way to sort it out?

File Path: C:\Users\Administrator\AppData\Local\GSA Platform Identifier

Mazhar

shaun

@mazs91 I suggested this feature. The idea is that PI saves a list of the URLs it has processed and then checks new URLs against it to save time and resources when building your own list.

I havent used PI in a few weeks but when it was first implimented im sure there was a tick box so the user could choose if they wanted the blacklist to be built or not.

Depending on what you are doing the feature may not be worth your time. I was processing millions of links at a time and finding that well over half were duplicates within a short space of time and that number was only ever going to increase.

mazs91

@shaun.. that was a very clever suggestion...yesterday I cleared my 12GB 70million url blacklist database and my Urls/min went from 14000 to 4000 with the same settings. This clearly shows that there are high number of duplicates that were skipped due to blacklist.

The blacklist feature is working perfectly. Problem is how these urls are stored. I am adding about 700MB in the blacklist daily, it reached 12GB yesterday and I cleared the blacklist. My VPS was running out of space. I was just wondering if other users are experiencing this issue and if I can do anything to sort it out..

Johan

@mazs91 ... does seem a large size, seems bloated

Would suggest that you download the DB Browser program for SQLite from here:

https://github.com/sqlitebrowser/sqlitebrowser/releases/tag/v3.9.1

When you run it, browse to where the database is located > then go to File > Compact Database

See whether that works for you, if the DB is bloated then i should reduce it considerably, maybe report back what you find? It's worked for me with other databases.

Cheers

s4nt0s

The blacklist database will automatically be compacted to the smallest size when cleared in the next Pi update.