How the posting really works?

busek

I am curious how gsa is posting on variable platforms - I guess it's the same schema. But what interests me is can we somehow disable growing duplicates in our global lists? Say we have like 200k of some platform and after posing it goes twice or more than that value as those urls are again identified. Would not it be faster to skip that identification (which leads us to growing duplicates) in every project run? We can make cleanup or verify it only once. Regarding the above I am not also sure of one thing - if we make remove duplicates doesn't it implicate ommiting some urls in currently running projects? I mean it changes urls order in that file after all. As far as I hear gsa load only like few hundreds url in bunches when posting for memory saving.

Sven

skipping identification is not a good idea. The site might have changed or no longer support posting. That all has to be detected first. However it is not very resource hungry and nothing to worry about.

busek

Ok - so is gsa removing url from the global identified list when currently running project will not detect it? I only worry about skipping urls because we have to remove dupes from those global lists.

Justin

@busek - what are you worried about exactly?

Sven

no url is removed from any site list unless you use the "remove dupe/cleanup" functions manually.

busek

Because we have to remove those dupes from identified folder (as it grows fast with time) I think that it changes posting order in particular projects, cause I do not think gsa is making temp list of that global file - it would be a waste of space.

Justin

Okay I understand what you are saying now @busek. If you remove dupes while SER is running, there is the risk that you might lose targets since you might be removing targets from SERs memory.

Sven

Actually no, there is no risk when deduping and SER is using the lists in projects.