Skip to content

GSA PI lock mechanism while deduplicating files

AliTabAliTab GSAserlists.com
in GSA Platform Identifier

Hello @Sven and @s4nt0s,

It seems that GSA PI is not using a locking mechanism when it's deduplicating. So, when my own code wants to access the same files that are being deduplicated at the same time, PI will crash. Can you please check this and let me know where I can find the lock file to make my own code respect PI and avoid writing to the same file when it's being processed?

Thanks a lot!

Comments

  • SvenSven www.GSA-Online.de
    Im no in office today and will have to take a look on this tomorrow.
    Thanked by 1AliTab
  • AliTabAliTab GSAserlists.com
    Any update on this @Sven ?
  • SvenSven www.GSA-Online.de
    i need to reboot the system to activate vm options in bios....seems like the new system doesn't allow me to run my old VM ... needs to wait a bit.
    Thanked by 1AliTab
  • SvenSven www.GSA-Online.de
    OK now Im ready to debug this...however, I see no issue in allowing a file that is deduped to be read/write accessable while the end file is stored to .tmp and then moved over!?
  • AliTabAliTab GSAserlists.com
    edited September 2023
    Thank you Sven for checking that. It seems you're right! I have figured out the bug on my own end.
  • AliTabAliTab GSAserlists.com



    Sven, there is a bug when "Append" is active. If the file doesn't exist, it should create the file and transfer the data. But now, if the file doesn't exist in the destination folder, the data will be lost.
  • SvenSven www.GSA-Online.de
    sorry for late reply. Its fixed in next update.
    Thanked by 1AliTab
  • AliTabAliTab GSAserlists.com
    Sven said:
    OK now Im ready to debug this...however, I see no issue in allowing a file that is deduped to be read/write accessable while the end file is stored to .tmp and then moved over!?
    Hi @Sven

    I'm reopening this thread once more to address a persistent issue. If a .lock file is not created alongside the file when other processes are reading and writing in the same folder, problems will still arise. Writing to a .tmp file is indeed the best option but still makes issues. During the process of deduplication for a file, a .lock file should be created next to it. This indicates to other processes that the file is currently being used by PI. Once deduplication is complete, the lock can be safely released. This is what's making my PI crash every time, especially in a multi-threaded environment. Could I expect an update regarding this in GSA PI?

    Thank you very much
  • SvenSven www.GSA-Online.de
    hmm will think about it, but a .lock file is still giving issues. What happens if another process wants to write to the file as well now? Is it ignoring the data or should it wait will .lock is gone?
  • AliTabAliTab GSAserlists.com
    Sven said:
    hmm will think about it, but a .lock file is still giving issues. What happens if another process wants to write to the file as well now? Is it ignoring the data or should it wait will .lock is gone?
    Currently, PI crashes when I access a file that is undergoing deduplication. This issue occurs in a multi-threaded environment, and the likelihood of PI crashing increases if I raise the number of threads to speed up the process.
  • AliTabAliTab GSAserlists.com
    Hello @Sven

    Any update for this request?
    Sven said:
    OK now Im ready to debug this...however, I see no issue in allowing a file that is deduped to be read/write accessable while the end file is stored to .tmp and then moved over!?
    Hi @Sven

    I'm reopening this thread once more to address a persistent issue. If a .lock file is not created alongside the file when other processes are reading and writing in the same folder, problems will still arise. Writing to a .tmp file is indeed the best option but still makes issues. During the process of deduplication for a file, a .lock file should be created next to it. This indicates to other processes that the file is currently being used by PI. Once deduplication is complete, the lock can be safely released. This is what's making my PI crash every time, especially in a multi-threaded environment. Could I expect an update regarding this in GSA PI?

    Thank you very much

  • SvenSven www.GSA-Online.de
    still on my to do list sorry.
    Thanked by 1AliTab
  • SvenSven www.GSA-Online.de
    please check latest version v2.10 and let me know if there is still and issue.
  • AliTabAliTab GSAserlists.com
    Sven said:
    please check latest version v2.10 and let me know if there is still and issue.

    Thank you, Sven. I'm currently testing that and will report back. Meanwhile, could you please explain the changes that have been made?

    Thanks again.

  • serkan1071serkan1071 TURKEY
    Sven said:
    sorry for late reply. Its fixed in next update.
    have you update (Parse)
    https://forum.gsa-online.de/discussion/31830/parse-save-error/p1
  • SvenSven www.GSA-Online.de
    @AliTab if it detects that a file is in use, it will wait and retry to write to it later.
    @serkan sorry, this is a bit more complicated to add and takes longer
  • AliTabAliTab GSAserlists.com
    Sven said:
    @AliTab if it detects that a file is in use, it will wait and retry to write to it later.
    @serkan sorry, this is a bit more complicated to add and takes longer
    So far working great, Thank you very much Sven
Sign In or Register to comment.