Discussion:FAQ new config

Draft squashfs/mountimg

How can I use many small files efficiently?

You can gain in performance and minimize the pressure under /data in the following cases:

• case1 your jobs are only reading under the directories where your zotfiles reside
• case2 your jobs are reading your zotfiles but add new files in them
• case3 your jobs generate zotfiles, but they will be accessed only for reading or adding new files afterwards

For case1:

• convert your zotfiles directories to squashfs images
• in your jobs:
  • mount those images using sudo mountimg
  • use those mounted directories for processing

For case2:

• convert your zotfiles directories to squashfs images
• in your jobs:
  • mount those images using sudo mountimg
  • use those mounted directories for processing but generate new file on the local filesystems of the node (ex: /tmp)
  • unmount the images with sudo mounting -u
  • add the new files to the images with mksquashfs-no-compression

For case3:

• generates your zotfiles on the local filesystems of the node (ex: /tmp)
• convert them to squashfs images under /data with mksquashfs-no-compression