Discussion:FAQ new config

Draft squashfs/mountimg

How can I use many small files efficiently?

You can gain in performance and minimize the pressure under /data in two cases:

• case1 your jobs are only reading under the directories where your zotfiles reside
• case2 your jobs generate zotfiles, but they will be accessed only for reading afterwards

For case1:

• convert your zotfiles directories to squashfs images
• in your jobs:
  • mount those images using sudo mountimg
  • use those mounted directories for processing

For case2:

• generates your zotfiles on the local filesystems of the node (ex: /tmp)
• convert them to squashfs images under /data