Discussion:FAQ new config : Différence entre versions
De ClustersSophia
(Draft squashfs/mountimg) |
(+ case adding new files) |
||
| Ligne 3 : | Ligne 3 : | ||
== How can I use many small files efficiently? == | == How can I use many small files efficiently? == | ||
| − | You can gain in performance and minimize the pressure under /data in | + | You can gain in performance and minimize the pressure under /data in |
| + | the following cases: | ||
* '''case1''' your jobs are only reading under the directories where your zotfiles reside | * '''case1''' your jobs are only reading under the directories where your zotfiles reside | ||
| − | * '''case2''' your jobs generate zotfiles, but they will be accessed only for reading afterwards | + | * '''case2''' your jobs are reading your zotfiles but add new files in them |
| + | * '''case2''' your jobs generate zotfiles, but they will be accessed only for reading or adding new files afterwards | ||
For '''case1''': | For '''case1''': | ||
| Ligne 14 : | Ligne 16 : | ||
For '''case2''': | For '''case2''': | ||
| + | * convert your zotfiles directories to squashfs images | ||
| + | * in your jobs: | ||
| + | ** mount those images using '''sudo mountimg''' | ||
| + | ** use those mounted directories for processing but generate new file on the local filesystems of the node (ex: /tmp) | ||
| + | ** unmount the images with '''sudo mounting -u''' | ||
| + | ** add the new files to the images with '''mksquashfs-no-compression''' | ||
| + | |||
| + | For '''case3''': | ||
* generates your zotfiles on the local filesystems of the node (ex: /tmp) | * generates your zotfiles on the local filesystems of the node (ex: /tmp) | ||
| − | * convert them to squashfs images under /data | + | * convert them to squashfs images under /data with '''mksquashfs-no-compression''' |
Version du 11 février 2019 à 08:36
Draft squashfs/mountimg
How can I use many small files efficiently?
You can gain in performance and minimize the pressure under /data in the following cases:
- case1 your jobs are only reading under the directories where your zotfiles reside
- case2 your jobs are reading your zotfiles but add new files in them
- case2 your jobs generate zotfiles, but they will be accessed only for reading or adding new files afterwards
For case1:
- convert your zotfiles directories to squashfs images
- in your jobs:
- mount those images using sudo mountimg
- use those mounted directories for processing
For case2:
- convert your zotfiles directories to squashfs images
- in your jobs:
- mount those images using sudo mountimg
- use those mounted directories for processing but generate new file on the local filesystems of the node (ex: /tmp)
- unmount the images with sudo mounting -u
- add the new files to the images with mksquashfs-no-compression
For case3:
- generates your zotfiles on the local filesystems of the node (ex: /tmp)
- convert them to squashfs images under /data with mksquashfs-no-compression
