Hardware : Différence entre versions

De ClustersSophia
Aller à : navigation, rechercher
 
(180 révisions intermédiaires par 4 utilisateurs non affichées)
Ligne 3 : Ligne 3 :
  
  
−
= new cluster - CPU & GPU =
 
  
−
The current platform includes:
+
= CPU clusters =
  
−
*    13 quad-Xeon 6 cores
 
−
*    44 dual-Xeon 6 cores
 
  
−
The number of cores available is
 
−
*    '''312''' Xeon 2.4Ghz cores :  [to-be-completed] Tflops ([to-be-completed] Tflops peak)
 
−
*    '''528''' Xeon 2.93Ghz cores :  [to-be-completed] Tflops ([to-be-completed] Tflops peak)
 
  
−
'''The total cumulated computing power is close to [to-be-completed] Tflops (peak, double precision)'''
 
  
−
==  Dell R900 Cluster ==
+
== Dell R7525 nodes (2020) ==
  
−
'''Dell R900''' quad-Xeon E7450 @ 2.4GHz (24 cores) : 13 nodes
+
[[Image:dellr7525.jpg|200px|right]]
  
−
[[Image:Dell-r900.jpeg|x100px|right]]
 
  
 +
'''Dell R7525''' dual-EPYC 7502 @ 2.5GHz (64 cores) : 1 node ( '''nef059''' )
  
−
*    RAM capacity : 64 GB RAM
+
*    RAM capacity : 384 GB RAM
−
*    2x146GB hard disk drive (RAID-1) SAS 15K RPM
+
*    storage : system 2x900 GB SATA SSD RAID-1 + local scratch data 7x7.5TB SAS HDD 44 TB RAID-5 + controller H745
−
*    4x gigabit network ports (one connected)
+
*    2x gigabit network ports (one connected)
−
*    1x infiniband QDR card  
+
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
  
−
==  Dell C6100 Cluster ==
 
  
−
'''Dell C6100''' dual-Xeon X5670 @ 2.93GHz (12 cores) : 44 nodes
+
'''Dell R7525''' dual-EPYC 7542 @ 2.9GHz (64 cores) : 1 node ( '''nef058''' )
  
−
[[Image:Dell-c6100.jpeg|x120px|right]]
+
*    RAM capacity : 1024 GB RAM
−
 
+
*    storage : system 2x223 GB SATA SSD RAID-1 + local scratch data 5x445GB SAS SSD 1.8 TB RAID-5 + controller H745
−
 
 
−
*    RAM capacity : 96 GB RAM
 
−
*    1x250GB hard disk drive SATA 7.2kRPM
 
 
*    2x gigabit network ports (one connected)
 
*    2x gigabit network ports (one connected)
−
*    1x infiniband QDR card  
+
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
  
 +
== Dell C6420 cluster (2019) ==
  
 +
'''Dell C6420''' dual-Xeon Cascade Lake SP Gold 6240 @ 2.60GHz (36 cores) : 4 nodes ( '''nef054 to nef057''' )
  
−
= legacy cluster - CPU & GPU =
+
[[Image:C6420.jpg|200px|right]]
  
−
The current platform includes:
 
  
−
*    6 quad-Opteron 12 cores
+
*    RAM capacity : 384 GB RAM
−
*    6 quad-Opteron 16 cores
+
*    storage : system 2x600 GB SATA RAID-1 + local scratch data 960 BB SATA SSD RAID-0 + controller H330
−
*    16 dual-Xeon 10 cores.
+
*    1x gigabit network port
−
*    21 dual-Xeon quad core.
+
*    1x infiniband FDR card
−
*    2 dual-Xeon hexa core.
+
*    hyperthreading active
−
*    2 mono-Xeon hexa core.
+
*    optimal performance with AVX-512, AVX/AVX2 support
−
*    22 Nvidia Tesla GPU
 
  
−
The number of cores available is
+
== Dell R940 node (2017) ==
  
−
*    '''288''' Opteron 2.2Ghz cores : ~2 Tflops (2.53 Tflops peak)
 
−
*    '''384''' Opteron 2.3Ghz cores : ~2.7 Tflops (3.5 Tflops peak)
 
−
*    '''320''' Xeon E5 2.8Ghz cores : ~5.9 Tflops (7.17 Tflops peak)
 
−
*    '''204''' Xeon 2.66Ghz cores : ~1.7 Tflops (2.17 Tflops peak)
 
−
*    '''9024''' Streaming Processor Cores (22 Nvidia GPU) 9.5Tflops peak (22.2 Tflops in single precision)
 
  
 +
'''Dell R940''' quad-Xeon SP Gold 6148 @ 2.40GHz (80 cores) : 1 node ( '''nef053''' )
  
−
'''The total cumulated computing power is close to 24 Tflops (peak, double precision)'''
+
[[Image:dellr940.jpg|200px|right]]
  
  
−
== Dell R815 Cluster ==
+
*    RAM capacity : 1024 GB RAM
 +
*    storage : system 2x600 GB SATA RAID-1 + local scratch data 1.92 TB SATA SSD + controller H740P
 +
*    4x gigabit network ports (one connected)
 +
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
 +
*    optimal performance with AVX-512, AVX/AVX2 support
  
 +
== Dell C6420 cluster (2017) ==
  
−
'''Dell R815''' quad-Opterons 6174 @ 2.2Ghz (48 cores) : 6 nodes
+
'''Dell C6420''' dual-Xeon Skylake SP Silver 4114 @ 2.20GHz (20 cores) : 16 nodes ( '''nef037 to nef052''' )
  
−
[[Image:Cluster-dellr815.jpg|x200px|right]]
+
[[Image:C6420.jpg|200px|right]]
  
  
−
*    RAM capacity : 256 GB RAM
+
*    RAM capacity : 192 GB RAM
−
*    2x600GB SAS HardDisk drive (RAID-0)
+
*    1x600GB 10kRPM SAS HardDisk drive
−
*    4x gigabit network ports (one connected)
+
*    1x gigabit network port
−
*    1x infiniband QDR card  
+
*    1x infiniband FDR card
−
<br clear=all>
+
*    hyperthreading active
 +
*    AVX-512 support, optimal performance with AVX/AVX2
  
−
==  Dell C6145 Cluster ==
+
== Dell C6220 cluster (2015) ==
  
 +
'''Dell C6220''' dual-Xeon E5-2650 v2 @ 2.60GHz (16 cores) : 8 nodes ( '''nef029 to nef036''' )
  
−
'''Dell C6145''' quad-Opterons 6376 @ 2.3Ghz (64 cores) : 6 nodes
+
[[Image:C6220-300x191.jpg|200px|right]]
  
−
[[Image:Cluster-c6145.jpg|200px|right]]
 
  
−
 
+
*    RAM capacity : 256 GB RAM
−
*    RAM capacity : 256 GB RAM (512GB on nef011 and nef012)
+
*    1x1TB SATA HardDisk drive
−
*    1x500GB SATA HardDisk drive
 
 
*    2x gigabit network ports (one connected)
 
*    2x gigabit network ports (one connected)
−
*    1x infiniband QDR card  
+
*    2x infiniband QDR card (one connected)
 +
*    hyperthreading not active
  
−
==  Dell C6220 Cluster ==
 
  
 +
== Dell C6220 cluster (2014) ==
  
−
'''Dell C6220''' dual-Xeon E5-2680 v2 @ 2.80GHz (20 cores) : 16 nodes
+
'''Dell C6220''' dual-Xeon E5-2680 v2 @ 2.80GHz (20 cores) : 16 nodes ( '''nef013 to nef028''' )
  
 
[[Image:C6220-300x191.jpg|200px|right]]
 
[[Image:C6220-300x191.jpg|200px|right]]
Ligne 105 : Ligne 99 :
 
*    2x gigabit network ports (one connected)
 
*    2x gigabit network ports (one connected)
 
*    1x infiniband FDR card (QDR used)  
 
*    1x infiniband FDR card (QDR used)  
 +
*    hyperthreading not active
  
  
−
==  Dell 1950 Cluster ==
+
== Dell C6145 cluster (2013) ==
  
 +
'''Dell C6145''' quad-Opterons 6376 @ 2.3Ghz (64 cores) : 6 nodes ( '''nef007 to nef012''' )
  
−
'''PowerEdge 1950''' dual-Xeon 5355 @ 2.66Ghz : 19 nodes
+
[[Image:Cluster-c6145.jpg|200px|right]]
  
−
[[Image:Cluster-dell.jpg|x200px|right]]
 
  
−
*    RAM capacity : 16 GB
+
*    RAM capacity : 256 GB RAM (512GB on nef011 and nef012)
−
*    2x73GB SAS in RAID-0, 146Go disk space available
+
*    1x500GB SATA HardDisk drive
−
*    2x gigabit netword ports (one connected)
+
*    2x gigabit network ports (one connected)
 
*    1x infiniband QDR card  
 
*    1x infiniband QDR card  
−
<br clear=all>
+
*    hyperthreading not supported
  
−
==  HP GPU cluster ==
 
  
 +
== Dell R815 cluster (2010) ==
  
−
'''HP DL160 G5''' nodes: dual-Xeon 5430 @ 2.66Ghz : 2 nodes
+
'''Dell R815''' quad-Opterons 6174 @ 2.2Ghz (48 cores) : 6 nodes ( '''nef001 to nef006''' )
  
−
*    RAM capacity: 16 GB
+
[[Image:Cluster-dellr815.jpg|x200px|right]]
−
*    500GB of local disk space
 
−
*    2x gigabit ethernet interface
 
−
*    1x 10GBit myrinet interface
 
−
*    2 GPUs connected with a PCIe gen2 16x interface
 
  
−
'''Nvidia Tesla''' S1070: 1 node
 
  
−
[[Image:Nvidia-tesla.jpg|200px|right]]
+
*    RAM capacity : 256 GB RAM
 +
*    2x600GB SAS HardDisk drive (RAID-0)
 +
*    4x gigabit network ports (one connected)
 +
*    1x infiniband QDR card
 +
*    hyperthreading not supported
 +
 
 +
= GPU nodes =
 +
 
 +
== HPE DL385 GPU nodes (2023) ==
 +
 
 +
'''HPE DL385''' dual-AMD EPYC 7313 @ 3.0GHz (16 cores) : 3 nodes ( '''nefgpu59 to nefgpu61''' )
 +
* Nvidia A40 PCIe GPUs cards
 +
**        10752 CUDA cores per card
 +
**        336 tensor cores per card
 +
**        48GB of RAM capacity per card
 +
**        Tensor performance peak : 149.6 TFlops per card
 +
**        Simple precision performance peak: 74.8 Tflops per card
 +
**        Double precision performance peak: 37.4 Tflops per card
 +
**        696 GB/s GPU memory bandwidth with error correction (ECC)
 +
*    2x gigabit network ports (one connected)
 +
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
 +
 
 +
'''HPE DL385''' dual-AMD EPYC 7513 @ 2.6GHz (32 cores) : 2 nodes ( '''nefgpu57 to nefgpu58''' )
 +
* Nvidia A100 - 80GB PCIe GPUs cards
 +
**        6912 CUDA cores per card
 +
**        432 tensor cores per card
 +
**        80GB of RAM capacity per card
 +
**        Tensor performance peak : 312 TFlops per card
 +
**        FP16 Compute: 78 Tflops per card
 +
**        FP32 Compute: 19.5 Tflops per card
 +
**        FP64 Compute: 9.7 Tflops per card
 +
**        1935GB/s GPU memory bandwidth with error correction (ECC)
 +
*    2x gigabit network ports (one connected)
 +
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
 +
 
 +
'''HPE DL385''' dual-AMD EPYC 7662 @ 2.0GHz (64 cores) : 1 node ( '''nefgpu56''' )
 +
* Nvidia A100 - 40GB PCIe GPUs cards
 +
**        6912 CUDA cores per card
 +
**        432 tensor cores per card
 +
**        40GB of RAM capacity per card
 +
**        Tensor performance peak : 312 TFlops per card
 +
**        Simple precision performance peak: 156 Tflops per card
 +
**        Double precision performance peak: 19.49 Tflops per card
 +
**        1555 GB/s GPU memory bandwidth with error correction (ECC)
 +
*    2x gigabit network ports (one connected)
 +
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
 +
{| class="wikitable"
 +
|+Node details
 +
|-
 +
| '''Node name'''
 +
| '''Funding team'''
 +
| '''GPU cards'''
 +
| '''Node CPU'''
 +
| '''Node RAM'''
 +
| '''Node storage'''
 +
|-
 +
| nefgpu56
 +
| STARS
 +
| 3x A100 40GB
 +
| 2x AMD EPYC 7662
 +
| 1024 GB
 +
| system & /tmp : RAID-1 2x 480SSD<br>/local/mixed : RAID-5 5x 960 SSD
 +
|-
 +
| nefgpu57
 +
| MAASAI
 +
| 2x A100 80GB
 +
| 2x AMD EPYC 7513
 +
| 512 GB
 +
| system & /local/mixed : RAID-1 2x 480SSD<br>/local/mixed : RAID-0 6x 960 SSD
 +
|-
 +
| nefgpu58
 +
| ZENITH
 +
| 2x A100 80GB
 +
| 2x AMD EPYC 7513
 +
| 512 GB
 +
| system & /local/mixed : RAID-1 2x 480SSD<br>/local/mixed : RAID-0 6x 960 SSD
 +
|-
 +
| nefgpu59
 +
| MORPHEME
 +
| 3x A40
 +
| 2x AMD EPYC 7313
 +
| 512 GB
 +
| system & /local/mixed : RAID-1 2x 480SSD<br>/local/mixed : RAID-0 4x 960 SSD
 +
|-
 +
| nefgpu60
 +
| ACUMES
 +
| 2x A40
 +
| 2x AMD EPYC 7313
 +
| 256 GB
 +
| system & /local/mixed : RAID-1 2x 480SSD<br>/local/mixed : RAID-0 4x 960 SSD
 +
|-
 +
| nefgpu61
 +
| ACUMES
 +
| 2x A40
 +
| 2x AMD EPYC 7313
 +
| 256 GB
 +
| system & /local/mixed : RAID-1 2x 480SSD<br>/local/mixed : RAID-0 4x 960 SSD
 +
|}
 +
 
 +
== Dell R7525 GPU nodes (2020) ==
 +
 
 +
'''Dell R7525''' dual-AMD EPYC 7282 @ 2.8GHz (16 cores) : 2 nodes ( '''nefgpu52 to nefgpu53''' )<br>
 +
'''Dell R7525''' dual-AMD EPYC 7413 @ 2.65GHz (24 cores) : 2 nodes ( '''nefgpu54 to nefgpu55''' )
 +
* Nvidia A40 PCIe GPUs cards
 +
**        10752 CUDA cores per card
 +
**        336 tensor cores per card
 +
**        48GB of RAM capacity per card
 +
**        Tensor performance peak : 149.6 TFlops per card
 +
**        Simple precision performance peak: 74.8 Tflops per card
 +
**        Double precision performance peak: 37.4 Tflops per card
 +
**        696 GB/s GPU memory bandwidth with error correction (ECC)
 +
*    2x gigabit network ports (one connected)
 +
*    infiniband EDR card (connected to FDR switch)
 +
*    hyperthreading active
 +
 
 +
{| class="wikitable"
 +
|+Node details
 +
|-
 +
| '''Node name'''
 +
| '''Funding team'''
 +
| '''GPU cards'''
 +
| '''Node CPU'''
 +
| '''Node RAM'''
 +
| '''Node storage'''
 +
|-
 +
| nefgpu52
 +
| NEO
 +
| 3x A40
 +
| 2x AMD EPYC 7282
 +
| 256 GB
 +
| system & /tmp : RAID-1 2x 480SSD<br>/local/mixed : RAID-5 5x 480 SSD
 +
|-
 +
| nefgpu53
 +
| ACENTAURI
 +
| 3x A40
 +
| 2x AMD EPYC 7282
 +
| 256 GB
 +
| system & /tmp : RAID-1 2x 480SSD<br>/local/mixed : RAID-5 5x 480 SSD
 +
|-
 +
| nefgpu54
 +
| 3IA
 +
| 3x A40
 +
| 2x AMD EPYC 7413
 +
| 512 GB
 +
| system & /tmp : RAID-1 2x 480SSD<br>/local/mixed : RAID-5 5x 960 SSD
 +
|-
 +
| nefgpu55
 +
| 3IA
 +
| 3x A40
 +
| 2x AMD EPYC 7413
 +
| 512 GB
 +
| system & /tmp : RAID-1 2x 480SSD<br>/local/mixed : RAID-5 5x 960 SSD
 +
|}
  
−
*    4 GPUs (2 per compute node)
 
−
*    Streaming Processor Cores: 960 (260 per GPU)
 
−
*    Simple precision performance peak: 3.73 Tflops
 
−
*    Double precision performance peak: 0.31 Tflops
 
−
*    RAM capacity : 16 GB (4GB per GPU)
 
  
 +
== SuperMicro 4029 node (2019) ==
  
−
==  Carri GPU cluster ==
+
'''SuperMicro 4029GP-TVRT''' : 1 node ('''nefgpu41''' )
  
 +
[[Image:SYS-4028GR-TVRT.jpg|200px|right]]
  
−
'''Carri HighStation 5600 XLR8''' nodes: dual-Xeon X5650 @ 2.66Ghz : 2 nodes
+
* 4x Nvidia Tesla V100 SXM2 GPUs cards
 +
**        5120 CUDA cores per card
 +
**        640 tensor cores per card
 +
**        32GB of RAM capacity per card
 +
**        Simple precision performance peak: 15.7 Tflops per card
 +
**        Double precision performance peak: 7.8 Tflops per card
 +
**        Tensor performance peak : 125 TFlops per card
 +
**        900 GB/s GPU memory bandwidth with error correction (ECC)
 +
*    CPU-GPU connection with a PCIe gen3 16x interface
 +
*    GPU-GPU connection with NVLink 2.0 (25GB/s per direction per link, 1 or 2 links per card pair)
 +
*    CPU : 2x Xeon SP Gold 6126 @ 2.6 GHz (24 cores)
 +
*    RAM capacity : 384 GB
 +
*    storage :
 +
**        system & /tmp : 240GB SSD SATA RAID-1
 +
**        /local/tmp scratch data : 4.8 TB SSD SATA RAID-5
 +
**        RAID controller SAS 12Gb/s
 +
*    2x 10Gbps ethernet ports (one connected in 1Gb/s)
 +
*    1x infiniband FDR card
 +
*    hyperthreading active
  
−
[[Image:Carri-tesla.jpg|200px|right]]
+
== Asus ESC8000 GPU node (2018) ==
  
−
*    RAM capacity: 72 GB
+
'''Asus ESC8000G4''' / Carri HighServer nodes : 1 node ('''nefgpu40''' )
−
*    2x160GB SSD disks
+
* 8x GeForce GTX 1080 Ti GPUs cards connected with a PCIe gen3 16x interface
 +
**        3584 CUDA cores per card
 +
**        11GB of RAM capacity per card
 +
**        Simple precision performance peak: 10.6 Tflops per card
 +
**        Double precision performance peak: 0.3 Tflops per card
 +
**        484 GB/s GPU memory bandwidth
 +
*    PCIe single-root topology (2 PCIe 96 lane switches)
 +
**        topology can be software modified (BIOS & reboot) to dual-root for an experiment campaign
 +
*    CPU : 2x Xeon SP Gold 5115 @ 2.4 GHz
 +
*    RAM capacity : 256 GB
 +
*    storage :
 +
**        system & /tmp : RAID-1 2x512 GB SATA SSD
 +
**        /local/tmp scratch data : RAID-0 4 TB SATA SSD
 +
**        RAID controller SAS 12Gb/s
 
*    4x gigabit ethernet ports (one connected)
 
*    4x gigabit ethernet ports (one connected)
−
*    1x infiniband QDR card
+
*    1x infiniband FDR card
−
*    7 Tesla C2070 (nefgpu03)/ 7 Tesla C2050(nefgpu04) GPUs connected with a PCIe gen2 16x interface
+
*    hyperthreading active
−
**        448 Streaming Processor Cores per card
 
−
**        Simple precision performance peak: 1.03 Tflops per card
 
−
**        Double precision performance peak: 0.51 Tflops per card
 
−
**        3GB of RAM capacity per card (6GB on nefgpu03)
 
−
 
 
  
−
==  Dell C6100/C410x GPU cluster ==
+
== Dell R740 GPU nodes (2019) ==
  
 +
'''Dell R740''' nodes: dual-Xeon Skylake or CascadeLake SP : 5 nodes ( '''nefgpu42 to nefgpu46''')
 +
* Nvidia Tesla V100 PCIe GPUs cards
 +
**        5120 CUDA cores per card
 +
**        640 tensor cores per card
 +
**        32GB of RAM capacity per card
 +
**        Tensor performance peak : 112 TFlops per card
 +
**        Simple precision performance peak: 14 Tflops per card
 +
**        Double precision performance peak: 7 Tflops per card
 +
**        900 GB/s GPU memory bandwidth with error correction (ECC)
 +
* Nvidia Tesla T4 GPUs cards connected with a PCIe gen3 16x interface
 +
**        2560 CUDA cores per card
 +
**        320 tensor cores per card
 +
**        16GB of RAM capacity per card
 +
**        Simple precision performance peak: 8.1 Tflops per card
 +
**        Double precision performance peak: 0.3 Tflops per card
 +
**        300 GB/s GPU memory bandwidth with error correction (ECC)
 +
*    2-4x ethernet ports (one connected in gigabit)
 +
*    1x infiniband FDR card
 +
*    hyperthreading active
  
−
'''Dell C6100''' nodes: mono-Xeon X5650 @ 2.66Ghz : 2 nodes
+
{| class="wikitable"
 +
|+Node details
 +
|-
 +
| '''Node name'''
 +
| '''Funding team'''
 +
| '''GPU cards'''
 +
| '''Node CPU'''
 +
| '''Node RAM'''
 +
| '''Node storage'''
 +
|-
 +
| nefgpu42
 +
| common
 +
| 3x Tesla T4
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 2x 480GB SSD
 +
|-
 +
| nefgpu43
 +
| common
 +
| 3x Tesla T4
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 2x 480GB SSD
 +
|-
 +
| nefgpu44
 +
| common
 +
| 3x Tesla T4
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 2x 480GB SSD
 +
|-
 +
| nefgpu45
 +
| common
 +
| 3x Tesla T4
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 2x 480GB SSD
 +
|-
 +
| nefgpu46
 +
| STARS
 +
| 3x Tesla V100
 +
| 2x Xeon Silver 4215
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 960GB SSD
 +
|}
  
−
[[Image:Dell-c410x.jpg|200px|right]]
+
== Dell T640 GPU nodes (2018-2019-2020) ==
  
−
*   RAM capacity: 24 GB
+
'''Dell T640''' nodes: dual-Xeon Skylake or CascadeLake SP : 21 nodes ( '''nefgpu{24-39} and nefgpu{47-51}''')
−
*   1x250GB SATA disk
+
* Nvidia RTX A6000 GPUs cards connected with a PCIe gen3 16x interface (Ampere)
−
*   2x gigabit ethernet ports (one connected)
+
**        10752 CUDA cores per card
−
*   1x infiniband QDR card
+
**        336 tensor cores per card
−
*   2 Tesla M2050 GPUs connected with a PCIe gen2 16x interface
+
**       48GB of RAM capacity per card
−
**        448 Streaming Processor Cores per card
+
**        Tensor performance peak : 309.7 TFlops per card
−
**        Simple precision performance peak: 1.03 Tflops per card
+
**        Simple precision performance peak: 38.7 Tflops per card
−
**        Double precision performance peak: 0.51 Tflops per card
+
**        768 GB/s GPU memory bandwidth with error correction (ECC)
−
**        3GB of RAM capacity per card  
+
* Nvidia Quadro RTX 6000 GPUs cards connected with a PCIe gen3 16x interface (Turing)
 +
**        4608 CUDA cores per card
 +
**        576 tensor cores per card
 +
**        24GB of RAM capacity per card
 +
**        Tensor performance peak : 130.5 TFlops per card
 +
**        Half precision performance peak: 32.6 Tflops per card
 +
**        Simple precision performance peak: 16.3 Tflops per card
 +
**        Double precision performance peak: 0.5 Tflops per card
 +
**        624 GB/s GPU memory bandwidth with error correction (ECC)
 +
**        optional NVLink 2.0 (2x25GB/s per direction) by pair of card
 +
* Nvidia Quadro RTX 8000 GPUs cards connected with a PCIe gen3 16x interface (Turing)
 +
**        4608 CUDA cores per card
 +
**        576 tensor cores per card
 +
**        48GB of RAM capacity per card
 +
**        Tensor performance peak : 130.5 TFlops per card
 +
**        Half precision performance peak: 32.6 Tflops per card
 +
**        Simple precision performance peak: 16.3 Tflops per card
 +
**        Double precision performance peak: 0.5 Tflops per card
 +
**        672 GB/s GPU memory bandwidth with error correction (ECC)
 +
**        optional NVLink 2.0 (2x25GB/s per direction) by pair of card
 +
* GeForce RTX 2080 Ti GPUs cards connected with a PCIe gen3 16x interface
 +
**        4352 CUDA cores per card
 +
**        544 tensor cores per card
 +
**        11GB of RAM capacity per card
 +
**        Half precision performance peak: 23.5 Tflops per card
 +
**        Simple precision performance peak: 11.8 Tflops per card
 +
**        Double precision performance peak: 0.4 Tflops per card
 +
**        616 GB/s GPU memory bandwidth
 +
* GeForce Titan RTX GPUs cards connected with a PCIe gen3 16x interface
 +
**        4608 CUDA cores per card
 +
**        576 tensor cores per card
 +
**        24GB of RAM capacity per card
 +
**        Half precision performance peak: 24.9 Tflops per card
 +
**        Simple precision performance peak: 12.4 Tflops per card
 +
**        Double precision performance peak: 0.4 Tflops per card
 +
**        672 GB/s GPU memory bandwidth
 +
**        Power limited to 210W to prevent overheat
 +
* GeForce GTX 1080 Ti GPUs cards connected with a PCIe gen3 16x interface
 +
**        3584 CUDA cores per card
 +
**        11GB of RAM capacity per card  
 +
**        Simple precision performance peak: 10.6 Tflops per card
 +
**        Double precision performance peak: 0.3 Tflops per card
 +
**        484 GB/s GPU memory bandwidth
 +
* GeForce GTX Titan X GPUs cards connected with a PCIe gen3 16x interface
 +
**        3072 CUDA cores per card
 +
**        12GB of RAM capacity per card  
 +
**        Simple precision performance peak: 7.0 Tflops per card
 +
**        Double precision performance peak: 0.2 Tflops per card
 +
**        336.5 GB/s GPU memory bandwidth
 +
*    2-4x ethernet ports (one connected in gigabit)
 +
*    1x infiniband FDR card
 +
*    hyperthreading active
  
−
=  new cluster - Storage =
+
{| class="wikitable"
 +
|+Node details
 +
|-
 +
| '''Node name'''
 +
| '''Funding team'''
 +
| '''GPU cards'''
 +
| '''Node CPU'''
 +
| '''Node RAM'''
 +
| '''Node storage'''
 +
|-
 +
| nefgpu24
 +
| EPIONE
 +
| 4x GTX 1080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local : RAID-0 1x 960GB SSD
 +
|-
 +
| nefgpu25
 +
| ZENITH
 +
| 4x GTX 1080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-0 4x 480GB SSD<br>/local/read : RAID-0 8x 480GB SSD read intensive
 +
|-
 +
| nefgpu26
 +
| WIMMICS
 +
| 4x GTX 1080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local : RAID-0 4x 480GB SSD
 +
|-
 +
| nefgpu27
 +
| ZENITH
 +
| 4x RTX 2080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-0 2x 400GB SSD<br>/local/read : RAID-0 4x 480GB SSD read intensive
 +
|-
 +
| nefgpu28
 +
| GRAPHDECO
 +
| 4x RTX A6000
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 400GB SSD
 +
|-
 +
| nefgpu29
 +
| EPIONE
 +
| 2x GTX Titan-X
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 480GB SSD
 +
|-
 +
| nefgpu30
 +
| common
 +
| 4x RTX 2080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 4x 480GB SSD
 +
|-
 +
| nefgpu31
 +
| common
 +
| 4x RTX 2080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 4x 480GB SSD
 +
|-
 +
| nefgpu32
 +
| common
 +
| 4x Titan RTX
 +
| 2x Xeon Silver 4110
 +
| 192 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : RAID-0 4x 480GB SSD
 +
|-
 +
| nefgpu33
 +
| CHORALE
 +
| 4x RTX 2080 Ti
 +
| 2x Xeon Silver 4110
 +
| 96 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-0 1x 480GB SSD
 +
|-
 +
| nefgpu34
 +
| GRAPHDECO
 +
| 4x RTX 6000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6240
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 960GB SSD
 +
|-
 +
| nefgpu35
 +
| STARS
 +
| 4x RTX 6000<br> w/ NVLink by pair
 +
| 2x Xeon Silver 4215
 +
| 192 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 960GB SSD
 +
|-
 +
| nefgpu36
 +
| STARS
 +
| 4x RTX 6000<br> w/ NVLink by pair
 +
| 2x Xeon Silver 4215
 +
| 192 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 960GB SSD
 +
|-
 +
| nefgpu37
 +
| GRAPHDECO
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6240
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 960GB SSD
 +
|-
 +
| nefgpu38
 +
| AYANA
 +
| 2x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Silver 4214R
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 960GB SSD
 +
|-
 +
| nefgpu39
 +
| STARS
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Silver 4216
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/mixed : RAID-5 5x 900GB SATA SSD
 +
|-
 +
| nefgpu47
 +
| I3S/SPARKS
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6230R
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 450GB SATA SSD<br>/local/mixed : RAID-5 5x 900GB SSD
 +
|-
 +
| nefgpu48
 +
| I3S/SPARKS
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6230R
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 450GB SATA SSD<br>/local/mixed : RAID-5 5x 900GB SSD
 +
|-
 +
| nefgpu49
 +
| I3S/SPARKS
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6238R
 +
| 768 GB
 +
| system & /tmp : RAID-1 2x 450GB SATA SSD<br>/local/mixed : RAID-5 5x 1.8TB SSD
 +
|-
 +
| nefgpu50
 +
| I3S/SPARKS
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6230R
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 450GB SATA SSD<br>/local/mixed : RAID-5 5x 900GB SSD
 +
|-
 +
| nefgpu51
 +
| I3S/SPARKS
 +
| 4x RTX 8000<br> w/ NVLink by pair
 +
| 2x Xeon Gold 6230R
 +
| 384 GB
 +
| system & /tmp : RAID-1 2x 450GB SATA SSD<br>/local/mixed : RAID-5 5x 900GB SSD
 +
|}
  
 +
== Dell T630 GPU nodes (2016-2017) ==
  
−
All nodes have access to common storage :
+
'''Dell T630''' nodes: dual-Xeon E5-26xx : 17 nodes ( '''nefgpu07 to nefgpu23''')
 +
* GeForce GTX 1080 Ti GPUs cards connected with a PCIe gen3 16x interface
 +
**        3584 CUDA cores per card
 +
**        11GB of RAM capacity per card
 +
**        Simple precision performance peak: 10.6 Tflops per card
 +
**        Double precision performance peak: 0.3 Tflops per card
 +
**        484 GB/s GPU memory bandwidth
 +
* GeForce GTX 1080 GPUs cards connected with a PCIe gen3 16x interface
 +
**        2560 CUDA cores per card
 +
**        8GB of RAM capacity per card
 +
**        Simple precision performance peak: 8.2 Tflops per card
 +
**        Double precision performance peak: 0.3 Tflops per card
 +
**        320 GB/s GPU memory bandwidth
 +
* GeForce GTX Titan X GPUs cards connected with a PCIe gen3 16x interface
 +
**        3072 CUDA cores per card
 +
**        12GB of RAM capacity per card
 +
**        Simple precision performance peak: 7.0 Tflops per card
 +
**        Double precision performance peak: 0.2 Tflops per card
 +
**        336.5 GB/s GPU memory bandwidth
 +
*    4x gigabit ethernet ports (one connected)
 +
*    1x infiniband FDR card
  
−
*  common storage : /home
+
{| class="wikitable"
−
**      '''7 TB''' (4 x 4TB SAS disks, RAID-10 array), infiniband QDR, NFS access, available to all users, quotas
+
|+Node details
−
**      to be extended : +12 TB in late 2015
+
|-
−
* legacy experimental scratch storage (/dfs) is to be removed in late 2015
+
| '''Node name'''
−
* capacity distributed and scalable common storage : /data
+
| '''Funding team'''
−
**      '''60TB''' , infiniband QDR, BeeGFS access
+
| '''GPU cards'''
−
**      to be extended : + 108 TB in late 2015
+
| '''Node CPU'''
−
**      to be extended : + 60 TB in early 2016
+
| '''Node RAM'''
−
**      permanent storage : 1TB quota per team + teams may buy additional quota (please contact helpdesk)
+
| '''Node storage'''
−
**      scratch storage : variable size (initially ~20TB), no quota limit, for transient storage (data may be purged)
+
| '''Hyper threading active ?'''
 +
|-
 +
| nefgpu07
 +
| EPIONE
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2620v3
 +
| 128 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
 +
| no
 +
|-
 +
| nefgpu08
 +
| ZENITH
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v3
 +
| 64 GB
 +
| system & /tmp : RAID-0 2x 300GB SAS 15kRPM<br>/local/mixed : 1x 800GB SSD<br>/local/read : RAID-0 2x 1.92TB SSD read intensive
 +
| no
 +
|-
 +
| nefgpu09
 +
| GRAPHDECO
 +
| 4x Titan X
 +
| 2x E5-2630v4
 +
| 48 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 400GB SSD
 +
| no
 +
|-
 +
| nefgpu10
 +
| STARS
 +
| 4x Titan X
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 1.6TB SSD
 +
| no
 +
|-
 +
| nefgpu11
 +
| STARS
 +
| 4x GTX 1080
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 1.6TB SSD
 +
| no
 +
|-
 +
| nefgpu12
 +
| STARS
 +
| 4x GTX 1080
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 1.6TB SSD
 +
| yes
 +
|-
 +
| nefgpu13
 +
| GRAPHDECO
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2650v4
 +
| 64 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 400GB SSD
 +
| yes
 +
|-
 +
| nefgpu14
 +
| STARS
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2620v4
 +
| 128 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 400GB SSD
 +
| yes
 +
|-
 +
| nefgpu15
 +
| STARS
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2620v4
 +
| 128 GB
 +
| system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM<br>/local : 1x 400GB SSD
 +
| yes
 +
|-
 +
| nefgpu16
 +
| EPIONE
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local : 1x 1.6TB SSD
 +
| yes
 +
|-
 +
| nefgpu17
 +
| ZENITH
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v4
 +
| 64 GB
 +
| system & /tmp : RAID-0 2x 600GB SAS 10kRPM<br>/local/mixed : 1x 1.6TB SSD<br>/local/read : RAID-0 2x 1.92TB SSD read intensive
 +
| yes
 +
|-
 +
| nefgpu18
 +
| common
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : 1x 1.6TB SSD
 +
| yes
 +
|-
 +
| nefgpu19
 +
| common
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : 1x 1.6TB SSD
 +
| yes
 +
|-
 +
| nefgpu20
 +
| common
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v4
 +
| 128 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local/tmp : 1x 1.6TB SSD
 +
| yes
 +
|-
 +
| nefgpu21
 +
| STARS
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2620v4
 +
| 128 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local : 1x 480GB SSD
 +
| yes
 +
|-
 +
| nefgpu22
 +
| STARS
 +
| 3x GTX 1080 Ti
 +
| 2x E5-2620v4
 +
| 128 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local : 1x 480GB SSD
 +
| yes
 +
|-
 +
| nefgpu23
 +
| TITANE-EPITOME
 +
| 4x GTX 1080 Ti
 +
| 2x E5-2630v4
 +
| 64 GB
 +
| system & /tmp : RAID-1 2x 600GB SAS 10kRPM<br>/local : 1x 1.6TB SSD
 +
| yes
 +
|}
  
 +
== Dell R730 GPU node (2016) ==
  
−
=  legacy cluster - Storage =
+
'''Dell R730''' nodes: dual-Xeon E5-2623v4 @ 2.6 GHz : 1 node ( '''nefgpu01''')
 +
* Tesla K80 GPU cards connected with a PCIe gen3 16x interface
 +
**        2x Tesla GK210 GPUs per card
 +
**        4992 CUDA cores per card
 +
**        2x 12GB RAM capacity per card with error correction (ECC)
 +
**        Simple precision performance peak: 5.61 Tflops per card
 +
**        Double precision performance peak: 1.87 Tflops per card
 +
**        2x 240 GB/s GPU memory bandwidth with error correction (ECC)
 +
*    4x gigabit ethernet ports (one connected)
 +
*    1x infiniband QDR card
 +
*    hyperthreading not active
  
 +
{| class="wikitable"
 +
|+Node details
 +
|-
 +
| '''Node name'''
 +
| '''Funding team'''
 +
| '''Number of GPU cards'''
 +
| '''Node CPU'''
 +
| '''Node RAM'''
 +
| '''Node storage'''
 +
|-
 +
| nefgpu01
 +
| MATHNEURO
 +
| 1x K80
 +
| 2x E5-2623v4
 +
| 32 GB
 +
| 2x 400GB SSD
 +
|}
  
−
All nodes have access to a common storage using NFS. The NFS server is accessed through the infiniband 40Gb network if available
+
= Storage  =
  
−
[[Image:Dell-md1000.jpg|200px|right]]
+
All nodes have access to common storage :
  
 +
* common storage : '''/home'''
 +
** '''31 TiB''', available to all users, quotas per user
 +
** 1 Dell PowerEdge R540 server with 2 RAID-10 array 10 x 10TB SAS 7.2 kRPM disks, infiniband FDR, NFS access
  
−
*   common storage : /home
+
* capacity distributed and scalable common storage : '''/data'''
−
**     '''7 TBytes''' (4 x 4TB SAS disks, RAID-10 array), NFS access, available to all users, quotas
+
**       '''~610TiB''' real capacity, 684TB theoretical capacity (06/2019)
−
*   experimental scratch storage based on distributed file system : /dfs
+
***      permanent storage : 1TiB quota per team + teams may buy additional quota (please contact [[Support|cluster administrators]])
−
**     19 TB, GlusterFS access, available to all users, no quotas (may change later)
+
***      scratch storage : variable size (initially ~40TiB), no quota limit, for temporary storage (data may be purged)
 +
**  BeeGFS filesystem on multiple hardware :
 +
*** 4 Dell PowerEdge R740xd ; 1.92TB metadata : RAID-1 array 2 x 1.92TB SSD mixed use MLC disks (or RAID-10 array 4 x 960GB) ; 2x {60 or 72}TB data : 2 x RAID-6 array 8 x {10 or 12}TB SAS 7.2 kRPM disks
 +
*** 2 Dell PowerEdge R730xd ; 800GB metadata : RAID-1 array 2 x 800 GB SSD mixed use MLC disks ; 2 x {36 or 48}TB data: 2 x RAID-6 array 8 x {6 or 8}TB SAS 7.2 kRPM disks
 +
*** infiniband FDR
  
−
*    legacy external storage ([http://www.dell.com/downloads/global/products/pvaul/fr/pvaul_md1000_specs_fr.pdf Dell MD1000])
+
More details about quotas at: [[User_Guide_new_config#_Disk_space_management|Disk space management]]
−
**        15 slots available, reserved for INRIA teams.
 
−
**        This storage is no longer under warranty, the only storage left is for [http://www-sop.inria.fr/teams/tropics TROPICS] 1TB RAID-1 .
 
−
**        Teams needing more storage should contact the cluster administrators
 
−
*    legacy external storage ([http://www.dell.com/fr/entreprise/p/powervault-md1200/pd Dell MD1200])
 
−
**        19TB for the [https://team.inria.fr/asclepios ASCLEPIOS] EPI
 
−
**        11TB for the [http://www-sop.inria.fr/morpheme MORPHEME] EPI
 

Version actuelle datée du 21 mars 2024 à 19:12



CPU clusters

Dell R7525 nodes (2020)

200px


Dell R7525 dual-EPYC 7502 @ 2.5GHz (64 cores) : 1 node ( nef059 )

  • RAM capacity : 384 GB RAM
  • storage : system 2x900 GB SATA SSD RAID-1 + local scratch data 7x7.5TB SAS HDD 44 TB RAID-5 + controller H745
  • 2x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active


Dell R7525 dual-EPYC 7542 @ 2.9GHz (64 cores) : 1 node ( nef058 )

  • RAM capacity : 1024 GB RAM
  • storage : system 2x223 GB SATA SSD RAID-1 + local scratch data 5x445GB SAS SSD 1.8 TB RAID-5 + controller H745
  • 2x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active

Dell C6420 cluster (2019)

Dell C6420 dual-Xeon Cascade Lake SP Gold 6240 @ 2.60GHz (36 cores) : 4 nodes ( nef054 to nef057 )

C6420.jpg


  • RAM capacity : 384 GB RAM
  • storage : system 2x600 GB SATA RAID-1 + local scratch data 960 BB SATA SSD RAID-0 + controller H330
  • 1x gigabit network port
  • 1x infiniband FDR card
  • hyperthreading active
  • optimal performance with AVX-512, AVX/AVX2 support

Dell R940 node (2017)

Dell R940 quad-Xeon SP Gold 6148 @ 2.40GHz (80 cores) : 1 node ( nef053 )

Dellr940.jpg


  • RAM capacity : 1024 GB RAM
  • storage : system 2x600 GB SATA RAID-1 + local scratch data 1.92 TB SATA SSD + controller H740P
  • 4x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active
  • optimal performance with AVX-512, AVX/AVX2 support

Dell C6420 cluster (2017)

Dell C6420 dual-Xeon Skylake SP Silver 4114 @ 2.20GHz (20 cores) : 16 nodes ( nef037 to nef052 )

C6420.jpg


  • RAM capacity : 192 GB RAM
  • 1x600GB 10kRPM SAS HardDisk drive
  • 1x gigabit network port
  • 1x infiniband FDR card
  • hyperthreading active
  • AVX-512 support, optimal performance with AVX/AVX2

Dell C6220 cluster (2015)

Dell C6220 dual-Xeon E5-2650 v2 @ 2.60GHz (16 cores) : 8 nodes ( nef029 to nef036 )

C6220-300x191.jpg


  • RAM capacity : 256 GB RAM
  • 1x1TB SATA HardDisk drive
  • 2x gigabit network ports (one connected)
  • 2x infiniband QDR card (one connected)
  • hyperthreading not active


Dell C6220 cluster (2014)

Dell C6220 dual-Xeon E5-2680 v2 @ 2.80GHz (20 cores) : 16 nodes ( nef013 to nef028 )

C6220-300x191.jpg


  • RAM capacity : 192 GB RAM
  • 1x2TB SATA HardDisk drive
  • 2x gigabit network ports (one connected)
  • 1x infiniband FDR card (QDR used)
  • hyperthreading not active


 Dell C6145 cluster (2013)

Dell C6145 quad-Opterons 6376 @ 2.3Ghz (64 cores) : 6 nodes ( nef007 to nef012 )

Cluster-c6145.jpg


  • RAM capacity : 256 GB RAM (512GB on nef011 and nef012)
  • 1x500GB SATA HardDisk drive
  • 2x gigabit network ports (one connected)
  • 1x infiniband QDR card
  • hyperthreading not supported


Dell R815 cluster (2010)

Dell R815 quad-Opterons 6174 @ 2.2Ghz (48 cores) : 6 nodes ( nef001 to nef006 )

Cluster-dellr815.jpg


  • RAM capacity : 256 GB RAM
  • 2x600GB SAS HardDisk drive (RAID-0)
  • 4x gigabit network ports (one connected)
  • 1x infiniband QDR card
  • hyperthreading not supported

GPU nodes

HPE DL385 GPU nodes (2023)

HPE DL385 dual-AMD EPYC 7313 @ 3.0GHz (16 cores) : 3 nodes ( nefgpu59 to nefgpu61 )

  • Nvidia A40 PCIe GPUs cards
    • 10752 CUDA cores per card
    • 336 tensor cores per card
    • 48GB of RAM capacity per card
    • Tensor performance peak : 149.6 TFlops per card
    • Simple precision performance peak: 74.8 Tflops per card
    • Double precision performance peak: 37.4 Tflops per card
    • 696 GB/s GPU memory bandwidth with error correction (ECC)
  • 2x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active

HPE DL385 dual-AMD EPYC 7513 @ 2.6GHz (32 cores) : 2 nodes ( nefgpu57 to nefgpu58 )

  • Nvidia A100 - 80GB PCIe GPUs cards
    • 6912 CUDA cores per card
    • 432 tensor cores per card
    • 80GB of RAM capacity per card
    • Tensor performance peak : 312 TFlops per card
    • FP16 Compute: 78 Tflops per card
    • FP32 Compute: 19.5 Tflops per card
    • FP64 Compute: 9.7 Tflops per card
    • 1935GB/s GPU memory bandwidth with error correction (ECC)
  • 2x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active

HPE DL385 dual-AMD EPYC 7662 @ 2.0GHz (64 cores) : 1 node ( nefgpu56 )

  • Nvidia A100 - 40GB PCIe GPUs cards
    • 6912 CUDA cores per card
    • 432 tensor cores per card
    • 40GB of RAM capacity per card
    • Tensor performance peak : 312 TFlops per card
    • Simple precision performance peak: 156 Tflops per card
    • Double precision performance peak: 19.49 Tflops per card
    • 1555 GB/s GPU memory bandwidth with error correction (ECC)
  • 2x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active
Node details
Node name Funding team GPU cards Node CPU Node RAM Node storage
nefgpu56 STARS 3x A100 40GB 2x AMD EPYC 7662 1024 GB system & /tmp : RAID-1 2x 480SSD
/local/mixed : RAID-5 5x 960 SSD
nefgpu57 MAASAI 2x A100 80GB 2x AMD EPYC 7513 512 GB system & /local/mixed : RAID-1 2x 480SSD
/local/mixed : RAID-0 6x 960 SSD
nefgpu58 ZENITH 2x A100 80GB 2x AMD EPYC 7513 512 GB system & /local/mixed : RAID-1 2x 480SSD
/local/mixed : RAID-0 6x 960 SSD
nefgpu59 MORPHEME 3x A40 2x AMD EPYC 7313 512 GB system & /local/mixed : RAID-1 2x 480SSD
/local/mixed : RAID-0 4x 960 SSD
nefgpu60 ACUMES 2x A40 2x AMD EPYC 7313 256 GB system & /local/mixed : RAID-1 2x 480SSD
/local/mixed : RAID-0 4x 960 SSD
nefgpu61 ACUMES 2x A40 2x AMD EPYC 7313 256 GB system & /local/mixed : RAID-1 2x 480SSD
/local/mixed : RAID-0 4x 960 SSD

Dell R7525 GPU nodes (2020)

Dell R7525 dual-AMD EPYC 7282 @ 2.8GHz (16 cores) : 2 nodes ( nefgpu52 to nefgpu53 )
Dell R7525 dual-AMD EPYC 7413 @ 2.65GHz (24 cores) : 2 nodes ( nefgpu54 to nefgpu55 )

  • Nvidia A40 PCIe GPUs cards
    • 10752 CUDA cores per card
    • 336 tensor cores per card
    • 48GB of RAM capacity per card
    • Tensor performance peak : 149.6 TFlops per card
    • Simple precision performance peak: 74.8 Tflops per card
    • Double precision performance peak: 37.4 Tflops per card
    • 696 GB/s GPU memory bandwidth with error correction (ECC)
  • 2x gigabit network ports (one connected)
  • infiniband EDR card (connected to FDR switch)
  • hyperthreading active
Node details
Node name Funding team GPU cards Node CPU Node RAM Node storage
nefgpu52 NEO 3x A40 2x AMD EPYC 7282 256 GB system & /tmp : RAID-1 2x 480SSD
/local/mixed : RAID-5 5x 480 SSD
nefgpu53 ACENTAURI 3x A40 2x AMD EPYC 7282 256 GB system & /tmp : RAID-1 2x 480SSD
/local/mixed : RAID-5 5x 480 SSD
nefgpu54 3IA 3x A40 2x AMD EPYC 7413 512 GB system & /tmp : RAID-1 2x 480SSD
/local/mixed : RAID-5 5x 960 SSD
nefgpu55 3IA 3x A40 2x AMD EPYC 7413 512 GB system & /tmp : RAID-1 2x 480SSD
/local/mixed : RAID-5 5x 960 SSD


SuperMicro 4029 node (2019)

SuperMicro 4029GP-TVRT : 1 node (nefgpu41 )

SYS-4028GR-TVRT.jpg
  • 4x Nvidia Tesla V100 SXM2 GPUs cards
    • 5120 CUDA cores per card
    • 640 tensor cores per card
    • 32GB of RAM capacity per card
    • Simple precision performance peak: 15.7 Tflops per card
    • Double precision performance peak: 7.8 Tflops per card
    • Tensor performance peak : 125 TFlops per card
    • 900 GB/s GPU memory bandwidth with error correction (ECC)
  • CPU-GPU connection with a PCIe gen3 16x interface
  • GPU-GPU connection with NVLink 2.0 (25GB/s per direction per link, 1 or 2 links per card pair)
  • CPU : 2x Xeon SP Gold 6126 @ 2.6 GHz (24 cores)
  • RAM capacity : 384 GB
  • storage :
    • system & /tmp : 240GB SSD SATA RAID-1
    • /local/tmp scratch data : 4.8 TB SSD SATA RAID-5
    • RAID controller SAS 12Gb/s
  • 2x 10Gbps ethernet ports (one connected in 1Gb/s)
  • 1x infiniband FDR card
  • hyperthreading active

Asus ESC8000 GPU node (2018)

Asus ESC8000G4 / Carri HighServer nodes : 1 node (nefgpu40 )

  • 8x GeForce GTX 1080 Ti GPUs cards connected with a PCIe gen3 16x interface
    • 3584 CUDA cores per card
    • 11GB of RAM capacity per card
    • Simple precision performance peak: 10.6 Tflops per card
    • Double precision performance peak: 0.3 Tflops per card
    • 484 GB/s GPU memory bandwidth
  • PCIe single-root topology (2 PCIe 96 lane switches)
    • topology can be software modified (BIOS & reboot) to dual-root for an experiment campaign
  • CPU : 2x Xeon SP Gold 5115 @ 2.4 GHz
  • RAM capacity : 256 GB
  • storage :
    • system & /tmp : RAID-1 2x512 GB SATA SSD
    • /local/tmp scratch data : RAID-0 4 TB SATA SSD
    • RAID controller SAS 12Gb/s
  • 4x gigabit ethernet ports (one connected)
  • 1x infiniband FDR card
  • hyperthreading active

Dell R740 GPU nodes (2019)

Dell R740 nodes: dual-Xeon Skylake or CascadeLake SP : 5 nodes ( nefgpu42 to nefgpu46)

  • Nvidia Tesla V100 PCIe GPUs cards
    • 5120 CUDA cores per card
    • 640 tensor cores per card
    • 32GB of RAM capacity per card
    • Tensor performance peak : 112 TFlops per card
    • Simple precision performance peak: 14 Tflops per card
    • Double precision performance peak: 7 Tflops per card
    • 900 GB/s GPU memory bandwidth with error correction (ECC)
  • Nvidia Tesla T4 GPUs cards connected with a PCIe gen3 16x interface
    • 2560 CUDA cores per card
    • 320 tensor cores per card
    • 16GB of RAM capacity per card
    • Simple precision performance peak: 8.1 Tflops per card
    • Double precision performance peak: 0.3 Tflops per card
    • 300 GB/s GPU memory bandwidth with error correction (ECC)
  • 2-4x ethernet ports (one connected in gigabit)
  • 1x infiniband FDR card
  • hyperthreading active
Node details
Node name Funding team GPU cards Node CPU Node RAM Node storage
nefgpu42 common 3x Tesla T4 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 2x 480GB SSD
nefgpu43 common 3x Tesla T4 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 2x 480GB SSD
nefgpu44 common 3x Tesla T4 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 2x 480GB SSD
nefgpu45 common 3x Tesla T4 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 2x 480GB SSD
nefgpu46 STARS 3x Tesla V100 2x Xeon Silver 4215 384 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 960GB SSD

Dell T640 GPU nodes (2018-2019-2020)

Dell T640 nodes: dual-Xeon Skylake or CascadeLake SP : 21 nodes ( nefgpu{24-39} and nefgpu{47-51})

  • Nvidia RTX A6000 GPUs cards connected with a PCIe gen3 16x interface (Ampere)
    • 10752 CUDA cores per card
    • 336 tensor cores per card
    • 48GB of RAM capacity per card
    • Tensor performance peak : 309.7 TFlops per card
    • Simple precision performance peak: 38.7 Tflops per card
    • 768 GB/s GPU memory bandwidth with error correction (ECC)
  • Nvidia Quadro RTX 6000 GPUs cards connected with a PCIe gen3 16x interface (Turing)
    • 4608 CUDA cores per card
    • 576 tensor cores per card
    • 24GB of RAM capacity per card
    • Tensor performance peak : 130.5 TFlops per card
    • Half precision performance peak: 32.6 Tflops per card
    • Simple precision performance peak: 16.3 Tflops per card
    • Double precision performance peak: 0.5 Tflops per card
    • 624 GB/s GPU memory bandwidth with error correction (ECC)
    • optional NVLink 2.0 (2x25GB/s per direction) by pair of card
  • Nvidia Quadro RTX 8000 GPUs cards connected with a PCIe gen3 16x interface (Turing)
    • 4608 CUDA cores per card
    • 576 tensor cores per card
    • 48GB of RAM capacity per card
    • Tensor performance peak : 130.5 TFlops per card
    • Half precision performance peak: 32.6 Tflops per card
    • Simple precision performance peak: 16.3 Tflops per card
    • Double precision performance peak: 0.5 Tflops per card
    • 672 GB/s GPU memory bandwidth with error correction (ECC)
    • optional NVLink 2.0 (2x25GB/s per direction) by pair of card
  • GeForce RTX 2080 Ti GPUs cards connected with a PCIe gen3 16x interface
    • 4352 CUDA cores per card
    • 544 tensor cores per card
    • 11GB of RAM capacity per card
    • Half precision performance peak: 23.5 Tflops per card
    • Simple precision performance peak: 11.8 Tflops per card
    • Double precision performance peak: 0.4 Tflops per card
    • 616 GB/s GPU memory bandwidth
  • GeForce Titan RTX GPUs cards connected with a PCIe gen3 16x interface
    • 4608 CUDA cores per card
    • 576 tensor cores per card
    • 24GB of RAM capacity per card
    • Half precision performance peak: 24.9 Tflops per card
    • Simple precision performance peak: 12.4 Tflops per card
    • Double precision performance peak: 0.4 Tflops per card
    • 672 GB/s GPU memory bandwidth
    • Power limited to 210W to prevent overheat
  • GeForce GTX 1080 Ti GPUs cards connected with a PCIe gen3 16x interface
    • 3584 CUDA cores per card
    • 11GB of RAM capacity per card
    • Simple precision performance peak: 10.6 Tflops per card
    • Double precision performance peak: 0.3 Tflops per card
    • 484 GB/s GPU memory bandwidth
  • GeForce GTX Titan X GPUs cards connected with a PCIe gen3 16x interface
    • 3072 CUDA cores per card
    • 12GB of RAM capacity per card
    • Simple precision performance peak: 7.0 Tflops per card
    • Double precision performance peak: 0.2 Tflops per card
    • 336.5 GB/s GPU memory bandwidth
  • 2-4x ethernet ports (one connected in gigabit)
  • 1x infiniband FDR card
  • hyperthreading active
Node details
Node name Funding team GPU cards Node CPU Node RAM Node storage
nefgpu24 EPIONE 4x GTX 1080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local : RAID-0 1x 960GB SSD
nefgpu25 ZENITH 4x GTX 1080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-0 4x 480GB SSD
/local/read : RAID-0 8x 480GB SSD read intensive
nefgpu26 WIMMICS 4x GTX 1080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local : RAID-0 4x 480GB SSD
nefgpu27 ZENITH 4x RTX 2080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-0 2x 400GB SSD
/local/read : RAID-0 4x 480GB SSD read intensive
nefgpu28 GRAPHDECO 4x RTX A6000 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 400GB SSD
nefgpu29 EPIONE 2x GTX Titan-X 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 480GB SSD
nefgpu30 common 4x RTX 2080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 4x 480GB SSD
nefgpu31 common 4x RTX 2080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 4x 480GB SSD
nefgpu32 common 4x Titan RTX 2x Xeon Silver 4110 192 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : RAID-0 4x 480GB SSD
nefgpu33 CHORALE 4x RTX 2080 Ti 2x Xeon Silver 4110 96 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-0 1x 480GB SSD
nefgpu34 GRAPHDECO 4x RTX 6000
w/ NVLink by pair 		2x Xeon Gold 6240 384 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 960GB SSD
nefgpu35 STARS 4x RTX 6000
w/ NVLink by pair 		2x Xeon Silver 4215 192 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 960GB SSD
nefgpu36 STARS 4x RTX 6000
w/ NVLink by pair 		2x Xeon Silver 4215 192 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 960GB SSD
nefgpu37 GRAPHDECO 4x RTX 8000
w/ NVLink by pair 		2x Xeon Gold 6240 384 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 960GB SSD
nefgpu38 AYANA 2x RTX 8000
w/ NVLink by pair 		2x Xeon Silver 4214R 384 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 960GB SSD
nefgpu39 STARS 4x RTX 8000
w/ NVLink by pair 		2x Xeon Silver 4216 384 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/mixed : RAID-5 5x 900GB SATA SSD
nefgpu47 I3S/SPARKS 4x RTX 8000
w/ NVLink by pair 		2x Xeon Gold 6230R 384 GB system & /tmp : RAID-1 2x 450GB SATA SSD
/local/mixed : RAID-5 5x 900GB SSD
nefgpu48 I3S/SPARKS 4x RTX 8000
w/ NVLink by pair 		2x Xeon Gold 6230R 384 GB system & /tmp : RAID-1 2x 450GB SATA SSD
/local/mixed : RAID-5 5x 900GB SSD
nefgpu49 I3S/SPARKS 4x RTX 8000
w/ NVLink by pair 		2x Xeon Gold 6238R 768 GB system & /tmp : RAID-1 2x 450GB SATA SSD
/local/mixed : RAID-5 5x 1.8TB SSD
nefgpu50 I3S/SPARKS 4x RTX 8000
w/ NVLink by pair 		2x Xeon Gold 6230R 384 GB system & /tmp : RAID-1 2x 450GB SATA SSD
/local/mixed : RAID-5 5x 900GB SSD
nefgpu51 I3S/SPARKS 4x RTX 8000
w/ NVLink by pair 		2x Xeon Gold 6230R 384 GB system & /tmp : RAID-1 2x 450GB SATA SSD
/local/mixed : RAID-5 5x 900GB SSD

Dell T630 GPU nodes (2016-2017)

Dell T630 nodes: dual-Xeon E5-26xx : 17 nodes ( nefgpu07 to nefgpu23)

  • GeForce GTX 1080 Ti GPUs cards connected with a PCIe gen3 16x interface
    • 3584 CUDA cores per card
    • 11GB of RAM capacity per card
    • Simple precision performance peak: 10.6 Tflops per card
    • Double precision performance peak: 0.3 Tflops per card
    • 484 GB/s GPU memory bandwidth
  • GeForce GTX 1080 GPUs cards connected with a PCIe gen3 16x interface
    • 2560 CUDA cores per card
    • 8GB of RAM capacity per card
    • Simple precision performance peak: 8.2 Tflops per card
    • Double precision performance peak: 0.3 Tflops per card
    • 320 GB/s GPU memory bandwidth
  • GeForce GTX Titan X GPUs cards connected with a PCIe gen3 16x interface
    • 3072 CUDA cores per card
    • 12GB of RAM capacity per card
    • Simple precision performance peak: 7.0 Tflops per card
    • Double precision performance peak: 0.2 Tflops per card
    • 336.5 GB/s GPU memory bandwidth
  • 4x gigabit ethernet ports (one connected)
  • 1x infiniband FDR card
Node details
Node name Funding team GPU cards Node CPU Node RAM Node storage Hyper threading active ?
nefgpu07 EPIONE 4x GTX 1080 Ti 2x E5-2620v3 128 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM no
nefgpu08 ZENITH 4x GTX 1080 Ti 2x E5-2630v3 64 GB system & /tmp : RAID-0 2x 300GB SAS 15kRPM
/local/mixed : 1x 800GB SSD
/local/read : RAID-0 2x 1.92TB SSD read intensive 		no
nefgpu09 GRAPHDECO 4x Titan X 2x E5-2630v4 48 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 400GB SSD 		no
nefgpu10 STARS 4x Titan X 2x E5-2630v4 128 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 1.6TB SSD 		no
nefgpu11 STARS 4x GTX 1080 2x E5-2630v4 128 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 1.6TB SSD 		no
nefgpu12 STARS 4x GTX 1080 2x E5-2630v4 128 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 1.6TB SSD 		yes
nefgpu13 GRAPHDECO 4x GTX 1080 Ti 2x E5-2650v4 64 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 400GB SSD 		yes
nefgpu14 STARS 4x GTX 1080 Ti 2x E5-2620v4 128 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 400GB SSD 		yes
nefgpu15 STARS 4x GTX 1080 Ti 2x E5-2620v4 128 GB system & /tmp : RAID-0 2x 1TB SATA 7.2kRPM
/local : 1x 400GB SSD 		yes
nefgpu16 EPIONE 4x GTX 1080 Ti 2x E5-2630v4 128 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local : 1x 1.6TB SSD 		yes
nefgpu17 ZENITH 4x GTX 1080 Ti 2x E5-2630v4 64 GB system & /tmp : RAID-0 2x 600GB SAS 10kRPM
/local/mixed : 1x 1.6TB SSD
/local/read : RAID-0 2x 1.92TB SSD read intensive 		yes
nefgpu18 common 4x GTX 1080 Ti 2x E5-2630v4 128 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : 1x 1.6TB SSD 		yes
nefgpu19 common 4x GTX 1080 Ti 2x E5-2630v4 128 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : 1x 1.6TB SSD 		yes
nefgpu20 common 4x GTX 1080 Ti 2x E5-2630v4 128 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local/tmp : 1x 1.6TB SSD 		yes
nefgpu21 STARS 4x GTX 1080 Ti 2x E5-2620v4 128 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local : 1x 480GB SSD 		yes
nefgpu22 STARS 3x GTX 1080 Ti 2x E5-2620v4 128 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local : 1x 480GB SSD 		yes
nefgpu23 TITANE-EPITOME 4x GTX 1080 Ti 2x E5-2630v4 64 GB system & /tmp : RAID-1 2x 600GB SAS 10kRPM
/local : 1x 1.6TB SSD 		yes

Dell R730 GPU node (2016)

Dell R730 nodes: dual-Xeon E5-2623v4 @ 2.6 GHz : 1 node ( nefgpu01)

  • Tesla K80 GPU cards connected with a PCIe gen3 16x interface
    • 2x Tesla GK210 GPUs per card
    • 4992 CUDA cores per card
    • 2x 12GB RAM capacity per card with error correction (ECC)
    • Simple precision performance peak: 5.61 Tflops per card
    • Double precision performance peak: 1.87 Tflops per card
    • 2x 240 GB/s GPU memory bandwidth with error correction (ECC)
  • 4x gigabit ethernet ports (one connected)
  • 1x infiniband QDR card
  • hyperthreading not active
Node details
Node name Funding team Number of GPU cards Node CPU Node RAM Node storage
nefgpu01 MATHNEURO 1x K80 2x E5-2623v4 32 GB 2x 400GB SSD

 Storage

All nodes have access to common storage :

  • common storage : /home
    • 31 TiB, available to all users, quotas per user
    • 1 Dell PowerEdge R540 server with 2 RAID-10 array 10 x 10TB SAS 7.2 kRPM disks, infiniband FDR, NFS access
  • capacity distributed and scalable common storage : /data
    • ~610TiB real capacity, 684TB theoretical capacity (06/2019)
      • permanent storage : 1TiB quota per team + teams may buy additional quota (please contact cluster administrators)
      • scratch storage : variable size (initially ~40TiB), no quota limit, for temporary storage (data may be purged)
    • BeeGFS filesystem on multiple hardware :
      • 4 Dell PowerEdge R740xd ; 1.92TB metadata : RAID-1 array 2 x 1.92TB SSD mixed use MLC disks (or RAID-10 array 4 x 960GB) ; 2x {60 or 72}TB data : 2 x RAID-6 array 8 x {10 or 12}TB SAS 7.2 kRPM disks
      • 2 Dell PowerEdge R730xd ; 800GB metadata : RAID-1 array 2 x 800 GB SSD mixed use MLC disks ; 2 x {36 or 48}TB data: 2 x RAID-6 array 8 x {6 or 8}TB SAS 7.2 kRPM disks
      • infiniband FDR

More details about quotas at: Disk space management

Récupérée de « https://wiki.inria.fr/wikis/ClustersSophia/index.php?title=Hardware&oldid=1132 »