Question

Question 2. Please describe the media combinations you have deployed; e.g., RAID-5, RAID-6, JBOD, networked cluster and filesystem (e.g., GPFS, Ceph, HDFS). If you have multiple media types in Question 1, please also briefly describe how the different underlying media are combined or virtualized.Please also give the (approximate) total user-available capacity for each media combination. This is sometimes called the “netto capacity” or “usable capacity”. For example, a RAID-6 system with 12 drives, each drive having 1 TiB capacity, would have a user-available capacity of 10 TiB.

Answers

CERN

EOS

JBOD deployment. Data is replicated twice, so usable = raw / 2.

Ceph

Block storage Standard Ceph deployment - triple replica Erasure encoding 2+1 on all-flash block instance S3 S3 is 4+2 erasure encoded. Rados Rados object interface uses EC Cephfs Mix of replication on flash (metadata) and EC (data)

Castor

RAID1 ~15PB usable

hephy-Vienna

Currently HW RAID-6 in simple Diskservers with XFS filesystem. User Directories are based on OpenAFS. We plan to move to EOS, with JBODs and Erasure Encoding.

KI-LT2-QMUL

storage presented as 4.8 PB Lustre file system using raid 6 in various configurations on different servers (10, 12 and 16 disk Raid 6 arrays)

UKI-LT2-RHUL

RAID6 1.4 PB.

RO-13-ISS

RAID6, 1.3 PiB

Nebraska

All our storage media are configured in JBODs with HDFS handling data management. Our total net capacity is a bit under 3.4 PB. Varying replication factors in different parts of the name space make the exact usable size indeterminate.

INFN-ROMA1

The bulk of disk space is using RAID-6 arrays, combined to serve our Grid (DPM) storage (~1.6 TB net space), ceph clusters (~200 TB net space) and gluster (~100 TB net space). Gluster and ceph are used for virtualization (oVirt, openstack), while ceph is also used as posix fs for users and S3-compatible for different backend types.

NDGF-T1

We don't see that directly, but typically 10-16 drive RAID-6 systems in most places with 1-4 raidsets per server. Ceph at one or two sites under commissioning, but it has not shown to be stable enough for production yet. Total net capacity 12.5PB.

BEgrid-ULB-VUB

MASS STORAGE: * a few RAID6 per node, formatted in xfs * from 22TB / raid to 55.6TB / raid usable * Total capacity: 5.2PB USER & VM STORAGE: * zfs mirrored on Solaris-like (Syneto) * HA system, each server controls a JBOD, data is mirrored between the 2 storage nodes * Total capacity: 120TB for users and 30TB for VMs

NCG-INGRID-PT

RAID-5 (volumes of 12 disks) or ZFS (raid-z of 12 disks) with Lustre; JBOD with Ceph (3 replicas). Ceph is used to support OpenStack cloud VMs. The main storage spaces for the Tier-2 are on Lustre.

IN2P3-IRES

To provide this raw capaticy, we are using two types of system: - A combination of RAID6 (netto 2.5 PB) - A CEPH cluster (netto 300 TB)

LRZ-LMU

Raid-6, 80 TB

CA-WATERLOO-T2

Mostly RAID6, especially for dCache storage. Have worked with GPFS on previous installation (CA-SCINET-T2), based on large DDN servers. New clusters using Lustre.

CA-VICTORIA-WESTGRID-T2

RAID 6. 576 TB + 720 TB usable

Taiwan_LCG2

1. We have DPM cluster which uses ZFS RAIDZ-3 , the total user capacity so far is 5.69P 2. We have two CEPH clusters, each of them has ~1PB user capacity. Ceph OSD uses raid-0 to stripe three hard drives to enhance the throughput. On top of that, we use multiple replicas in CEPH pool to protect data. 3. We have a small EOS cluster with RAID5 to provide 735.6 TB user capacity.

IN2P3-SUBATECH

RAID6 with 12-16 disks, usable capacity 1504TB, partitions xfs

asd

MPPMU

INFN-LNL-2

Older systems have RAID-6 arrays, newer ones "distributed parity" arrays. Each array is mounted on a server and formatted with XFS file-system. All these XFS partitions are then aggregated via dCache or xrootd. Total net capacity is ~5000 TB

Australia-ATLAS

RAID-6

SiGNET

Raid-6 for Storage Element, simple replication size=2 for Ceph, size=3 for Ceph metadata using bluestore and MDS on SSDs.

KR-KISTI-GSDC-02

RAID6 about 800TB

UKI-LT2-IC-HEP

RAID6 8PB

BelGrid-UCL

RAIDZ2 over JBOD

UKI-SOUTHGRID-BRIS-HEP

JBOD, HDFS, 0.65 PB usable capacity

GR-07-UOI-HEPLAB

no RAID used

UKI-SOUTHGRID-CAM-HEP

RAID6, xfs filesystem. 240TB

USC-LCG2

It is a single disk

EELA-UTFSM

RAID-6: 28TB; RAID-6 (ZFS raidz2): 440TB

DESY-ZN

RAID-6, 100TB netto per system

PSNC

RAID5/6 , HDD : 2 PiB in total

UAM-LCG2

RAID-6. Capacity: 1.6PiB

T2_HU_BUDAPEST

RAID-5 XFS 1.5 PiB

INFN-Bari

RAID6

IEPSAS-Kosice

dCache RAID-6 with 288 drivers , each having 2 or 3 or 4 or 6 TB capacity would have a user-available capacity of 820 TB XROOTD RAID-6 with 144 drivers , each having 2 or 3 or 4 or 6 TB capacity would have a user-available capacity of 420 TB EOS RAID-0 JBOD with 48 drivers , each having 4TB capacity a total media capacity 192 TB and 72 drivers , each having 6 TB capacity a total media would have a user-available capacity of 528 TB

IN2P3-CC

All disk server for WLCG are configured with RAID6 on 12 disks, for a capacity of 24 374 TiB of usable capacity

NONE_DUMMY

blah

WEIZMANN-LCG2

DDN7000 RAID6 (8+2) Net 1PB

RU-SPbSU

RAID-6 + 2 HOT SPARE

USCMS_FNAL_WC1

Exclusively RAID6 -- see prev. answer

RRC-KI-T1

SAS 2GiB in SAS enclosures - RAID-6, ~320 TiB user-available capacity SAS 2GiB in iSCSI enclosures - RAID-6, ~460 TiB user-available capacity SATA 2GiB in 3U storage servers - RAID-6, ~ 672 TiB user-available capacity SATA 6GiB in 3U storage servers - RAID-6, ~ 6384 TiB user-available capacity SATA 10GiB in 3U storage servers - RAID-6, ~ 4200 TiB user-available capacity tapes without compresion ~15PB user-available capacity

vanderbilt

We use https://www.lstore.org/ as the filesystem and encode with 6+3 data+parity

UNIBE-LHEP

1132 TB (SE) RAID6, 229 TB (ARC cache) JBOD/RAID10, lustre

CA-SFU-T2

RAID-6, 3.3 PB

_CSCS-LCG2

Using distributed RAID and RAID6. Spectrum Scale is used as filesystem

T2_BR_SPRACE

2 PiB distributed over 13 dCache pools. Some pools built on top of a RAID-5 via hardware disk controller and some on top of a software RAID-5

T2_BR_UERJ

We have JBOD in all worker nodes with HDFS and RAID5 in our head nodes. Total user-available capacity of 100TB.(HDFS)

GSI-LCG2

Lustre over ZFS, 25 PiB usable capacity (shared), out of which 2.6 PiB reserved to ALICE

UKI-NORTHGRID-LIV-HEP

Useful capacity is assumed to be ~ between 92 and 95% of raw, hence total usable would be ~ 1.6 TB.

CIEMAT-LCG2

Mostly for our data we use boxes with 1/2 RAID-6 reserving 2 hard drives per box as a spare units. For our data servers (dCache) our netto capacity is ~ 2.2 PiB.

a

T2_US_Purdue

HDFS build from JBOD disks in individual storage nodes. No RAID, except for system disk RAID1s. Our default replication in HDFS is 2, but for files not available elsewhere (e.g. /store/user, /store/group, /store/results) we use replication of 3 for increased resilience against disk failures. This leaves us with roughly 4PB user-available capacity total.

IN2P3-LAPP

High perf. High Avail. Storage Service - used as distributed File System across the cluster - GPFS front-end DELL R630 servers X 4 on the SAN infrastructure described above - Total usable capacity : 240 TB - migration to CEPHFS planned on common hardware (non SAN) High capacity Storage Service - DELL enterprise RAID-6 servers (X 27) with SAS disk (8 TiB, 10 TiB) - Total usable capacity : 2340 TB

TRIUMF-LCG2

We use RAID-6 for disk systems, plus hot spare drives, total usable capacity is 10550TiB.

For tape media, we provide 30369TiB capacity, with 499TiB usable HSM buffer (clustered RAID-6 with reserved drives)

KR-KISTI-GSDC-01

We have RAID-6 and it is attached with SAN to storage servers. No special filesystem is deployed but we are using POSIX (ext4 or xfs) filesystem. As described above, we have 3 different disk storages and their usable capacities are: 1) 1,687TB 2) 1,615TB and 3) 650TB.

GRIF

RAID-6, 40 and 80 TB

IN2P3-CPPM

RAID 6

IN2P3-LPC

RAID-6, 1.8 PB netto capacity

IN2P3-LPSC

SATA RAID6 net = 1400 TiB

ZA-CHPC

as above witht 384TB for ALICE and 260TB for ATLAS

JINR-T1

ZFS on HBA or JBOD, from 117 to 137 TB

praguelcg2

motly RAID-6, total site usable capacity around 5 PiB

UKI-NORTHGRID-LIV-HEP

Useful capacity is assumed to be ~ between 92 and 95% of raw, hence total usable would be ~ 1.6 TB. Nearly all bulk storage systems are 24 drive NAS servers, with one RAID6 array of 24 drives, system drives are separate. Some 36 drive systems are used, split into two arrays of 18 drives each.

INDIACMS-TIFR

RAID6 with 12 disks

TR-10-ULAKBIM

RAID-6, SAN & RAID-6, LustreFS

prague_cesnet_lcg2

RAID-6

TR-03-METU

RAID6 system with 12 drives, each derive having 2TiB, user-available capacity is 472 TiB

aurora-grid.lunarc.lu.se

GPFS (128TB)

SARA-MATRIX_NKHEF-ELPROD__NL-T1_

RAID6

FMPhI-UNIBA

RAID-5, RAID-6, user capacity 1298 TiB

DESY-HH

SAS disks in raid 6 or DDP with 18PB.

T3_PSI_CH

4xJBODs (RAID-6) with 12 TB drives and 1x DDP NetApp

SAMPA

RAID-5,JBOD, XFS, 900TiB and 350TiB

INFN-T1

GPFS over RAID-6 (~20%) and the rest over Distributed RAID. 37PB usable capacity.

GLOW

RAID-0, JBOD, HDFS (raw space ~ 8.2PB, usable capacity ~ 4.1PB)

UNI-FREIBURG

ZFS raidz2 [equiv. to RAID-6] (each 12 or 15 HDDs), partly with SSD cache, 2.92 PiB net

Ru-Troitsk-INR-LCG2

RAID6 370TiB

T2_Estonia

See question 1 answer.

pic

For disk resources, RAID-6 is used. Initially discs are distributed with RAID60, to have more parallelism and better response time. Logical volumes in servers are re-configured and reduced in capacity in time, to improve reliability and reduce data losses due to failing HDDs in the servers (ex. We move from a one pool config in RAID6 to a three pool config in RAID6). This reduces the net space available per server when the server is aging, but improves reliability of the servers. The servers have no redundancy for the network elements. A person is on duty 24x7 and we do have operational support for all of the services that run in production, even with stock on-site for many components. All of the disk servers are managed by a dCache instance in PIC. The current usable capacity is 9.75 PB, and the namespace is mounted via NFS4.1 in the compute nodes (this data access is used by some non-LHC VOs).

Ceph is being tested at PIC at the moment. A Ceph cluster with a 400TB capacity has been installed with old hardware for testing purposes. Currently, we are testing CephFS as a replacement for the NFS or posix-like disk service. Additionally, we are also testing RBD using iSCSI as a storage backend for our virtualization platform. We are also interested in testing Object store, eventually, as a possible backend for Hadoop, or owncloud.

ifae

The ifae site is hosted at PIC. For disk resources, RAID-6 is used. Initially discs are distributed with RAID60, to have more parallelism and better response time. Logical volumes in servers are re-configured and reduced in capacity in time, to improve reliability and reduce data losses due to failing HDDs in the servers (ex. We move from a one pool config in RAID6 to a three pool config in RAID6). This reduces the net space available per server when the server is aging, but improves reliability of the servers. The servers have no redundancy for the network elements. A person is on duty 24x7 and we do have operational support for all of the services that run in production, even with stock on-site for many components. All of the disk servers are managed by a dCache instance in PIC. The current usable capacity is 9.75 PB, and the namespace is mounted via NFS4.1 in the compute nodes (this data access is used by some non-LHC VOs).

Ceph is being tested at PIC at the moment. A Ceph cluster with a 400TB capacity has been installed with old hardware for testing purposes. Currently, we are testing CephFS as a replacement for the NFS or posix-like disk service. Additionally, we are also testing RBD using iSCSI as a storage backend for our virtualization platform. We are also interested in testing Object store, eventually, as a possible backend for Hadoop, or owncloud.

NCBJ-CIS

Dell/EMC Ision OneFS NAS storage with a proprietary data striping (6TB user-available)

RAL-LCG2

Disk: Ceph using Erasure Coding (8+3). Usable capacity 39PB.

Tape: Castor (completely separate from Ceph). There is a 2PB usable buffer in front of the tape endpoint using RAID6. Usable tape capacity is 59PB. We do compress files but there is also some loss of capacity due to tapes not being completely full all the time (due to deletion). These approximately balance out.

T2_IT_Rome

JBOD

BNL-ATLAS

All of the mass disk storage is RAID-6, either software raid on JBOD, or hardware raid (for the oldest storage). The usable capacity is ~20PB.

FZK-LCG2

34PB usable capacity on declustered raids (called dynamic disk pools, 8+2P) for GPFS data 45TB usable capacity (SSDs) on RAID-10 for GPFS meta data (not directly user accessible)

INFN-NAPOLI-ATLAS

All systems are organized in RAID6 volumes, one volume corresponding to 1 jbod of 12 disks. The total usable capacity is 2080 TiB

-- OliverKeeble - 2019-08-22

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2019-09-20 - OliverKeeble
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback