Software RAID support in Stratis

[mulkieran]

We would like to investigate integrating software RAID support in Stratis.

At present, if a user would like to make their data more secure, we suggest that they use a RAID device as the device that they add to the pool. We are currently adding the ability for a pool to take advantage of a RAID device that the pool uses being enlarged. Thus, by managing the RAID devices that is a Stratis blockdev the user can increase the size of a pool and make use of the underlying RAID device's data safety guarantees.

The new proposal is to integrate support for software RAID directly into Stratis, via dm-raid.

[drckeefe]

Stratis should consider:

• Allowing a minimum of two device.
• Starting with just the basic management options needed
• State that this will be for newly created pools
• Support only 1 type of RAID (at least initially)

[jbaublitz]

Adding onto that, I think the operations we should support at the beginning are creation, destruction, adding devices, and swapping out failed disks.

[jbrassow]

Please keep in mind that complexity will depend on how this is managed. If you use something like MD at the base layer, it will take care of issues like 'missing disk on start-up'. If you write your own interface, you will need to take care of all of that...

[jbaublitz]

@jbrassow Can you elaborate on this a little bit? Does dm-raid, for example, not handle missing devices on start up like md-raid does? We'd like some more information on specifically what the cases are that md-raid handles that dm-raid does not.

[shodanshok]

There is a specific reason why you plan to use dm-raid rather than relying on md / mdadm from inside stratis?

[mulkieran]

There are a lot of considerations. One that drives us toward dm-raid is that we generally try to avoid using a command-line application; with dm-raid we can use ioctls.

[shodanshok]

While I understand you motivation, I somewhat feel that by going the dm-raid route you will need to take care of many corner cases already solved in md / mdadm. Considering how much stratis is behind the original roadmap (with RAID initially planned for 2.0 release), I fear that this can slow down the entire project.

[shodanshok]

Uhm, I just read here that dmraid is going to be deprecated: https://bugzilla.redhat.com/show_bug.cgi?id=1489587

Are you going to support it yourself?

[jbaublitz]

I don't think this actually means the devicemapper target for RAID is deprecated. That is not provided by a package, it's in the kernel. I think this is a specific package related to dmraid. I'm not entirely sure what it does, but appears to be some kind of tool for interacting with the kernel targets.

[jbaublitz]

We'd also like to get some feedback about some preliminary work required to support RAID properly. Our current metadata abstraction is probably going to require making an important decision regarding future features. Our current options seem to be:

• Encrypt pool level metadata
• Leave pool level metadata unencrypted
• Split pool level metadata into two regions

Overview

For a bit of background, we are planning, regardless of which we do, to move the static header to the beginning of the device. This will leave it unencrypted. Because of what we store in the static header, this is likely not a problem from a security perspective. It contains UUIDs, device size, etc. Because allocations are controlled by the pool level metadata, malicious data could not be injected into a device that's been increased in size if the pool level metadata is encrypted because any such data added to a given device would be overwritten when the thin pool extends over that part of the newly sized device.

Encrypt pool level metadata

From a security perspective, this seems desirable. The above discussion makes me think that we'd probably want to stop any such case from happening.

This, however, brings in complication around stacking.

We would ideally like to put encryption on top of RAID if RAID is enabled to avoid encryption amplification by putting it under RAID on each member device. This means that if a pool starts as encrypted without RAID, I see no clean way to move from an MDA region on each block device at the beginning of the unlocked encrypted device to a RAID device with encryption on top with one single MDA at the beginning of the unlocked RAID device.

Leave pool level metadata unencrypted

This allows us to keep our abstraction for the static header and MDA virtually unchanged. This also means that if we require storing any information about RAID, integrity, encryption, etc. for setup, this will be accessible as needed on the physical device. However, this could allow any root user to modify the allocations in the metadata which we'd like to protect against in the encryption case.

Split MDA into two separate regions

This idea may not be worth it, but it could potentially allow us to store two separate MDA regions, one being encrypted and relating to everything above encryption, and one being at the blockdev layer next to the static header for all layers up to encryption. This may be the best balance in terms of security and providing enough information for setup of layer from encryption to the bottom of the stack. This could also allow us to have a single MDA in the unencrypted case, moving it to a separate region if the unencrypted pool is converted to an encrypted pool.

@lvmguy I still fail to see a path in this case from converting the many encrypted MDA regions to a single encrypted MDA region with RAID underneath. Furthermore, I fail to see a way to convert from an encrypted pool without RAID to an encrypted pool with RAID without first decrypting the entire pool, converting to RAID, then reencrypting the RAID device. Let me know what your thoughts are.

[mauelsha]

@jbaublitz
If you store the MD RAID information within the new header you want to split out at the beginning of devices as of your 'Overview' section above, all constraints on encrypted pools relative to data and metadata disappear as you'll be able to activate the MD RAID array before accessing any Stratis pool metadata. This'll involve an interim dm linear mapped devices layer though.

Brief outline of a discovery+activation sequence with MD RAID array at the lowest device level:

1. stratisd scans block devices for split out header
2. discovered header holds MD RAID array marker+Array_UUID
3. stratisd creates linear mappings for all devices marked as RAID array members or optionally a subset identified by selected Array_UUID mapped past the header to the beginning of what should be accessible at the start of the RAID array member disks (i.e. MD RAID metadata at expected offset)
4. stratisd runs the adequate of 'mdadm -As', optionally trimmed down to an MD array UUID (i.e. 'mdadm -Asu $Array_UUID')
5. stratisd discovers pool metadata as currently on the activated RAID array(s)
6. stratisd adds stacked devices on top of the RAID array for encrypted/non-encrypted pool data and metadata respectively

These steps don't cover any migration of current Stratis layouts to the above presumed new one.

[DemiMarie]

@jbaublitz Thanks for pinging me!

I would prefer to encrypt the pool level metadata, and figure out how to make stacking work with it.

[jbaublitz]

@jbaublitz If you store the MD RAID information within the new header you want to split out at the beginning of devices as of your 'Overview' section above, all constraints on encrypted pools relative to data and metadata disappear as you'll be able to activate the MD RAID array before accessing any Stratis pool metadata. This'll involve an interim dm linear mapped devices layer though.

Brief outline of a discovery+activation sequence with MD RAID array at the lowest device level:

1. stratisd scans block devices for split out header

2. discovered header holds MD RAID array marker+Array_UUID

3. stratisd creates linear mappings for all devices marked as RAID array members or optionally a subset identified by selected Array_UUID  mapped past the header to the beginning of what should be accessible at the start of the RAID array member disks (i.e. MD RAID metadata at expected offset)

4. stratisd runs the adequate of 'mdadm -As', optionally trimmed down to an MD array UUID (i.e. 'mdadm -Asu $Array_UUID')

5. stratisd discovers pool metadata as currently on the activated RAID array(s)

6. stratisd adds stacked devices on top of the RAID array for encrypted/non-encrypted pool data and metadata respectively

These steps don't cover any migration of current Stratis layouts to the above presumed new one.

@mauelsha Yes, this was actually the approach we planned on using for both RAID and encryption. We don't have any problems with that design. More what I'm wondering about is you had specifically mentioned that we should leave room for RAID metadata to facilitate converting from no RAID to RAID.

Mostly what I'm concerned about is if we stack encryption on top of RAID and then put the pool level metadata on top of both, I see no clear path to convert from a pool that was created without RAID to a pool that has RAID.

@DemiMarie Are you okay with the restriction that encrypting all or some of the pool level metadata imposes in terms of conversion from no RAID to RAID at a later date?

[mauelsha]

@jbaublitz
Making room for RAID metadata is part of a migration design.
Can be addressed by mapping space into new linear mapped devices (one per pool backing device) to suit the MD requirement to store its metadata (superblock, optional write-intent bitmap and an amount of reshape space). I.e. you start out with the current Stratis ondisk layout and map that to an appropriate offset in a new linear target's segment adding a mapping to new space for RAID metadata to another segment which presumes available space to allow for that. Making room for your new header upfront the backing devices requires data relocation though. So if you relocate content at the beginning of the device to make room to hold the new Stratis header which in turn stores mapping information talked about above together with the information that the pool's now on RAID1(E) you can set up the encrypted stack you stated. Again, that presumes adequate free space in the pool allowing to relocate the content at the beginning of each pool device and to allocate capacity for the RAID metadata (which varies in offset and size depending on the specific format).

[DemiMarie]

@DemiMarie Are you okay with the restriction that encrypting all or some of the pool level metadata imposes in terms of conversion from no RAID to RAID at a later date?

I think having room upfront for extra metadata if needed is a better approach. That restriction would not be desirable.

[DemiMarie]

How do you plan to handle the parity RAID write hole? The only decent approach I can think of is to use a journal, with three-way mirroring for the journal device. There are other solutions, but they require the storage stack to never modify a block in-place, so they are only suitable for software such as bcachefs that obeys this restriction.

[mauelsha]

Good point which can be addressed by the MD journal support once Stratis decides to support striped RAID.

[jbaublitz]

@DemiMarie We've been discussing which RAID version to use and we've been tentatively thinking it would be RAID 1E. From what I can tell, this would be vulnerable to the RAID write hole, but it's unlikely we'd be able to have three way mirror in the case where we only have two RAID member devices. @mauelsha Does dm-raid or md-raid support a journal for a replication factor of 3 in RAID 1E?

[DemiMarie]

@jbaublitz My personal preference is to never modify a block in-place. This is a hard requirement of zoned storage devices, and enables a tremendous number of other features, such as data integrity (via checksumming or MAC), parity RAID (with no write hole), snapshots, and even prevention of replay attacks (via a Merkle tree). If one insists on in-place modification, then the features I mentioned are very expensive. With always-CoW, they are very cheap, which is why every solution I know of that provides them is always-CoW.