Sometimes while testing file systems or applications you want to simulate some errors on the disk level. The first time I heard about this need was from Baptiste Daroussin during his presentation at AsiaBSDCon 2016. He mentioned how they had built a test lab with it. The same need was recently discussed during the PGCon 2019, to test a PostgreSQL instance. If you are FreeBSD user, I have great news for you: there is a GEOM provider which allows you to simulate a failing device.

GNOP allows us to configure transparent providers from existing ones. The first interesting option of it is that we can slice the device into smaller pieces, thanks to the ‘offset option’ and ‘stripsesize’. This allows us to observe how the data on the disk is changing. Let’s assume that we want to observe the changes in the GPT table when the GPT flags are added or removed (for example the bootme flags which are described here). We can use dd every time and analyze it using absolute values from the disks. To examine GPT (the offset may differ on your machine) we can use the following to analyze the GPT Header:

# dd if=/dev/md0 count=1 skip=1 | hexdump -C
00000000  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000010  07 d1 c9 f8 00 00 00 00  01 00 00 00 00 00 00 00  |................|
00000020  ff ff 1f 00 00 00 00 00  28 00 00 00 00 00 00 00  |........(.......|
00000030  d7 ff 1f 00 00 00 00 00  81 56 16 9d ec 02 ea 11  |.........V......|
00000040  a0 83 4c cc 6a 69 5d 5f  02 00 00 00 00 00 00 00  |..L.ji]_........|
00000050  80 00 00 00 80 00 00 00  90 0e bf a4 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

With the following command we can analyze the GPT partition array:

# dd if=/dev/md0 count=1 skip=2 | hexdump -C
00000000  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000010  67 ba 23 a3 ec 02 ea 11  a0 83 4c cc 6a 69 5d 5f  |g.#.......L.ji]_|
00000020  28 00 00 00 00 00 00 00  27 00 10 00 00 00 00 00  |(.......'.......|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000080  b6 7c 6e 51 cf 6e d6 11  8f f8 00 02 2d 09 71 2b  |.|nQ.n......-.q+|
00000090  22 72 fd a4 ec 02 ea 11  a0 83 4c cc 6a 69 5d 5f  |"r........L.ji]_|
000000a0  28 00 10 00 00 00 00 00  d7 ff 1f 00 00 00 00 00  |(...............|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Playing more with the bs, count and skip value we can get more detailed information about the GPT table. What if we would like to observe them? Each time we would need to remember the absolute values of the GPT from the beginning of the disk. Thanks to the GNOP provider we can create a slice from the disk, and the new device will point to the interesting parts of the GPT table.

# gnop create -o 512 -s 512 /dev/md0

The commands above create a new device – md0.nop. The device starts at the 512 offset and its size is also 512 bytes. Thanks to that we can use dd to read the data relative to the start of the GPT header. Unfortunately, at the time of writing, the limitation of this util is that we can only create a single nop device from a single provider - but this may change in near feature.

Additionally, while creating it we can configure how this transport layer should behave. One of the options is to configure the failure rate of the IO request.
# gnop create -w 80 -r 60 /dev/md0

After using the command above, we created a new md0.nop. While using this provider the probability that the write request will fail is 80%, and 60% for the read ones.

Some time ago I also added three new options, to simulate the slower IO of the disk. We can configure the disk request delay and the probability rate of it happening. For example, I used it to test behavior of ZFS RAIDZ2 when one of the disks is slower than the rest of them.

# gnop create -x 50 -d 1000 /dev/md0

While using new md0.nop provider the probability that the request will be delayed for one second is 50%.

GEOM NOP is a very powerful tool which allows you to test a failing disk, as well as a great tool for the disk forensics. You should have it in your toolbox!