I/O errors on USB3 BTRFS drive

[jfabernathy]

I'm hoping that someone can tell me if I'm doing something wrong or if I have a real hardware issue.

My Media server is working fine now and I thought about doing a backup of the data mirror which is BTRFS to a USB3 drive for storage in case of a very bad failure of a lot of things. I made the USB3 drive with 1 partition formatted btrfs. I did this so I could copy any snapshots as well as the media data.

So to start with I was on /dev/sdd1 but I used the BLKID in the fstab to avoid confusion is devices changed. I started the rsync and watched it a little and walked away. Came back to find a screen full of I/O errors. I looked a journalctl and dmesg and there are indeed lots of errors, but mostly BTRFS errors that I don't understand. Also when I do lsblk, I don't see /dev/sdd1 anymore, but I do see /dev/sde1.

I assumed I had a drive with errors, so I took it out to the garage where I had a PC old enough that I can hookup the drive directly with SATA cables and set the interface to IDE so I can run deep diagnostics with Spinrite 6, which is old but tortures the drive and finds bad sectors. I ran for 4 hours and found absolutely nothing wrong with the drive. So I reformatted the drive and put it back in the USB3 case and tried again. Sames issues and errors.

Is there something special about BTRFS use in external USB3 drives that I should know about??

[jfabernathy]

So I opened the USB external case and replace the drive with another and formatted the drive and started over I watched for 5 minutes and saw a failure so I CTL-C'ed and captured the console log.

mythtv/recordings/
mythtv/recordings/10401_20220108210000.ts
mythtv/recordings/10401_20220108210000.ts.-1.100x56.png
mythtv/recordings/10401_20220108210000.ts.png
mythtv/recordings/10401_20220115210000.ts
rsync: [receiver] write failed on "/mnt/external-backup/mythtv/recordings/10401_20220115210000.ts": Read-only file system (30)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)
sending incremental file list
rsync: [generator] failed to set times on "/mnt/external-backup/mythtv/recordings": Read-only file system (30)
rsync: [generator] delete_file: unlink(mythtv/recordings/.10401_20220115210000.ts.wEbHRO) failed: Read-only file system (30)
rsync: [generator] failed to set times on "/mnt/external-backup/nas": Read-only file system (30)
rsync: [generator] failed to set times on "/mnt/external-backup/nas/.TemporaryItems": Read-only file system (30)
rsync: [generator] failed to set times on "/mnt/external-backup/nas/.TemporaryItems/folders.501": Read-only file system (30)
rsync: [generator] failed to set times on "/mnt/external-backup/nas/.Trash-1000": Read-only file system (30)
mythtv/recordings/
mythtv/recordings/10401_20220115210000.ts
rsync: [receiver] mkstemp "/mnt/external-backup/mythtv/recordings/.10401_20220115210000.ts.Fk2Kxl" failed: Read-only file system (30)
mythtv/recordings/10401_20220115210000.ts.-1.100x56.png
mythtv/recordings/10401_20220115210000.ts.png
rsync: [receiver] mkstemp "/mnt/external-backup/mythtv/recordings/.10401_20220115210000.ts.-1.100x56.png.qWFD4b" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/external-backup/mythtv/recordings/.10401_20220115210000.ts.png.7TBb5X" failed: Read-only file system (30)
mythtv/recordings/10401_20220122210000.ts
rsync: [receiver] mkstemp "/mnt/external-backup/mythtv/recordings/.10401_20220122210000.ts.TWOiRv" failed: Read-only file system (30)
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(503) [generator=3.2.3]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(701) [sender=3.2.3]

my fstab entry for the USB external drive is:

UUID=355e83ab-c1aa-40ab-9284-a7ff8b96de20 /mnt/external-backup btrfs   defaults,noatime,compress=zstd,space_cache=v2,auto defrag 0 0

If this is hardware, I'm thinking the USB external case. Or is there something about this use case with BTRFS?

[jfabernathy]

I think I found the issue. After getting the same errors on 2 different drives, I took one of those drives and put it in a USB2/ESATA case and plugged that into the same computer. I formatted it btrfs just like I originally did and mounted it the same. I'm not doing an rsync from my mirror to the external drive and I'm way past where I had issues before. So I'm guessing it was indeed a USB3 case issue and now it's time to return it.

[oshunluvr]

The first thing that sticks out is the typo in the mount line. You can't have spaces in the options so "auto defrag" is actually just "auto" with "defrag" being garbage. However, I doubt that is the problem.

I would agree that the first suspect is the USB enclosure. I have had one fail before and it wasn't very old. Seems odd that rsync would "give up" like that, it's a very robust tool. Have you tried reformatting the drive in the enclosure to EXT4 instead and see if you get similar results? If you do not, it would be worth reporting to the developers. They respond directly to inquires on the btrfs IRC channel - I'm checking with them now.

I've never seen this, but I don't ever use rsync with btrfs nor btrfs on USB drives. I just haven't tried it. I do use rsync on a large USB drive in an enclosure, but it's ntfs due to needing windows compatibility (it's a work thing).

I know that it is not recommended to use send|receive over USB channels because of the likelihood of the USB channel dropping off even momentarily which would cause a failed backup. The recommend manner is to send the subvolume to a file, then copy the file over and then receive it from the file. I think it has to do with btrfs send notifying the user when this occurs. It might even be fixed by now.

[oshunluvr]

Oops, we cross posted. Glad you found the issue.

I've been using this one for almost 3 years without issue and barely 20 bucks: https://www.amazon.com/gp/product/B0...?ie=UTF8&psc=1

...and no, I'm not a paid spokes-person

[jfabernathy]

This is actually a little bizarre issue. Now that I have my ESATA external drive formatted btrfs and mounted with the autodefrag spelled right, working with rsync like it should, I thought about testing the USB3 external drive case in question on another computer. So I setup an rsync of my rootfs on the Dell XPS 15 9500 which is fairly new, to the USB3 drive and it's working right.

So I think the problem is the media computer is old enough that the USB3 interfaces where not supplied in the Intel chipset, but were 3rd parts chips. So I'm betting the real culprit is the USB3 ports on my old motherboard. This application does not require the latest CPU and works fine on a Core i7-2600K but I guess USB3 was too new to be bug free yet.

So I guess I'll drink a cheaper bourbon for a few weeks and save up to a new motherboard and CPU.

[oshunluvr]

Cheap Bourbon - horrible idea.

Well, if you get to New Bern I got like 7 bottles of various Bourbons as gifts on my 60th birthday last month. We can do a taste test!

[oshunluvr]

I upgraded my media server last year too. I got tired of the lack of sata ports mostly, as well as lack of power.

It's a Gigabyte H97N-WIFI, mini-itx with an i3-4160T CPU @ 3.10GHz, I put 8GB RAM on it and it does all I need.

I saw that you're using Myth-tv. I'm a Plex guy - bought a lifetime server ticket in 2014 when they were just getting rolling - $79.

[GreyGeek]

I have five drives (3 512Gb spinners, a 1Tb spinner and a Samsung EVO 860 500Gb SSD) hooked to external USB3 caddies. ALL are formatted with BTRFS. Most are two years old, or older. None have ever thrown a single error of any type at any time. The 1 Tb spinner is my go-to external backup driver. It's relatively new because it was the primary and only drive in this new HP-17nc1xxx laptop I got for Xmas.

I also have a USB-to-SATA cable that works flawlessly as well. You just plug it into the drive connector. It has no housing.

[jfabernathy]

I have five drives (3 512Gb spinners, a 1Tb spinner and a Samsung EVO 860 500Gb SSD) hooked to external USB3 caddies. ALL are formatted with BTRFS. Most are two years old, or older. None have ever thrown a single error of any type at any time. The 1 Tb spinner is my go-to external backup driver. It's relatively new because it was the primary and only drive in this new HP-17nc1xxx laptop I got for Xmas.

I also have a USB-to-SATA cable that works flawlessly as well. You just plug it into the drive connector. It has no housing.

The more I think about this, I think I remember issues with this motherboard model back when I was working. I used to work for Intel and this is a board from when Intel was still in the desktop motherboard business. This is probably a case of not having the external hardware that could stress that USB3 chipset the way we can today with all the new stuff.

I'm hoping to have the new motherboard, CPU, and RAM next week. What I'm hoping I can do is somehow "copy" my current boot SATA SSD over to a M2.nvme drive that I'll use in the new motherboard. If I can do that successfully, then I can just plug the btrfs mirror into the new motherboard SATA ports and I hoping to have this working without a lot of problems.