btrfs send fails with ERROR send ioctl failed with -5 Input/output error

[jlittle]

Attempting to send/receive my Eoan root filesystem to my backup drive:

! sudo btrfs sub snapshot -r /mnt/top/@_eoan_u /mnt/top/eoan_u_incr/@_2019-11-16
Create a readonly snapshot of '/mnt/top/@_eoan_u' in '/mnt/top/eoan_u_incr/@_2019-11-16' 
! sudo btrfs send /mnt/top/eoan_u_incr/@_2019-11-16 | sudo btrfs receive /mnt/stuff/eoan_u_incr
At subvol /mnt/top/eoan_u_incr/@_2019-11-16
At subvol @_2019-11-16
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream

It's the send that failed:

! sudo btrfs send /mnt/top/eoan_u_incr/@_2019-11-16 | wc
At subvol /mnt/top/eoan_u_incr/@_2019-11-16
ERROR: send ioctl failed with -5: Input/output error

The system log has:

17/11/2019 00.04	myriam	kernel	[ 1434.833658] BTRFS error (device sda2): did not find backref in send_root. inode=501907, offset=0, disk_byte=14432169984 found extent=14432169984

Google finds various mailing list threads, but only one from earlier this year. There was a kernel bug with these errors fixed in 4.3; maybe this is a regression.

This may be related to the problem Vinny reported, as I've struck it on the first send/receive since upgrading to 19.10 and getting the 5.3 kernel, but it's not specific to any directory or files. I have successfully backed up the / filesystem with rsync.

The @home subvolume send/receive, from and to the same btrfs file systems, ran ok.

All the snapshots of the root subvolume since I release-upgraded at the beginning of the month that I've tried give this error.

Amy advice or suggestions gratefully received. Particularly,

• if I want to file a bug somewhere, where would it be?
• Would it be advisable to back up, wipe the device (a 250 GB SSD), and make a new btrfs?

[vinnywright]

the problem I reported on was in the newer 5+ series kenels btrfs tree checker . finding a wrong time stamp (2025) on files.

what kernel are you running .

look at "dmesg" for btrfs errors and disk IO and CRC ? CRS , errors.

VINNY

[oshunluvr]

I would try booting to an earlier kernel to verify it's a kernel issue (has been in the past). Why do you think it's not specific to "inode=501907, offset=0, disk_byte=14432169984 found extent=14432169984" as reported? Seems rather specific to me. This might tell you what's stored there if anything:

sudo btrfs inspect-internal inode-resolve 501907 /

Does a balance fail? Doing that (plus trim) might clear out that inode.

If that fails, you can identify what exists on that inode and see what's there if anything.

Assuming you have the room on the device, snapshot, delete the file(s) on the suspect inode, attempt send on the snapshot, if successful restore (copy) deleted file(s) to received subvol, re-send repaired subvol back to host.

[jlittle]

what kernel are you running .

5.3.0-23-generic, to be specific. There's been a couple of new ones recently.

look at "dmesg" for btrfs errors and disk IO and CRC ? CRS , errors.

dmesg just reports the system log entry in my post.

[vinnywright]

do you have an older kernel still on your system you can boot to , like a 4.15 or so ?

if so boot to it and see if the error still happens

VINNY

[jlittle]

Why do you think it's not specific to "inode=501907, offset=0, disk_byte=14432169984 found extent=14432169984" as reported?

I should have noticed that, was late at night, I'd attempted the backup before retiring.

$ sudo btrfs inspect-internal inode-resolve 501907
//usr/lib/libosp.so.5.0.0
$ ls -l /usr/lib/libosp.so.5.0.0
-rw-r--r-- 1 root root 1869648 Apr 13  2017 /usr/lib/libosp.so.5.0.0
$ wc -c /usr/lib/libosp.so.5.0.0                                         ~
1869648 /usr/lib/libosp.so.5.0.0
$ lsof -f -- /usr/lib/libosp.so.5.0.0
$

Does a balance fail?

I set a balance running, it's still going after about an hour and a half, generating millions of lines of "critical" system log entries. I've tried to cancel it, but

$ sudo btrfs balance status /                                              ~
Balance on '/' is running, cancel requested
0 out of about 1 chunks balanced (95 considered), 100% left

So I'm taking that as failure. There's been about 60 MiB/s of writes on the device that whole time, more than enough to write the whole device, which shows as 53 GiB used (data + metadata) on a 250 GB device. I've tried to kill the process, but the kill has no effect, and it's still accumulating cpu time. I'll post this and do a shutdown, and then a hard reset if necessary.

[jlittle]

Well, a shutdown that took ages and stopped at the last step without powering off, hard reset, and the balance is still running, still spewing system log entries.

How do I start the OS without restarting the btrfs balance?

[jlittle]

The kernel thread that won't die. I booted from my trusty, ancient, systemrescuecd v5.1 stick. And as soon as I mounted the btrfs, the balance fired up. That's gentoo with some 4 point something kernel.

At this point I'm hours past the point where a reformat and restore from backup would have completed. I'm starting to worry about the health of the SSD with all the writing going continuously. I'm researching fsck options for btrfs now.

[jlittle]

Well, things continued to go downhill.

I tried btrfs check, and it found some errors. So I did btrfs check --repair.

That gave horrible errors and crashed, something like "Aborted - core dumped". It had made some changes before crashing, so I ran it again, and it failed with different horrible errors. Now, the btrfs refused to mount, with nasty errors. I tried btrfs rescue chunk-recover, and it seemed to do something but still it couldn't mount, with different errors. btrfs rescue super-recover said there was no problems with the super blocks.

So, it was a lost cause. I booted a cosmic iso and used KDE partition manager to delete and recreate the btrfs. I recreated the subvolumes, and copied them back from the backups. Some grub tinkering (my backup was a little old) and Kubuntu is back. I completed the send/receives uneventfully.