[Solved] ERROR: send ioctl failed with -5: Input/output error;

[Chopstick]

Hi there!
As some may remember I've been using BTRFS and BTRBK as my backup solution for a while now. I tend to just forget about these things and just let them run, so when I had to restore my system yesterday (no idea what happened - just threw me into Grub and would not go further), I noticed that my root subvolume had not been sent to the external hard disk for about 6 month (last backup in April), while my home backup went without a hitch and all are there. I tried to run btrbk manually, and got the error below; then I tried to do a manual (non-incrimental) send and that got the same error:

# btrfs send /mnt/btrfs_pool/btrbk_snaps/\@.20221012/ | btrfs receive /media/me/Backup/btrbk_backup/
At subvol /mnt/btrfs_pool/btrbk_snaps/@.20221012/
At subvol @.20221012
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream
​

I run this as root and /media/me/Backup/ is where my external drive is mounted.At first it seems to run fine for ~10min and then I get this error... I googled it and did not find anything useful: one thread on GitHub that was closed without solution and one by jlittle here, which apparently horribly went sideways (no solution) - any idea? this is really bugging me - especially that a manual send does not work! And send with the @home backup still works... seems really odd to me... I'm wondering if the has something to do with this being the system subvolume? But I am sending a snapshot, not the actual system subvolume...

Thanks for any help!

[jlittle]

I've seen this error on a btrfs send. (It was after a period of failures to resume from sleep that required a motherboard reset by popping the CMOS battery to restart.) Using KSystemLog I found

17/11/2019 00.04 myriam kernel [ 1434.833658] BTRFS error (device sda2): did not find backref in send_root. inode=501907, offset=0, disk_byte=14432169984 found extent=14432169984

Only btrfs send had problems, all the data could be accessed, so I backed up using rsync.

I tried to fix it by running btrfs balance, and that did not go well, with millions of critical kernel errors. The kernel btrfs driver got stuck writing continuously, and nothing could stop it. Even booting from a very old USB with gentoo got stuck in the same loop on attempting to mount the btrfs. Further repair attempts just crashed with horrible errors.

I considered it a pass, if not a win, that I could copy all the data out of the btrfs. Using a live USB, I wiped the partition and created a new btrfs using the "KDE Partition Manager", and copied all the data back. It's been great since.

I suggest backing up the problem subvolume not using btrfs send/receive, maybe rsync -aAXv. Then you might want to dive into trying to repair things, just don't start with a btrfs balance. Or, as with hindsight I should have done, cut your losses and wipe and recreate the btrfs.

[Chopstick]

Did you run balance with or without filters? I often run balance with the dusage/musage filters...

[jlittle]

I ran

sudo btrfs balance -ddevid=1 -ddrange=14432169984..14434039632

I didn't document why I used that filter, and now I've forgotten. Presumably I chose the -ddrange filter based on the error in the system log.

[Chopstick]

I would prefer a solution that does not require reformatting the disk, since that was quite a lot of work. My system subvolume is in the same encrypted LUKS partition and the rest of the system, so I would have to reformat everything and set it up again... I could also add that the current root volume is actually a snapshot of the original one that I had to restore recently (snapshots on the same disk still work). However, the error mentioned above was actually already present before I restored, so the snapshot must have inherited this from the original.

[Chopstick]

So, I think I figured out the root cause of this error on my system, but I think I need some advice on solutions.

As recommended by jlittle I made a manual backup with rsync; then I booted into a live-USB-key and ran 'btrfs check --repair' and 'btrfs balance -dusage=90 and -musage=90' a couple of times. the --repair did find a whole bunch of broken references the first time and I also cleared the space cache, because of some other warning.
Unfortunately none of this did fix this problem; however, I got another line in the error message 'Bad file descriptor':

ERROR: send ioctl failed with -5: Input/output error
ERROR: failed to read stream from kernel: Bad file descriptor
ERROR: unexpected EOF in stream[/FONT]​

At the same time, rsync had complained about two empty symlinks (empty as in, pointing to an empty string, but not broken). Here are the offenders:

lrwxrwxrwx 1 root root 0 Apr 20 20:35 /etc/alternatives/builtins.7.gz -> ''/
lrwxrwxrwx 1 root root 0 Apr 20 20:35 /usr/share/man/man7/builtins.7.gz -> ''/​

After I removed these empty symlinks from my snapshot, the btrfs send was successful and I was also able to make another incremental send (with those symlinks removed - if I don't, it crashes again, as before)! I am not sure if the 'btrfs check --repair' I did earlier was necessary for this, but it might have been, since it added the additional info to the error message (and it did repair a bunch).

Now the issue is, how to implement the solution: basically I need to exclude these two symlinks from my snapshots. As I see it, there are essentially two options: just delete these symlinks in the root subvolume and hope nothing breaks (what good could these empty links possibly be anyway?), or write a script that manually snapshots and removes the offending symlinks, before btrbk runs and sends them to the external disk.

The former (just delete the two symlinks for good) would obviously be a lot less hassle - any thoughts on the risks involved? Does anyone know where these are coming from and what they are good for?
Thanks!

[jlittle]

tl;dr you don't need them, ancient cruft that doesn't work.

The link /etc/alternatives/builtins.7.gz, and the file /usr/share/man/man7/builtins.7.gz are used as "a hack to force bash builtins into the whatis database and to get the list of builtins to come up with the man command". The first should point to the second, and the second is 18 lines of man page source. AFAICT the hack doesn't work, because if I run whatis typeset it says "typeset: nothing appropriate." I think it's like dead code, only less significant, dead man page set up from a bygone era, obsolete decades ago. I've no idea how they got screwed up on in your install; all I can think of is an interrupted update. Conceivably an APT update was running when some snapshot was taken in the history of your install.

[Chopstick]

Thanks! that actually really helped!

After your explanation I deleted the symlinks and reinstalling bash-related packages and it worked - they are now pointing to something real! With that, I think the backups should work fine!

Edit: I can confirm, the backup now runs as intended - issue solved - thanks a lot jlittle!