starting sector number, 4532393984 exceeds the msdos-partition-table-imposed maximum

[rwbehne1]

GREY GEEK: If he deletes all his data how is he to recover it?

rwbehne1: DON'T WIPE THE DISK OR WRITE ANYTHING TO IT RIGHT NOW!

Possibly you have not lost the data, but you may need to do some recovery. I see the partition tables match up now. See the unallocated section in between sda3 and 4? It looks to me like the old sda4 is not in the partition table but it's still there. The old sector boundaries of sda4 were 4075401216 to 4532393983 - those sectors are in the unallocated portion of your disk, so it's very likely the data is all still there. You can also see the current sda3 partition ends at 4075401215 - exactly one sector before the old sda4 began.

testdisk or parted should be able to recreate the missing partition.

Ok, YOU just got my full attention!

You're saying along the lines of what I've been thinking. But this is unfamiliar territory for me. Could you be a little more graphic about how to repair it, since I'm just learning this?

Here's what gdisk says right now:

Number  Start (sector)    End (sector)  Size       Code  Name
1              34            2047   1007.0 KiB  EF02  BIOS boot partition
2            2048       209922047   100.1 GiB   8300  
3       209922048      4075401215   1.8 TiB     8300  Linux filesystem
4      [B]5403539456      5860532223[/B]   [B]217.9 GiB[/B]   8300  Linux filesystem

...and fdisk -l shows:

Device          Start        End    Sectors   Size Type
/dev/sda1          34       2047       2014  1007K BIOS boot
/dev/sda2        2048  209922047  209920000 100.1G Linux filesystem
/dev/sda3   209922048 4075401215 3865479168   1.8T Linux filesystem
/dev/sda4  [B]5403539456 5860532223[/B]  456992768 [B]217.9G[/B] Linux filesystem

I have verified that sda3 is in fact my old 1.8T /home partition, the size is right and it is still intact. I can mount it and browse around in it.

I know that 4075401216 to 5403539455 actually is unused space, not formatted. The 217.9G /usr/local partition used to be in there, following immediately behind the /home partition, but yesterday I moved it with KDE part tool to the end of the drive where it is now. Therefore 5403539456 to 5860532223 ought to still be the old /usr/local (the size is right), but KDE part tool shows it as having btrfs when it should be ext4, and it's labeled as / instead of /usr/local. I can mount it, but it looks blank because of the wrong label for the filesystem. (No, I refuse to write anything to it because I think it can be recovered.)

That's what changed - the labels for that partition.

So I think what has to be done is to change the btrfs back to say ext4 again, and change the / back to say /usr/local and that should recover that partition.

Let's try that first. I have installed testdisk, and parted is already installed. I have never used either before, and I don't want to guess, so I'm going to defer to your judgment on how to proceed oshunluvr.

My plans after recovering the partition:
1) Once that partition is recovered I'll want to find out what caused this problem in the first place and fix that too.
2) Then I can safely install a small system on the sda2 and boot into that, so I can stop using this USB drive for the rest of the work. (I think using the external drive may work faster when using a regular install instead of a USB stick.)
3) Next, I have to find a way to quickly move as much data as possible from sda3 to the external drive. That way I can make a new temporary partition in the unused space only big enough to move the remaining files from sda3 to the new partition, and then delete the empty sda3. That will free up space so I can grow sda2 to fill the available space, and move files from the old ext4 partitions into their new home in sda2.
4) After all files are moved from the ext4 partitions they'll be empty, so I can delete them and then grow the sda2 to fill the entire drive.

This is what I love about Linux: There's always more than one way to accomplish any task!

[oshunluvr]

Well, if you moved partitions about in the middle of this you may have trashed it. As I said in the beginning, mucking about with partitions can be dangerous business, especially without a backup. If it we're me, I'd stop focusing on the future plans for a minute and work on recovering your data. Likely you attempted the move before the new partition table went into effect so there's no telling what actually happened. You used several tools with overlapping functions. On a positive note, there is still a last resort - a tool called "photorec" which has an excellent track record of finding files on a drive, however since you have no where else to put data I have no idea who it would work for you, but...

...first things first: Make a partition where your old one was using gdisk like I said in the last post. Then attempt to mount it and see what happens. Under no circumstances should you be writing anything to anywhere on this disk except the partition tables - there's no actual data in the partition table space.

I can't really "be graphic" in the sense of screen shots or whatever, since GUI tools won't be of use.

Post back the results.

[rwbehne1]

I would first attempt to rebuild the missing partition as I suggested using gdisk. Launch it, select "n" for new partition, allow it to be the default partition number "5", then in first sector put 4075401216, in last sector put 4532393983, leave the sector type the default "8300", then write it to disk. Then run "sudo partprobe" (if it errors again, reboot to the liveUSB), then run the partition manager and see where we are.

root@kubuntu:~# gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
 MBR: protective
 BSD: not present
 APM: not present
 GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sda: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): F9C78116-9D2E-4304-A329-E3A7704EF117
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 8-sector boundaries
Total free space is 1328139151 sectors (633.3 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
  1              34            2047   1007.0 KiB  EF02  BIOS boot partition
  2            2048       209922047   100.1 GiB   8300  
  3       209922048      4075401215   1.8 TiB     8300  Linux filesystem
  4      5403539456      5860532223   217.9 GiB   8300  Linux filesystem

Command (? for help): n
Partition number (5-128, default 5): 
First sector (4075401216-5860533134, default = 4075401216) or {+-}size{KMGTP}: 4075401216
Last sector (4075401216-5403539455, default = 5403539455) or {+-}size{KMGTP}: 4532393983
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 
Changed type of partition to 'Linux filesystem'

Command (? for help): p
Disk /dev/sda: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): F9C78116-9D2E-4304-A329-E3A7704EF117
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 8-sector boundaries
Total free space is 871146383 sectors (415.4 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
  1              34            2047   1007.0 KiB  EF02  BIOS boot partition
  2            2048       209922047   100.1 GiB   8300  
  3       209922048      4075401215   1.8 TiB     8300  Linux filesystem
  4      5403539456      5860532223   217.9 GiB   8300  Linux filesystem
  5      4075401216      4532393983   217.9 GiB   8300  Linux filesystem

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
root@kubuntu:~# partprobe
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
root@kubuntu:~#

[oshunluvr]

Ok, try and mount /dev/sda5 somewhere.

[rwbehne1]

I mounted it and...

root@kubuntu:/media/root/_usr_local# dir
BACKUPS  Calibre-customization  games   i2p      lib64       man        sbin   systemd-custom-scripts  webmin
bases    dosutils               gopher  include  libexec     nohup.out  share  TARBALLS
bin      etc                    HOLD    lib      lost+found  opt        src    tNOS
root@kubuntu:/media/root/_usr_local#

Well, that's PART of it, but somethings missing! This is what it was before the partition move. This is what I would expect. However, after I moved the partition I moved a Calibre database directory into the partition, and I see it is not here. I expect that the ext4 really is the actual /usr/local, but after the partition move. I think we should test my theory by fixing sda4 the same way - it couldn't hurt, eh?. If we find the missing Calibre directory then that means I'm right. So shall we try it? Should I use exactly the same method, but with 5403539456 to 5860532223?
Note again that the partition is the correct size as I recall:

4 5403539456 5860532223 217.9 GiB 8300 Linux filesystem

[oshunluvr]

Re-doing the partition table for sda4 won't do anything. The partition table doesn't contain file system format. If the file system is in fact ext4 you might be able to manually force mount it as ext4. There is a conversion tool that rolls btrfs back to ext4 IF it was converted from ext4 to btrfs. Unless you ran a conversion on the ext4 filesystem, it's likely to do more harm than good.

At this point, I'd run testdisk and analyse the disk for trouble.

[oshunluvr]

Where did you move the Calibre database from?

[rwbehne1]

Re-doing the partition table for sda4 won't do anything. The partition table doesn't contain file system format. If the file system is in fact ext4 you might be able to manually force mount it as ext4. There is a conversion tool that rolls btrfs back to ext4 IF it was converted from ext4 to btrfs. Unless you ran a conversion on the ext4 filesystem, it's likely to do more harm than good.

At this point, I'd run testdisk and analyse the disk for trouble.

Ok, while testdisk is running, how do I manually force mount it as ext4?

I don't think it was converted, just relabeled.

[rwbehne1]

Where did you move the Calibre database from?

1) I moved the partition all the way to the end of the drive.
2) I moved the Calibre directory from /home (sda3) to sda4.
Then I did other things elsewhere, and didn't touch those partitions again until now.

[GreyGeek]

Well, if you moved partitions about in the middle of this you may have trashed it. As I said in the beginning, mucking about with partitions can be dangerous business, especially without a backup.

One is on dangerous ground when disagreeing with the lover of oceans, but I've rarely seen a successful recovery of data by restoring starting and ending sectors of a partition, even if one had the foresight to write them down before changing them. This situation was looking to me like a slow motion train wreck and I just wanted to cut to the chase. If nothing else it is a lesson on the necessity of having off line backups, and nothing is better and easier at that in the Linux world than Btrfs.

.... On a positive note, there is still a last resort - a tool called "photorec" which has an excellent track record of finding files on a drive,....

Photorec is excellent indeed. (It is part of testdisk, which is in the repository) I have used it in two cases with outstanding success to recover data from HD's that failed to read because of sector errors. I used dd to create an image.dd file of the drives and then used photorec to extract from that image what it could. In one case photorec recovered over 95% of 50,000 jpg images of official documents and indexes linking them to the paper documents. In another case it recovered 1,490 files out of 1,510 files, including the thesis, that were on a grad student's corrupted laptop HD. For the Dept of Revenue I had photorec also save corrupted files and on those I used a binary editor to extract readable portions of documents and indexes.

[oshunluvr]

Assuming you didn't write anything to the old /home partition, the calibre folder and it's contents are likely still intact there. extundelete should be able to restore the original folder.

As far as the new sda4, try running sudo parted -l and see what it lists as the file system. To attempt mounting it as ext4, try sudo mount /dev/sda4 -t ext4 /mnt/somewhere

[oshunluvr]

I've rarely seen a successful recovery of data by restoring starting and ending sectors of a partition, even if one had the foresight to write them down before changing them.

I had a disk several years ago - like 12-14 - that I re-partitioned in error. I was able to use testdisk to restore the entire thing without losing anything on it. After that, I kept off-line backups of all my partition tables just in case. I did actually run some testing using dd to backup and restore partition tables with total success.

Now, with the relative low cost of storage media and the ease of backing up with btrfs, I just make full backups more often. Also, GPT tables have a built in backup that can be loaded and relocated if a sector goes bad. Drives are much more robust these days.

The primary thing when something goes wrong with accidental deletes or partition table troubles is to stop using the disk immediately. IF you do that, your chances of at least a partial recovery are very good.

[oshunluvr]

Assuming you didn't write anything to the old /home partition, the calibre folder and it's contents are likely still intact there. extundelete should be able to restore the original folder.

As far as the new sda4, try running sudo parted -l and see what it lists as the file system. To attempt mounting it as ext4, try sudo mount /dev/sda4 -t ext4 /mnt/somewhere

Also, sudo file -sL /dev/sda4 may reveal some info...

[GreyGeek]

....

The primary thing when something goes wrong with accidental deletes or partition table troubles is to stop using the disk immediately. IF you do that, your chances of at least a partial recovery are very good.

The first example I mentioned was about a Xerox Unix imaging system that created 50Kb (IIRC) images of documents citizens gave to state employees (tax returns, etc...). The clerks kept getting errors on the system and rather than reporting them the supervisor, who knew nothing about computers, just told them to reboot. They did that for awhile (days? weeks?) until the system refused to boot. Since I was the "Unix" (Linux) expert I was given the HD to attempt to recover the files. I plugged it into my SuSE 6.3 PostgreSQL server and used dd to create a 1Gb image on my server's 2Gb drive. (BIG disks in those days ) I let the program run over night. Time stamps indicated it took about 10 hours to do the scan.

[rwbehne1]

Assuming you didn't write anything to the old /home partition, the calibre folder and it's contents are likely still intact there. extundelete should be able to restore the original folder.

As far as the new sda4, try running sudo parted -l and see what it lists as the file system. To attempt mounting it as ext4, try sudo mount /dev/sda4 -t ext4 /mnt/somewhere

* As I indicated, the old /home is ok. No need for any more work there - for now.

* As to the old /usr/local, which used to be where sda4 now is, and which we have recovered,
BUT of which I had previously moved to where sda5 is and renamed to "tail", mysteriously has been labeled as btrfs when it was and still should be ext4,
AND of which still contains my Calibre directory with the entire database of my ebook collection, testdisk reports:

TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

Disk /dev/sda - 3000 GB / 2794 GiB - CHS 364801 255 63

The harddisk (3000 GB / 2794 GiB) seems too small! (< 3275 GB / 3050 GiB)
Check the harddisk size: HD jumpers settings, BIOS detection...

The following partitions can't be recovered:
    Partition               Start        End    Size in sectors
>  MS Data               4532524928 5860663167 1328138240 [tail]
  MS Data               5069264768 6397403007 1328138240 [tail]

So apparently the only possible way to recover my Calibre directory from there is by using photorec. Or dd. Unless there's another method.

PLEASE: I don't want to hear another word about off-line backups from ANYBODY - unless it's to hear that someone has bought me a brand new WD 3T drive and is shipping it overnight to me. THAT will solve many problems at once. I simply cannot do what I do not have the resources to do. I have to work with what I have got, and my too-small external drive is one of SIX, of which FIVE are useless (input/output error) and this one remaining is FLAKEY. I formatted it to btrfs, but when I tried to copy files to it, it took an hour to copy what should have taken three minutes. It keeps stalling.

So I need to try to recover the Calibre directory from sda4, period. That has to be done before screwing with anything else.
Then, I want to install a small system, exclusively to sda2, without damaging anything else!

So, without any more diversions to other topics, Please everyone, one thing at a time, focus! At this moment the priority is recovering the Calibre directory, until the new drive from GrayGeek (or whoever insists on me first doing a backup) arrives here.

So now I have to learn how to use photorec.

Ok, I am running photorec now. Since the only space large enough is on sda3 I am putting the recovered files there, in the Lost & Found directory (apropos, I think.)