fsck and/or e2fsck fail on check - system will not boot

[bag]

OK, so I'm stuffed... Normally, I know where I am and which direction to search but here I have no clue... The usual searches on this forum haven't turned up anything to help me so I'm all alone on this particular one...

Normally, I've also got a back-up too - but not this time, so a problem with my RAID(1) setup is scaring the hell out of me. I know, I know, but Keep(?) has never really worked properly for me and hadn't been working at all since I upgraded to 8.10 and I haven't had time to fix it yet...

There's only one word for this moment... HELP!!!!

Let me start at the start - usually a good idea.

We had a powercut. I presume that my UPS shut everything down gracefully - it usually does. But I can't be sure. It might seem not given all the problems I'm having.

However, on booting - I got all sorts of problems. Mostly with filesystems. It checked stuff, told me some was OK, some wasn't but that it was dropping to a shell. I could Ctrl-D to continue booting - which I did, but the system just hanged...

I left it for sufficient time to be sure that it wasn't doing something useful - and so after 30 minutes I Ctrl-Alt-Del kick-started it.

Same thing - more or less.

I've tried both the latest kernel and the failsafe version, and the previous one and its failsafe - nothing will get me even to a graphical log-in prompt.

I can get to a command line (on the failsafe kernel of the previous version) but that's the only one. The story I get then is all about an Invalid argument whilst trying to open my /dev/md1 and check for an ext2 filesystem. Note, I say ext2 here - specifically noted because the log file seems to show a check for an ext3 file system...

It tells me that the superblock could not be read. There is a log in /var/log/fsck/checkfs which, when viewed in nano explains the following:

"fsck.ext3: Invalid argument while trying to open /dev/md1
/dev/md1:
The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and ir really contains an ext2 filesystem (and not swap or ufs or somethign else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

fsck died with exit status 8"

So, I dutifully exited nano and ran "e2fsck -b 8193 /dev/md1" and got almost exactly the same - the output was:

"e2fsck 1.41.3 (12-Oct-2008)
e2fsck: Invalid argument while trying to open /dev/md1

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and ir really contains an ext2 filesystem (and not swap or ufs or somethign else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>"

The problem is that it tells me to run the exact same command as the one I've just run. So I'm more than a little confused.

So then, what can I do, what can I run to try and fix the superblock if it's broken? Please don't ask what filesystem the md1 RAID is, I can't remember. I followed a How-to at the time and it took a lot of tweaking - but I really can't remember what it was.

How can I check that? What do I need to do to ensure that I can recover the disks?

Please note that the log file shows that all the other filesystems that it had checked had been fine. sda1 has 7.9% non-contiguous files (which will need sorting out later on) but other than that, nothing to report. All show up as clean.

Other info that might prove useful: Last week, following an update (don't know which one caused the issue) I started to have boot problems. It was the / partition that caused the issue. There didn't seem to be enough time to pick up the partition - I solved it by a suggestion on one of the posts here - add a rootdelay=500 into the kernel boot parameters in grub. Worked a treat. Not had any problems with it since then.

I am REALLY stuck here on this one. Desperately need to get into the system to at least make a backup of the data if I can.

Any help would be very much appreciated. If you need more information or details to help me solve this then please just let me know and I will get it for you.

Many thanks in advance.

Bag.

[mfabry]

Re: fsck and/or e2fsck fail on check - system will not boot

You might try booting from Live CD to fsck faulty partition. Use fdisk -l <device> to list all the partitions (should show filesystem type).

[dibl]

Re: fsck and/or e2fsck fail on check - system will not boot

It tells me that the superblock could not be read.

That's a bad one -- it typically indicates a failed drive (or USB stick, in my recent case). Sorry.

[toad]

Re: fsck and/or e2fsck fail on check - system will not boot

There is a utility called SystemRescueCd available from http://www.sysresccd.org/Main_Page

It has a bunch of tools on it incl. testdisk

HTH

[bag]

Re: fsck and/or e2fsck fail on check - system will not boot

Ok folks, so I have an update for you.

Don't know what happened. And I mean I REALLY don't know what happened because I did absolutely nothing, but I rebooted once more for luck and it went through and booted normally!!!

And by that I mean that I'm into KDE4.1, using a graphical interface, I can see the disk in question, and read fine from it. Everything works as it did before. OK, OK, some things are broken - but then they were before too, so I can't complain...!

In my eyes, this is a minor miracle. I am busy making backups of everything on the RAID disks right now. Once I'm sure that it's all done, I'll reboot it to see if it does it again or if it fails...

So then, I have a few questions before I reboot.

1. What should I do whilst the machine is up and running before I do a reboot to try and 'fix' a problem that no longer seems to be there? I want to maximise the chances that it will reboot perfectly again... so what can I do to repair and prepare it all before I reboot?

2. It turns out the RAID drive is ext3. The external hard drive that I'm backing up to is VFAT. When I copy across the files from one to the other, I have just quickly checked that the number of folders and files is the same. It is. But the space required for storing them is different (when comparing the number of bytes). Does this mean that the copy is corrupted and not complete? Or does it mean that since different file systems store things differently, they will need different amounts of space for the same data? For info, the vfat version (which is the copy/back-up) is larger!

A sub-issue of that second one is: How can I quickly check the data integrity of the backup? I had a notion that an md5 hash check of one against the other would prove integrity - but have no clue if this is true and how I would go about it. Also, if the file systems store data differently, would that mean a different md5 hash for each structure?

Questions questions... Those clever enough to know any answers are most humbly invited to increase my knowledge and understanding...

Thanks,

Bag.

[dibl]

Re: fsck and/or e2fsck fail on check - system will not boot

Caveat: I have no particular expertise in data recovery.

I think you're doing the right thing, copying off your data to the external disk. There is a command line method to compare the files in two Unix/Linux directories, but I'd have to research it, and you can do that as well as I.

As far as "validating" it, I don't know what you can really do, given that the source of the data (the failing hard drive) is of questionable integrity, and might have already compromised the data files. In other words, if a file copied to your external disk is discovered to be corrupt, what are you able to do about it? It seems to me you can either send the failing drive to a commercial data recovery service and pay for their best efforts, or else accept any losses that you are able to discover -- I hope there actually aren't any.

Hard drive failures can be sudden and catastrophic, or they can be gradual and incremental. It sounds like your situation might be of the latter variety -- it finally did read the boot sector successfully, but we don't know whether it ever will again, or not.

I've considered a RAID(0) setup myself -- I have 3 WD-1500s and I could use a pair of them in a RAID(0) array and get some performance improvement. But I'm painfully aware that the risk of failure of two drives working together is exactly twice the risk of failure of one drive -- that's what has been holding me back. I think you just helped me clarify my thinking on it .....

[barbolani]

Re: fsck and/or e2fsck fail on check - system will not boot

bag, neither I'm an expert on data recovery but....

The file sizes, as reported by the ls command or file properties, of the copy and the original should match exactly to the byte. Otherwise, your copy is corrupted.

However, if you're comparing the total sizes by, for example, comparing the output of the fc command or the du command, then yes, depending on the file system of each partition they can differ even if the copies are not corrupted. This is because each file system allocates space in different chunk sizes and is normal.

For extra security, you can use the cmp command to compare the original and the copy.

[toad]

Re: fsck and/or e2fsck fail on check - system will not boot

The file sizes, as reported by the ls command or file properties, of the copy and the original should match exactly to the byte. Otherwise, your copy is corrupted.

Okay, so I'm a bleeding layman as well, but don't file sizes depend on storage units? If you say that your basic unit is 4 bytes instead of 8 surely file size changes with it... Now if you copied from vfat 16 to vfat 16 things should stay the same, but from vfat 16 to vfat32 things may change. Same for vfat 16 to ext3. I think we need some help here from somebody in the know

[barbolani]

Re: fsck and/or e2fsck fail on check - system will not boot

Disk drives talk to the Linux driver in terms of larger units of allocation, I think that they are sectors.

Storage units are allocated according to file sizes. The filesystem keeps the actual file size as part of the metadata (inode) information, but actual space allocation in the disk is done in sectors.

Consider for example a FAT file system partition of 1GB with 4096 byte sectors. The FAT file system stores each file in whole sectors, plus some space used in the FAT for the metadata (file name, attributes and so on) So we have 1GB = 2^10 MB = 2^20 KB = 2^30 bytes. Which means 2^30/4096 = 262144 sectors.

On such a partition you would not be able to store more than 262144 files, even if each one is a single byte in size. Thus, you'll need 1GB to store 262144 bytes as each 1 byte file takes a whole sector. And that's the best case, you'll have to reserve some space for the metadata.

Of course FAT is very simple, and more advanced filesystems, such as NTFS, reiser, ext3 etc make much more efficient use of the disk space, but the general rule still applies: disk space usage will not match actual file sizes.

[toad]

Re: fsck and/or e2fsck fail on check - system will not boot

Thanks, barbolani. My terminology was way out - sectors was the word I was looking for.

If I understand you correctly a bunch of files should take up the same space regardless of the file system they are on - correct?

@ bag

Have you dared rebooting yet ?

[bag]

Re: fsck and/or e2fsck fail on check - system will not boot

I'm back... and I have rebooted.

It ain't pretty.

I have a system that seems to boot or show the /dev/md1 drive depending on which way the wind blows... At the moment, I'm in , but the info on my /dev/md1 "Common" drive is not readable. And no USB drives attached to the machine are readable either.

There are some interesting things in the current dmesg though. I'm still very new to this, but I'll copy and paste some things in below that others more experienced may be able to help interpret for me.

[  13.782072] usb-storage: device scan complete                                                
[  13.784938] usb-storage: device scan complete                                                
[  13.785079] usb-storage: device scan complete                                                
[  13.786505] scsi 7:0:0:0: Direct-Access   Verbatim STORE N GO    5.00 PQ: 0 ANSI: 0 CCS                        
[  13.787023] scsi 8:0:0:0: Direct-Access   Generic USB SD Reader  1.00 PQ: 0 ANSI: 0                          
[  13.787521] scsi 8:0:0:1: Direct-Access   Generic USB CF Reader  1.01 PQ: 0 ANSI: 0                          
[  13.788020] scsi 8:0:0:2: Direct-Access   Generic USB SM Reader  1.02 PQ: 0 ANSI: 0                          
[  13.788518] scsi 8:0:0:3: Direct-Access   Generic USB MS Reader  1.03 PQ: 0 ANSI: 0                          
[  13.790013] Driver 'sd' needs updating - please use bus_type methods

Although this is not directly related to my internal drives (which are SATA), would the driver sd have anything to do with this?

Also, internal SCSI disks are showing problems:

[  13.898626] sd 0:0:0:0: [sda] Attached SCSI disk                                              
[  13.898725] sd 1:0:0:0: [sdb] 781422768 512-byte hardware sectors (400088 MB)                                
[  13.898752] sd 1:0:0:0: [sdb] Write Protect is off                                             
[  13.898755] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00                                            
[  13.898800] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA                     
[  13.898873] sd 1:0:0:0: [sdb] 781422768 512-byte hardware sectors (400088 MB)                                
[  13.898898] sd 1:0:0:0: [sdb] Write Protect is off                                             
[  13.898900] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00                                            
[  13.898945] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA                     
[  13.898949] sdb: unknown partition table                                                  
[  13.907156] sd 1:0:0:0: [sdb] Attached SCSI disk                                              
[  13.907244] sd 2:0:0:0: [sdc] 781422768 512-byte hardware sectors (400088 MB)                                
[  13.907271] sd 2:0:0:0: [sdc] Write Protect is off                                             
[  13.907274] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00                                            
[  13.907319] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA                     
[  13.907395] sd 2:0:0:0: [sdc] 781422768 512-byte hardware sectors (400088 MB)                                
[  13.907419] sd 2:0:0:0: [sdc] Write Protect is off                                             
[  13.907421] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00                                            
[  13.907466] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA                     
[  13.907470] sdc: unknown partition table                                                  
[  13.918025] sd 2:0:0:0: [sdc] Attached SCSI disk                                              
[  13.920260] sd 5:0:0:0: [sdd] 490232832 512-byte hardware sectors (250999 MB)                                
[  13.921257] sd 5:0:0:0: [sdd] Test WP failed, assume Write Enabled                                     
[  13.921261] sd 5:0:0:0: [sdd] Assuming drive cache: write through                                      
[  13.921881] sd 5:0:0:0: [sdd] 490232832 512-byte hardware sectors (250999 MB)                                
[  13.923256] sd 5:0:0:0: [sdd] Test WP failed, assume Write Enabled                                     
[  13.923258] sd 5:0:0:0: [sdd] Assuming drive cache: write through                                      
[  13.923301] sdd:<6>sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                    
[  15.787142] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[  17.620140] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[  17.620145] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[  19.454134] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]

The response of No sense [current] is repeated about a gazillion times during the boot sequence.

Then later on, after a load of "No sense [current]"s it posts up...

[ 201.054215] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 202.802145] md: md1 stopped.                                                        
[ 202.888214] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]

This is presumably the system giving up on getting info from the /dev/md1 RAID array.

However, it then continues with another couple of hundred of No Sense [current]s until it pops up with a query about a kernel bug...

[ 375.255278] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 375.255282] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 377.088280] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 377.088284] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 378.944281] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 378.944284] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 380.777281] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 380.777285] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 382.611282] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 382.611285] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 382.700640] PM: Starting manual resume from disk                                              
[ 382.700643] PM: Resume from partition 8:2                                                  
[ 382.700645] PM: Checking hibernation image.                                                 
[ 382.700826] PM: Resume from disk failed.                                                  
[ 382.750817] kjournald starting. Commit interval 5 seconds                                         
[ 382.750836] EXT3-fs: mounted filesystem with ordered data mode.                                       
[ 384.444287] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 384.444293] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 386.278287] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 386.278294] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 388.075729] udevd version 124 started                                                    
[ 388.111288] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 388.111294] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 388.547246] i2c-adapter i2c-0: nForce2 SMBus adapter at 0x2d00                                       
[ 388.547309] i2c-adapter i2c-1: nForce2 SMBus adapter at 0x2e00                                       
[ 388.751372] input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3                            
[ 388.775217] ck804xrom ck804xrom_init_one(): Unable to register resource 0x00000000ff000000-0x00000000ffffffff - kernel bug?         
[ 388.781411] ACPI: Power Button (FF) [PWRF]                                                 
[ 388.781527] input: Power Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input4                       
[ 388.808036] ACPI: Power Button (CM) [PWRB]                                                 
[ 388.975394] parport_pc 00:07: reported by Plug and Play ACPI                                        
[ 388.975439] parport0: PC-style at 0x378, irq 7 [PCSPP]                                           
[ 388.997047] parport0: Printer, Hewlett-Packard HP LaserJet 5L

After a couple of hundred additional No Sense [current]s then it has some problems reading a superblock (whatever that is! - I'll google it in a second).

[ 582.491360] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 582.491364] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information                                  
[ 584.100478] kjournald starting. Commit interval 5 seconds                                         
[ 584.100674] EXT3 FS on sda1, internal journal                                                
[ 584.100679] EXT3-fs: mounted filesystem with ordered data mode.                                       
[ 584.146969] kjournald starting. Commit interval 5 seconds                                         
[ 584.147252] EXT3 FS on sda7, internal journal                                                
[ 584.147257] EXT3-fs: mounted filesystem with ordered data mode.                                       
[ 584.185321] kjournald starting. Commit interval 5 seconds                                         
[ 584.185621] EXT3 FS on sda8, internal journal                                                
[ 584.185625] EXT3-fs: mounted filesystem with ordered data mode.                                       
[ 584.220926] kjournald starting. Commit interval 5 seconds                                         
[ 584.221186] EXT3 FS on sda6, internal journal                                                
[ 584.221190] EXT3-fs: mounted filesystem with ordered data mode.                                       
[ 584.254864] kjournald starting. Commit interval 5 seconds                                         
[ 584.255095] EXT3 FS on sda5, internal journal                                                
[ 584.255099] EXT3-fs: mounted filesystem with ordered data mode.                                       
[ 584.255346] EXT3-fs: unable to read superblock                                               
[ 584.324368] sd 5:0:0:0: [sdd] Sense Key : No Sense [current]                                        
[ 584.324373] sd 5:0:0:0: [sdd] Add. Sense: No additional sense information

then there are a few more hundred No Sense [current]s mingled in with some USB info and devices found... and that's it.

All in all, it doesn't look good. But I have some questions to clarify.

If the sd driver is wrong/needs updating, could this change things? How do I update it?

What exactly does "sd 5:0:0:0: [sdd] Sense Key : No Sense [current]" mean? Am I right in thinking that it means that the sensors attached to the drives are not working? Or that the reporting mechanism is screwed up somehow? Would a replacement "sd" driver fix that? Shot in the dark but theoretically, could the sd driver be corrupted and not able to read a perfectly good sensor? (He says, hoping on hope)

What does it really mean when the system can't read the superblock? I know it's bad, but does it mean that the data on the drive is corrupted, or that the hard drive itself is stuffed? Or both? In theory, could I wipe the two 400GB drives that make up my mirrored RAID drive and start again? Are they actually 'physically' OK and usable now that I have backed up my data?

Have I lost my two hard drives and need new ones to replace them?

Your answers to these questions (and more!) are eagerly awaited. Thanks in advance,

Bag.

[toad]

Re: fsck and/or e2fsck fail on check - system will not boot

My "guess" is that the sd driver is perfectly fine seeing that no one else has every complained about it. It would also be very unLinux like to suddenly not update something or get its knickers in a twist 'cos of some driver that previously functioned perfectly okay.

The only option I see is that one or both of your drives are telling you politely that they've had enough. Save your essential data, use dselect to get a list of packages (Rog wrote an excellent post on rescuing/transferring a system but I can't remember where...) and install on another (new) drive.

Now test your disks thoroughly.

My 2cs' worth.

[bag]

Re: fsck and/or e2fsck fail on check - system will not boot

Toad,

Thanks for that. The only reason is that I was hoping that since the system was claiming that the sd driver needed updating, and that all errors after that seemed to quote the sd driver, that if I fixed the driver, maybe I'd see something different...

I can dream...!

A quick Google shows that this a kernel issue - but doesn't seem to be causing major problems, or be high enough on the list to be fixed... It was reported in January 2006 - and appears to still be unfixed.

I found an extremely informative article in multiple chunks at http://www.cyberciti.biz/tips/unders...uperblock.html which goes on to explain how to recover from a file system breakdown. It has recovery commands for finding the superblock and replacing it with a back-up (which the filesystem handily makes for you, in case this very thing happens).

However, it states that for RAID it becomes more complicated. It links to the article here - http://www.tldp.org/HOWTO/Software-R...x-HOWTO-4.html which states that I can rebuild the array (or try at least) using command line tools. The only problem is that the utilities mentioned don't seem to work (are not installed) and I'm struggling to find suitable tools that I could use...

If I was to try and just recover the superblocks on each drive - would that solve the issue or make it worse (could it be worse?)? Is it worth a try - given that I've backed up the data?

Can anyone point me in the right direction to be able to find tools and commands that I can try? I have mdadm installed - but the man page seems a bit 'dense'. There are so many options in there that I can't sort out the wheat from the chaf. Can't really seem to find a command suitable to do what I need... Does anyone have any expertise on RAID recovery and mdadm?

From my limited knowledge, and the errors I see, the main problem is not being able to see the superblock. If I can sort that out, I "might" be one step closer to recovery...

Any advice is very much appreciated.

Bag.

[bag]

Re: fsck and/or e2fsck fail on check - system will not boot

OK, so a quick update now that I have done some more searching...

Apparently, according to this page - http://kev.coolcavemen.com/2007/03/h...d-superblocks/ creating a RAID array does not overwrite the data that already exists on the disks there. As such, you can create a new array/recreate an array over the top of the existing one and it will recover itself - potentially with the data still intact on the discs.

I have tried this to a point - and then backed out. The messages I was getting (see below) didn't match the ones shown in the post so I wasn't sure if it would actually create without overwriting or delete the files... Got scared and backed down.

Anyone know if this might help me? Should I go for it? Any advice from people more knowledgeable would be very helpful.

thanks.

Bag.

~$ sudo mdadm --create /dev/md1 --verbose --level=1 --raid-devices=2 /dev/sdb /dev/sdc
mdadm: /dev/sdb appears to contain an ext2fs file system
  size=390711296K mtime=Thu Jan 1 01:00:00 1970
mdadm: /dev/sdb appears to be part of a raid array:
  level=raid1 devices=2 ctime=Fri Sep 7 19:12:51 2007
mdadm: /dev/sdc appears to contain an ext2fs file system
  size=390711296K mtime=Sun Dec 7 09:05:55 2008
mdadm: /dev/sdc appears to be part of a raid array:
  level=raid1 devices=2 ctime=Fri Sep 7 19:12:51 2007
mdadm: size set to 390711296K
Continue creating array? n