[RESOLVED] NVME and multipath causing problems.

[oshunluvr]

I have two Adata Blade nvme drives I bought at the same time. They sat on the shelf for awhile before I installed them. Here's the oddball issue: One of the two drives has firmware the Linux Ubuntu kernels past 5.4.0 don't like and therefore won't access it.

The error at boot time is

Aug  1 12:47:55 office kernel: [    1.989585] nvme nvme3: pci function 0000:24:00.0
Aug  1 12:47:55 office kernel: [    1.993603] nvme nvme3: Invalid MNAN value 0
Aug  1 12:47:55 office kernel: [    1.993610] nvme nvme3: Removing after probe failure status: -22​

Kernel 5.4.0 uses this drive just fine. After that there was a patch to the kernel that reads the mnan value (whatever that is) and doesn't like the zero I guess.

The apparent fix is to update the firmware BUT Adata (never buying one of their products again) only offers a firmware update by using their Windows based "Toolbox". I've tried using it with Wine (nope) and a USB with Windows installed on it, but it just crashes the OS.

I suppose I could figure out the commit that produced the problem, but compiling kernels is something I have don;t in almost 20 years and not keen on doing that again.

I can't remove the drive (to update it on a different PC) without breaking down my water cooling system so that's a large chore. I grimace at the idea I have to install Windows just for this one task but I guess I'm stuck unless someone else has a suggestion.

If I could get my hands on the firmware I can use nvme-cli tools to install it, but as far as I can tell I can't get the firmware without the crappy toolbox (everyone complains about it). I've been hanging on many months hoping for a change int he situation, but nothing yet.

I would love to be able to COPY the firmware off of the "good" drive and then upload it to the other, but I can't seem to locate a tool to do that.

Looking for suggestions...

...other than installing Windows.

[GreyGeek]

Do drives have licensing #'s in the eprom to check against a firmware burn in order to prevent that?

[oshunluvr]

I have no idea.

[oshunluvr]

OK, well I got around to researching this more and discovered a kernel boot parameter: "nvme_core.multipath=0". It turns out the MNAN thing and another thing labled ANA are part of a NVME feature called "multipath". Turning this off disables the MNAN check and thus the drive is now available.

I attempted to research it but didn't find much in the way of what the heck multipath actually does or if I need/want it. Seems to not be installed by default in KDEneon but Suse and RedHat talk about it. They often referred to ""nvme over fibre" so I suspect it's a tool for much higher end systems than a home desktop PC, although it appears I have the ability to enable it. The way they described it makes it seem like a native NVME version of LVM. Identical NVME drives can be logically attached via multipath and then accessed as a single device. If I have gleaned this correctly, I don't need the ability.

But at least I can use the drive now...

[oshunluvr]

A bit more research into the drives reveals this on the drive that worked with multipath enabled in the kernel:

fr : 3.2.F.46
cmic : 0
[3:3] : 0 ANA not supported
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port

while the drive that only worked with multipath disabled showed this:

fr : 3.2.F.2A
cmic : 0x4d
[7:4] : 0x4 Reserved
[3:3] : 0x1 ANA supported
[2:2] : 0x1 SR-IOV
[1:1] : 0 Single Controller
[0:0] : 0x1 Multi Port

So it seems that the source of the problem is indeed the firmware version ( "fr" in the above ), but in an opposite way. As long as multipath was enabled, the drive with multipath in the firmware did NOT work. Once multipath was turned off, they both work.

[oshunluvr]

Also worth noting, the two other NVME drives (Samsung) also do not have multiport enable in their firmware.

Off to play with my drive space!!!

[GreyGeek]

How much drive space did they add? (Inquiring minds want to know! )

[oshunluvr]

Each of the 4 nvme drives are 1tb. I also have two platter drive of 2tb each that are the backups. I will probably post about the nvme drives and the differences of the interfaces.

[oshunluvr]

Final note: I suspect this isn't an issue with Win10 because it doesn't have multipath support by default like the latest Linux kernels do.

[GreyGeek]

1TB drives seem so small these days!

Back in the early 80's a client running an Apple ][ with two DISK ][ drives ran out of drive space. I had written an Point Of Sale app for his parts and fasteners store.
I attached a Corvus 5MB HD to his Apple. One had to let it "warm up" for 5 minutes before turning on the Apple.
http://www.cbm.sfks.se/pics/corvus-2.jpg
Backing up the Corvus was accomplished by using a VCR. It took about 20 minutes to back up the Corvus. The client backed up twice, first over the lunch break and secondly after work.
We both though "Wow, that's a barn! We won't need additional space for years!" We were wrong. In less than a year he filled up the Corvus 5MB and replaced the Apple+Corvus with an IBM XT with a 10MB HD, for about the same price as the Corvus. I had taught his son how to program (he later earned a PhD in computer programming) and he kept modifying the software as his dad came up with new ideas. I stopped tracking their expansion when they moved to the first 1GB Barracuda drives.

On Amazon 2 & 4 TB drivers are getting cheap!