How I (sort of) conquered UEFI

[SteveRiley]

Since UEFI will be the future, I figured I needed to quit my bitching and set about getting to know this unholy beast. For several hours this weekend I've read way too much, installed Ubuntu way too many times, and even managed to temporarily kill a portion of my UEFI's NVRAM!

UEFI is so new and different that it can appear very daunting. You can permanently ruin a UEFI if you aren't careful. But if you take the time to master it, you'll discover that its framework eases the burden of installing and maintaining operating systems. The old concept of burying a boot loader in some unused drive sectors is history, as is the need to worry about primary-vs-extended partitions and remembering which one is active. Partition #1, formatted FAT-32, stores all UEFI-related information in /boot/efi. Each operating system places its own boot loader in a subdirectory beneath this head -- in the case of Ubuntu, it's EFI/ubuntu. I'm happy to report that I now have today's build of Precise running in UEFI mode on my ThinkPad X1.

The journey was tortuous and at times I nearly gave up. But BIOS+MBR is on the way out, so undertaking this knowledge adventure was necessary. First step: partition my SSD. I'm picky about this, I like to ensure proper alignment. Since Ubuntu's Partman tool can't offer that much flexibility, I booted with a USB containing GParted Live. My X1 has a 128 GB SSD. With gdisk running in a console, I partitioned the drive thusly:

• Extra menu ... Zap all existing GPT structures
• Main menu ... New GPT (GUID partition table)
• Extra menu ... Sector alignment ... 8 sectors (which maps to my SSD's 4kB sectors)
• Main menu ... New partition (#1) ... default start (changed by alignment config from 34 to 40), size +512M, type EF00 (EFI system)
• New partition (#2) ... default start, size +30G, type 8300 (Linux filesystem)
• New partition (#3) ... default start, size +80G, type 8300 (Linux filesystem)
• New partition (#4) ... default start, size [see below], type 8200 (Linux swap)
  • Want partition alignment to end correctly
  • (last usable sector - default start) / 8
  • toss fraction, multiply result by 8, enter this as number of sectors
• Write partition table and exit

Now it's time to install. The first thing I learned: my usual method of building a PC, from the netboot boot.img.gz, doesn't work! That installer lacks UEFI support. I'd install, but find no /boot/efi. Of course I didn't realize this until after a couple hours of futzing around...only when I configured the UEFI not to automatically fall back to legacy mode. Once I made that change, and the USB drive failed to boot, I realized what was going on. So I downloaded the alternate installer instead.

After reading a lot of stuff, and based on the experiences of others, I first tried to get ELILO working as the boot loader for Ubuntu. UEFI duplicates a lot of what GRUB does anyway, and ELILO is simpler, so this seemed a natural choice. But no matter how many variations of configurations I attempted, the kernel simply wouldn't boot. The initrd loaded without error, but the computer would reboot every time the kernel was about six seconds into its loading process. I couldn't find any logs anywhere, and cranking up ELILO's verbosity mode didn't reveal anything. The ELILO setup process is a bit opaque, and once while using efibootmgr to examine the UEFI's NVRAM, I accidentally deleted the item in variable Boot0004. Whoops! Oh crap, did I just destroy this $1200 laptop?

Turns out that's the Intel Management Engine, which I had already disabled. But again since I'm picky, and I don't like broken stuff, I had to divert and learn about the EFI Shell. Here's where I began to realize that UEFI's maintenance is simpler once you get over the learning curve. I formatted a USB drive as FAT32, created the directory hierarchy /boot/efi, and copied the EFI Shell file into that as bootx64.efi. Reboot and voila, the EFI Shell. Some tips if you're following along:

• Enter mode to find out what text modes UEFI can accommodate on your PC
• Enter mode num num to obtain the densest font you can
• help is your friend; learn more with help command
• Follow every command that displays something with -b to view the output one page at a time

Since I happen to have another ThinkPad handy (the T520, on which I'm typing this post) with the same UEFI, I booted into that one and copied the contents of Boot0004 onto the USB. Each of the below is a command typed at the prompt:

• fs0:
• dmpstore boot0004 -s b4var

The first command switches the "context" to the USB drive. The colon is required, it's part of the device's name. The second command dumps the contents of the boot variable; the switch means save to a file, and I called the file b4var. It landed in the root of my USB drive. I exited the shell on the T520, moved the USB drive to the X1, and booted the shell. Then I restored the missing variable with these commands:

• fs1:
• dmpstore boot0004 -l b4var

The shell identified my SSD as permissible for UEFI functions because of its GPT and therefore assigned a different device ID to the USB drive. So first I had to switch to its context. Then I used the same command as before (yeah, "dump" is a bit silly here, but it is what it is) with a different switch, this time indicating a load.

Aside: during my first attempt at this, I hadn't realized that the device names would be different on the X1. fs0: didn't have the file! Oh no, is there some hidden digital signature that prevents the X1 from seeing a variable dumped from the T520? I double-checked the drive on another PC, and yeah, b4var was present. Then I discovered the map command, which lists all devices and their names. When I saw two "fs" devices, I knew what to do. After switching to fs1:, and issuing an ls command, I saw the file. So let this sink in for moment, and I believe you'll see why I'm beginning to think that UEFI is kind of cool. Everything's a file. There's a shell. I can view parts. I can move parts around. I can recover from a mistake. Arguably, it ought to be more difficult to delete variables that can't be regenerated, and that's a major design flaw. But I'm impressed with what I've learned here, and can understand how UEFI simplifies maintenance tasks -- once you get over the tortuous educational process.

So anyway...eager to verify that my recovery worked, I booted the Ubuntu alternate installer into rescue mode, ran efibootmgr, and indeed -- Boot0004 was back!

Alright. So now let's try yet another install, and allow GRUB. The installer has always detected partition #1 and correctly marked it as the EFI system partition. When it came time, near the end, to install a boot loader, rather than skipping that, I just decided to let the installer automatically set up GRUB. And to my surprise, it worked perfectly. Contrary to what I've read elsewhere, Ubuntu's setup scripts appear to have shed their bugs. grub-efi, not grub-pc, was downloaded and configured. And all in one step: unlike BIOS+MBR, with UEFI+GPT we know that boot loaders always go in whatever partition is type EF00 and in subdirectory /boot/efi. Gone is the chore of telling your installer where your boot partition is.

Time for first boot! And it's exactly what I expect. Everything is as it should be. I've completed my pre-KDE steps and...yep, kde-standard has just finished installing. Time for some pruning and grafting, then one last reboot and I'll be back into my favorite desktop on what's rapidly becoming my favorite laptop. See y'all in a bit.

[ardvark71]

Hi Steve...

Could I encourage you to submit your work and findings to the developers at Canonical? From what I see here, most folks are not going to bother learning what you have to get Kubuntu or any other distro to install. Perhaps a set of simplified instructions or, even better, installer operations can be created to make working with or bypassing UEFI much easier.

Another idea: Do you think the developers can make an "open source," linux friendly EUFI available with the various distros? I figure if Microsoft can add to UEFI code, so can Canonical or Red Hat or any other distro maker.

I admit I still don't know that much about it, so please forgive me if I sound ignorant.

Regards...

[SteveRiley]

Could I encourage you to submit your work and findings to the developers at Canonical? From what I see here, most folks are not going to bother learning what you have to get Kubuntu or any other distro to install. Perhaps a set of simplified instructions or, even better, installer operations can be created to make working with or bypassing UEFI much easier.

My lengthy post above is mostly of interest to those who like to tinker with the inner workings of their operating systems. The tl;dr version of what I wrote is this: Ubuntu Precise, with its defaults, shows enough maturity to work just fine with UEFI. So long as you don't go diverting from GRUB or fiddling around inside an EFI shell, you probably won't notice the difference. What Canonical should improve is Partman; it ought to be at least as smart as gdisk and get the partition alignment right. Performance will suffer if your first partition begins on sector 34.

Another idea: Do you think the developers can make an "open source," linux friendly EUFI available with the various distros? I figure if Microsoft can add to UEFI code, so can Canonical or Red Hat or any other distro maker.

Much of UEFI is already documented. I haven't investigated whether Intel provides all the source code under some kind of license.

What we should still worry about -- and I intend to become a public nuisance regarding this -- is the secure-boot aspects of UEFI. I refuse to accept the claims that such levels of tight hardware-software integration are necessary to keep a computer secure. The attack scenarios that secure-boot proponents envision are simply unrealistic; furthermore, secure-boot does nothing to address the most popular and dangerous attacks.

[GreyGeek]

....

Much of UEFI is already documented. I haven't investigated whether Intel provides all the source code under some kind of license.

I've learned that most of UEFI is open sourced under the GPL! Most vendors are using Intel's version, called Tianocore:
http://sourceforge.net/apps/mediawik...?title=Welcome

The documentation is a "mere 1,500 pages", and is not complete.

The various vendor proprietary binary drivers are linked to with LGPL parts of the UEFI code. This is exactly how Apple burgled KDE code in extracting Konqueror code to write Safari.

According to some folks I've read and the 1 hour video I watched of Matthew Garrett from RedHat, the code is HUGE and buggy. And hard to read. In another post I linked to Matthew's video, but I'll add it here again for easy access:

What we should still worry about -- and I plan to become a public nuisance regarding this -- is the secure-boot aspects of UEFI. I refuse to accept the claims that such levels of tight hardware-software integration are necessary to keep a computer secure. The attack scenarios that secure-boot proponents envision are simply unrealistic; furthermore, secure-boot does nothing to address the most popular and dangerous attacks.

It was obvious to me from Matthew's talk and dissection of the UEFI that it isn't even very good at preventing the implanting of HD Trojans or keyboard loggers. And, once Windows is booted UEFI have NO protective value in keeping Windows secure. Because of the certification requirements of Win8, contrary to Microsoft's claims, it is also obvious that the ONLY purpose of UEFI is to block Linux at all levels: stop replacing Win8 with Linux, stop dual booting Win8 with Linux, stop device drivers from releasing drivers for Linux. Say Acer gave Canonical a certification key to enable the Ubuntu kernel to boot. Does that key allow Kubuntu to boot? What about the 100's of device drivers that Ubuntu/Kubuntu connects to. How many keys does Ubuntu have to get so that it can load all the drivers necessary to drive the devices on all Acer computer models running Win8? Say Acer was very cooperative and it took them only 6 months to get things squared away... what about HP, Dell, Lenovo, etc.... How many people will Canonical have to hire to do all this certification work?

What about Linux Mint, PCLOS, openSUSE, RedHat, Fedora, etc.... Microsoft has created a shotgun certification tool designed to stop Linux in it tracks at ALL levels, simultaneously.

Microsoft, of course, claimed innocence and stated that certification was up to the PC OEMs, not them. The PC OEMs said that if they disable EUFI locking they lose their Win8 certification. It is the EULA - Windows refund all over again, but with much more serious consequences. It is also obvious that Microsoft has already greased the appropriate politicians and Federal agencies. But, Microsoft pulled their lamb's wool off to reveal the wolf they are when they announced their ARM device policy:
http://www.softwarefreedom.org/blog/...ocks-down-ARM/

Microsoft confirms UEFI fears, locks down ARM devices

By Aaron Williamson | January 12, 2012

At the beginning of December, we warned the Copyright Office that operating system vendors would use UEFI secure boot anti-competitively, by colluding with hardware partners to exclude alternative operating systems. As Glyn Moody points out, Microsoft has wasted no time in revising its Windows Hardware Certification Requirements to effectively ban most alternative operating systems on ARM-based devices that ship with Windows 8.


The Certification Requirements define (on page 116) a "custom" secure boot mode, in which a physically present user can add signatures for alternative operating systems to the system's signature database, allowing the system to boot those operating systems. But for ARM devices, Custom Mode is prohibited: "On an ARM system, it is forbidden to enable Custom Mode. Only Standard Mode may be enable." [sic] Nor will users have the choice to simply disable secure boot, as they will on non-ARM systems: "Disabling Secure [Boot] MUST NOT be possible on ARM systems." [sic] Between these two requirements, any ARM device that ships with Windows 8 will never run another operating system, unless it is signed with a preloaded key or a security exploit is found that enables users to circumvent secure boot.


While UEFI secure boot is ostensibly about protecting user security, these non-standard restrictions have nothing to do with security. For non-ARM systems, Microsoft requires that Custom Mode be enabled—a perverse demand if Custom Mode is a security threat. But the ARM market is different for Microsoft in three important respects:

But, being "enabled" doesn't mean that the hardware is compatible with dual booting.

[GreyGeek]

.....
I'm happy to report that I now have today's build of Precise running in UEFI mode on my ThinkPad X1.
......

Way to go!!! 8)

(Where is a genuflecting "I am not worth" icon when you need one?)

[GreyGeek]

Steve, both of your Lenovo laptops had the same Boot0004 file, it appears. Do you think that file (and the rest) would be the same as those used by other hardware vendors? I.e., what are the odds that the various vendors would make modifications to the UEFI code to make their various drivers unique and non-transferable?

Because, if the Boot0004 file is the same for all PCs which EUFI installed, then we could have someone send us a file if something happened to our copy.

[SteveRiley]

Steve, both of your Lenovo laptops had the same Boot0004 file, it appears. Do you think that file (and the rest) would be the same as those used by other hardware vendors? I.e., what are the odds that the various vendors would make modifications to the EUFI code to make their various drivers unique and non-transferable?

Because, if the Boot0004 file is the same for all PCs which EUFI installed, then we could have someone send us a file if something happened to our copy.

The NVRAM variables contain not only pointers to the various operating system boot loaders, but also pointers to the elements that make up your UEFI implementation. There is no standard for how these are ordered. My hack repair worked only because I happen to have two reasonably similar ThinkPads with the same UEFI. I doubt that a UEFI NVRAM from, say, a Dell would compare at all to that on an HP.

Later this evening I'll post a list of my X1's NVRAM variables so you can see what it looks like.

[SecretCode]

I've learned that most of EUFI is open sourced under the GPL!

The various vendor proprietary binary drivers are linked to with LGPL parts of the EUFI code.

It was obvious to me from Matthew's talk and dissection of the EUFI

It's UEFI, Mr Geek.

[GreyGeek]

Ya, I knew that. It was just a typo. I use "blUE FIn tuna" as a menomic to remember it, but it doesn't help when you fingers have menomics of their own.

Thanks for the heads up!

[sithlord48]

at least im not the only one to have noticed that. (also thank you gg for fixing you signature size)

Great job Steve! keep up the good work

[SteveRiley]

Great job Steve! keep up the good work

Thanks! I feel somewhat derelict here in my duties at KFN. Originally I promised to spend some time on the vBulletin theme, and then these two ThinkPads arrived...and, well, kid in a candy shop and all that. But Claydoh has started working on a theme, and it's looking very nice.

[SecretCode]

Ya, I knew that. It was just a typo. I use "blUE FIn tuna" as a menomic to remember it, but it doesn't help when you fingers have menomics of their own.

Thanks for the heads up!

Maybe this will help: according to UEFI or EUFI? EUFI means artificial uterus!

[lo_koraco]

Well, if you just hadn't fiddled and let the installer do its job in the first place...
The Ubuntu wiki is outdated in this segment, shamefully so, as is the "official documentation". Most of the UEFI boot stuff there is related to Macs. Who cares!

[SteveRiley]

Well, if you just hadn't fiddled and let the installer do its job in the first place...
The Ubuntu wiki is outdated in this segment, shamefully so, as is the "official documentation". Most of the UEFI boot stuff there is related to Macs. Who cares!

Historically, the installer hasn't performed properly, and yes, the documentation is poor.

UEFI is mystery. I don't like mysteries -- I want to know what's going on inside my computer. The only way to de-mystify something is to tear it down, understand the constituent parts, and then put it back together. I assume you've dismantled at least one radio or TV or toaster or hair dryer?

UEFI presents oddities, some that will be system-dependent for a while. For example...

• On these ThinkPads of mine, if I disable UEFI and allow only legacy BIOS, then GPT disks won't boot -- the UEFI boot manager throws an error. That's not supposed to happen.
• If the computer is in UEFI-only mode (that is, legacy BIOS is disabled), the installer for Oneiric still tried to create an EFI system partition. Result: no boot after installation. I haven't verified whether the installer for Precise still does this.
• My primary installation method, using a boot.img.gz, can't deal with UEFI and doesn't even know how to tell me that.
• GRUB is kind of pointless given that UEFI has a boot manager. ELILO is supposed to work, according to the experience of others; naturally it made sense to use that (simplicity is good) but it appears either to be broken now or just not compatible with my hardware.
• Partman's partitioning is sub-par.

Knowledge of this stuff matters. So long as UEFI remains mostly a black box, vendors will feel free to incorporate modifications and restrictions that limit functionality. But if more people become aware of how UEFI behaves and can explain what it does well, then we can make more intelligent arguments against vendor customization and lock-in. I have a greater understanding, now, between the UEFI itself and the optional secure-boot mechanism a vendor can choose to use. And since Apple's EFI is different enough from UEFI to cause real problems, again I believe that detailed understanding is a requirement.

[lo_koraco]

Don't get me wrong, I'm with you on this. I'm not of the opinion that Microsoft has plans to stick it to Linux with their recent decisions. Rather, I fear the whole industry is using the UEFI spec to achieve the same position as one sees on mobile devices - a "complete" system where you can only do a certain amount of things. My sister's laptop, an Asus, has UEFI. It's designed to run in legacy BIOS mode, but it doesn't boot from USB in such a configuration. Which means I have to install Ubuntu (I don't do release upgrades) via a CD. To paraphrase xkcd, every time I do that Thomas J. Watson loses a string in his harp (or gets poked with a trident, depending on his geographical location). But that's a small issue, since she uses Ubuntu. What if her computer had secure boot enabled, with Windows, got infected, and I tried to fix it with a Live USB? SysResCD or something.

Nup, she'd probably have to take it to a shop and pay the guy 10.000 dollars just to get up from the chair. Or just buy a new computer. I'm sure the whole leech industry surrounding the rotten Windows world would just love it if we had to compute on their terms, and it seems that's the direction we're headed, Under the mantra "the user is a huge idiot". Us in the fringe world may get burned in the process, but I suspect we'll come out on top of the whole mess, not in the least because Linux users will be among the first to get into the habit of opening a shell in UEFI.