[SOLVED] RAID 0 Install?

[MadMage999]

Hi everyone.

I am wondering why kubuntu (currently 18.04.1) does not support installing on software raid.

I tried installing mdadm first and manually creating a raid but the kubuntu installer crashed installing to it (md0). Kudos that it at least recognized the raid and gave the option to install (I guess?).

I also installed ubuntu server and then added kubuntu-desktop to it, which is not quite satisfactory either, as things like the network manager applet don;t work (don't see the interfaces at all).

So, what I'm wanting to know is if it's planned for the future. It seems to me raid is pretty mainstream, even for desktop users in late 2018. Ubuntu server installer seems to support it pretty well, but I can't imagine why kubuntu shouldn't have support for installing to a software raid by either using a small portion of the first drive as /boot or another drive altogether (outside the array) for /boot.

Thanks

[oshunluvr]

While I agree with your sentiment - that installing to RAID should be a basic function, the problem isn't with "Kubuntu" as a whole but rather the installer - Ubiquity. IMO it has a several problems. If you want resolution, I suggest doing some basic troubleshooting and documenting the issue with error output/log files and reporting a bug.

Having said that, I don't think mdadm RAID is very popular any longer - at least not with the users here. I cannot remember the last time someone posted about mdadm RAID here - it's been many years. I can only guess it's due to several things like hard drive capacity and SSD speeds obviating the desire for RAID, as well as motherboards supporting RAID natively (something I advise against using). I myself used RAID extensively from 2004 to about 2012. During that time, SSDs became affordable and the BTRFS file system emerged. I began using BTRFS and my need for the complexity of layering drive formats (EXT3/4 on top of MDADM) went away. If you were interested in a more modern solution to your quest for RAID0, you might research BTRFS a bit.

Regardless, good luck with your issue. I did read a post from a developer that switching from Ubiquity to Calamares is very near, so that alone might solve the problem.

[MadMage999]

Thanks. I hope the new installer makes life easier for everyone. I guess we'll see.

I have some limited experience using btrfs for raid under unraid server OS. I will look into it some more, but I wonder if anyone had heard about the suitability of btrfs for storing virtual machine images?

I understand why many are not as excited about raid as they once were, and it makes sense that things like SSDs and especially NVME would contribute to that. Personally I am still striping 4 NVME SSDs in RAID 0. Never too fast.

Sadly, I have installed fedora now and it set up the desired raid with not a hiccup. I love kubuntu, but it just won't work for me without these capabilities.

I hope I can continue using kubuntu when/if the new installer addresses this, and thanks for your nicely worded reply.

[oshunluvr]

To my knowledge, it's still verboten to store dynamically sized images on btrfs. This means swapfiles and dynamically sized virtual disks are subject to corruption if stored on a btrfs file system with COW enabled. You can create a btrfs file system and mount it with COW disabled, but this defeats much of the benefit of using btrfs. I just use a swap partition and have a separate partition formatted to EXT4 for my VM machines. It seemed to be the simplest and safest solution.

Wow - 4 nvme devices in RAID0 ? Talk about a speed-demon!

I'm curious how much real world difference there is considering the exponential increase in speed over SATA already baked in when using nvme. @here, my SSDs are about 4x times faster than my HDDs and my Samsung 970 PRO 512GB nvme drive is 6x faster than the SSDs (also fast - Samsung 256GB 840 PROs). My start times are down to a few seconds but other than file copying, the differences are pretty subtle or at least case specific. Of course, I'm never going backward! I added the nvme drive a couple months ago, moved my installs (multi-booter of Linux here) from the 840 PRO SSDs (in RAID0) to the solo nvme and got about a 3x times speed increase in transfer rate. I'm not trying to talk you into backing off - I'm a hardware guy too and speed never goes to waste.

Technically, BTRFS in RAID mode is not as fast as EXT4 correctly tuned using MDADM but the differences are small percentage points. The primary benefits to BTRFS are a more simple usage, flexibility on-the-fly (add and subtract devices while still using the system), snapshot and backup capabilities built-in to the filesystem, free space distribution, etc. It's not intended as a strictly performance fs. In the old days on Linux (90's) I spread various parts of my OS across different drives on different IDE channels (/var here, /usr there, etc.) to gain a small speed boost. Then came MDADM and a 4 hard drive RAID and I was off to the races! Now I keep it simpler.

[MadMage999]

Well, that's good news for me. I don't use dynamically sized vm disk images. I assume using statically sized averts the problem, but I'll read on it some more.

I did as you suggested and checked into btrfs a little more. Seems you can install kubuntu to a single drive with btrfs and then add the other array drives and setup striping. Wow. I see what the hype is around btrfs now.

You really straightened me out, thanks. I think I've been going about this all wrong. If it's simple as installing to a single btrfs and then adding drives and balancing them for striping, then I really don't need what I though I did. So far so good.

[oshunluvr]

You welcome, and BTW - Welcome to KFN (and hopefully, Kubuntu).

BTRFS is super easy to use. I even wrote a Dolphin submenu to help manage subvolumes without using the CLI, but I doubt that's an issue for you. With the added features of BTRFS, once I discovered it, I never went back to MDADM.

Not only can you create or destroy RAID setups on the fly (while still using the file system), but you can add and remove devices and re-stripe to another version anytime you want. The only caveats I'm aware of: There was a "write hole" when using RAID5/6 but I believe it's been patched, and last I checked when using RAID0 if a device fails, you can only mount it in degraded mode once (ostensibly to replace the failed drive). After that the RAID goes read-only and you have to move the data off of it. There was some reason why this was initially designed this way but I think the devs. are working on changing this also. The good news is with subvolumes and the ease of moving them around, rebuilding from a read-only to a new read-write wouldn't be difficult, just time consuming. I've never actually experienced a total failure of of "live" device as I monitor my SMART reporting and replace drives when they start to show age - before they fail.

The real benefits to BTRFS are the subvolumes. Data is segregated like partitions, but can exist on the same partition together. They're mounted like partitions would be except that since they all reside on the same file system they share the free space - effectively expanding or contracting as needed. Any subvolume can be snapshot-ed or sent from one btrfs file system to another as a backup method. It's really slick once you get into it. @here, I have one 500GB btrfs file system that has 8 Linux distros installed to it and I even have a stand-alone GRUB install that also resides in a subvolume. I have written extensively on this forum about BTRFS so much we opened a BTRFS subforum. Lots of good questions and info in there.

It also occurred to me that you may be able to boot to the Kubuntu USB live session, install Calamares (I believe it's in the repos, but I'm not on Kubuntu 18.04 at the mo'), and see if it works better for your usage. It'd be worth a try if you'd rather stick with what you know.

[GreyGeek]

It would be better when using VirtualBox to create static virtual HDs, not dynamic ones. When running PostgreSQL and MySQL it would be better to establish the database on a secondary HD using EXT4 or via https to a database server. But, that’s just my opinion

[MadMage999]

"Welcome to KFN (and hopefully, Kubuntu)."

Yeah, I think kubuntu is where I'm settling down after trying a lot of different distros.

"The real benefits to BTRFS are the subvolumes. Data is segregated like partitions, but can exist on the same partition together. They're mounted like partitions would be except that since they all reside on the same file system they share the free space - effectively expanding or contracting as needed."...

That IS slick.

"I have one 500GB btrfs file system that has 8 Linux distros installed to it and I even have a stand-alone GRUB install that also resides in a subvolume."

NICE! I guess that's a multiboot situation?

"It also occurred to me that you may be able to boot to the Kubuntu USB live session, install Calamares (I believe it's in the repos, but I'm not on Kubuntu 18.04 at the mo'), and see if it works better for your usage. It'd be worth a try if you'd rather stick with what you know."

Thanks, as long as there are no issues with this btrfs setup, I'll probably stick with that, though I am curious to see the new installer.

[MadMage999]

It would be better when using VirtualBox to create static virtual HDs, not dynamic ones. When running PostgreSQL and MySQL it would be better to establish the database on a secondary HD using EXT4 or via https to a database server. But, that’s just my opinion

Hmmm, I need to start reading on this. LOL, I've used btrfs, but I had no idea. I hadn't considered mysql yet, tho it seems inevitable. Thanks for the heads up.

[GreyGeek]

I'd take PostgreSQL over mysql any day, but that's just me. One thing I will NEVER do again is run a distro that does not allow the installation of Btrfs as the root file system using the installer.

[MadMage999]

I'd take PostgreSQL over mysql any day, but that's just me. One thing I will NEVER do again is run a distro that does not allow the installation of Btrfs as the root file system using the installer.

Mind if I ask why you prefer PostgreSQL?

[GreyGeek]

Keep in mind that I retired ten years ago, and haven't kept up with developments in MySQL or PostgreSQL.

My last job was writing GUI front ends to Oracle databases using Qt. Those databases were for analytics and data warehousing for our State Revenue Dept. Back then PostgreSQL was 90-95% compatible with Oracle. When I looked at MySQL a lot of ATOM functionality were merely code stubs with no functions behind them, just to avoid incompatibility crashes, so I went with PostgreSQL. I wrote C++ source using Kate on my SuSE desktop against a PostgreSQL running on a SuSE server in my office. I used compiler defines to switch between PostgreSQL and Oracle code, based on the environment during the compile. I could code, compile and test 3-5X faster on my SuSE box using Kate and Kdbg than I could running VC++ 6.0 on Windows, but by 2008 our conversion to Linux was derailed by a new Tax Commissioner and we became a Windows shop, even though I continued to use SuSE as my primary workstation. The Windows executable was put on the LAN server and the Windows workstations would access it.

#ifdef _WIN32  // on Windows
#define DBDRIVER "QOCI"
#define DBHOST "orc1"
#define DBNAME "orc1"
#else // on Linux
#define DBDRIVER "QPSQL"
#define DBHOST "localhost"
#define DBNAME "homestead"
#endif

and that allows processing SQL queries like this on either backend:

    sQuestion.append("Do you want to generate an All Counties\n");
    sQuestion.append("cross tab report for Homestead?\n");
    int reply = QMessageBox::question(this, tr("QMessageBox::showQuestion()"),
        sQuestion, QMessageBox::Yes, QMessageBox::No);
    if (reply == QMessageBox::Yes){
        ui.leStatus->setText("Preparing XTab Report...");
        this->setCursor(Qt::WaitCursor);
        qApp->processEvents();
        QString queryStr = "SELECT cntyname, M1,M2,M3,M4,S1,S2,S3,S4,VA,";
        queryStr.append("(SUM(M1)+SUM(M2)+SUM(M3)+SUM(M4)+SUM(S1)+SUM(S2)+SUM(S3)+SUM(S4)+SUM(VA)) AS Totals ");
        queryStr.append("FROM(SELECT cntyname, ");
        queryStr.append("SUM(CASE WHEN offuse = 'M-1' THEN 1 ELSE 0 END)AS M1,");
        queryStr.append("SUM(CASE WHEN offuse = 'M-2' THEN 1 ELSE 0 END)AS M2,");
        queryStr.append("SUM(CASE WHEN offuse = 'M-3' THEN 1 ELSE 0 END)AS M3,");
        queryStr.append("SUM(CASE WHEN offuse = 'M-4' THEN 1 ELSE 0 END)AS M4,");
        queryStr.append("SUM(CASE WHEN offuse = 'S-1' THEN 1 ELSE 0 END)AS S1,");
        queryStr.append("SUM(CASE WHEN offuse = 'S-2' THEN 1 ELSE 0 END)AS S2,");
        queryStr.append("SUM(CASE WHEN offuse = 'S-3' THEN 1 ELSE 0 END)AS S3,");
        queryStr.append("SUM(CASE WHEN offuse = 'S-4' THEN 1 ELSE 0 END)AS S4,");
        queryStr.append("SUM(CASE WHEN offuse = 'VA' THEN 1 ELSE 0 END)AS VA ");
        queryStr.append("FROM ");
        queryStr.append(this->propertydb);
        queryStr.append(" GROUP BY cntyname ORDER BY cntyname)");
        queryStr.append("GROUP BY cntyname,M1,M2,M3,M4,S1,S2,S3,S4,VA ORDER BY cntyname");
        QSqlQueryModel *XtabModel = new QSqlQueryModel;
        XtabModel->setQuery(queryStr);
        while ( XtabModel->canFetchMore() )
            XtabModel->fetchMore();
        if (XtabModel->rowCount() > 0){
            ui.leStatus->setText("Processing XTab rows ...");
...

I suspect that after ten years both tools have improved considerably and the differences between them are probably negligable, but PostgreSQL is what I chose and what I know, and although SQL is a standard query language the encoding of it is not standard. I also love Qt as an API and C++ as a language. Ten years later they are still the supreme tool set. Instead of LAMP I'd go with LAPP.