[ENCRYPTION] OverlayFS discoveries... ;-]

[xennex81]

I had turned a truecrypt container mounted in my home directory into the upper dir of a overlay into a non(extra)-encrypted mount point where a different volume (LVM) was also mounted.

But to my surprise I discovered that the upper dir is used for every kind of changes to the filesystem. If problems arise with the upper dir mount point, strange things may occur. I believe at one point my truecrypt was dismounted, but the overlayfs kept writing change to the now 'empty' mountpoint in my home dir.

I had suspected or believed that , well, quite silly perhaps, but let us not judge me now . I had believed that the root-level directories (for that mountpoint) that came from the upper dir would be used for writing changes back to the upper dir, and root-level directories that came from the lower dir would be used for writing changes back to the lowerdir. I was a bit mistaken.

There was also in the upper dir (the truecrypt mount) a symlink that had the same name as the directory that it was about in the lower dir. So changes written supposedly back to the upper dir (the truecrypt container) followed the symlink on that truecrypt container to find a level deeper into the truecrypt container. But, because the truecrypt container had since been dismounted, but the updates had not made it through to the overlayfs, it was still following that symlink and writing changes into the now empty (or, soon to be not empty anymore) mount point of the truecrypt in my home folder (that was no longer mounted in truecrypt, but was still overlayed with the different volume).

Funky **** man!!

In my chaos I then deleted the new directory structure in my home dir mount point, and later undeleted it again and it is now sitting in my /root/ folder ;-).

I am seriously trying to get back to the exact point where I started deviating...

And I am seriously thinking this is calling for subversion to be put to use .

Since we are also talking (or I am) about programming code now (java files) (and one javacc file).

My mind was filling up with even more chaos due to all the changes and I didn't know what was what or where anymore . Then someone where I reside currently broke my concentration to such an extent that I lost all hope of recovering the where's what and how's how. So I even started deleting all kinds of backups of older files just to make sure only a single version remains (in my mind).

Bah.

[xennex81]

I think I am going to keep the truecrypt as upperdir now and any changes to the lowerdir structure that 'accidentily' get written into the upperdir I will just sync back to the lowerdir every week when the upperdir is dismounted (I have mounted the lowerdir onto the lowerdir). The alternative is to have the lowerdir mounted elsewhere as well but I really wanted a clean look on my filesystem....

It is possible also to hide the "real" mount point of the lowerdir (which is then the lowerdir itself) in the overlayed dir by creating a symlink in the lowerdir that points to ., but which overwrites (hides) the actual mount directory.

So:

/merged
/merged/real
/home/truecrypt

You mount the 'real' volume of the lowerdir in /merged/real

you create a symlink /merged/real/real -> .

Now mount the overlayfs with lowerdir /merged/real, upperdir /home/truecrypt, onto /merged

/merged/real is now hidden by the symlink to .

But if you now delete the symlink (which is just very much possible) the original /merged/real does not (immediately) resurface.

However if you dismount the overlayfs immediately the /merged/real directory (with the complete contents of the real volume of lowerdir) will be back. MOREOVER the REMOVAL OF THE SYMLINK is now only registered in /home/truecrypt as "overlay-whiteout" which means....

My god.

The .... madness.

Now the actual directory of the real mountpoint of your lowerdir volume (/merged/real) (that is to say, the symlink that you have placed there to hide that directory) is hidden by the /home/truecrypt mount.

If you then remove the whiteout (from the real truecrypt dir) the filesystem comes into disarray.

I believe that is what happened to me before .

Now it says:

l ? ? ? ? ? real

A remount of the overlay will reveal the deleted (hidden) symlink that hides the mountpoint of lowerdir.

LOL OL OL.

But the real dir is hidden regardless of anything else anyway . If you mount something else on top of it hahahahahaha.

So the only way to get access to the lowerdir is to not mount it on top of itself anywhere. If you want to write changes back to it.

You can however mount -o bind the lowerdir somewhere else, before you .... etc., but this is the same thing anyway.

----====----

The issue I have is that in e.g. Dolphin, every physical volume that gets mounted gets shown. It seems hardly possible to hide the "real" mounts for longer than a session, and to show different mounts as volumes.

I may be mistaken. I just added two of those locations as "places" in Dolphin. Previously a hidden item came back straight away on next boot.

I just want a clean user interface just in case people get into the laptop and start messing around with stuff .

It is also cleaner to your own mind if things work elegantly.

I guess this whole Kubuntu thing is a big experiment for me .

.

"Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed. If the underlying filesystem is changed, the behavior of the overlay is undefined, though it will not result in a crash or deadlock."

Okay. That explains.

I was thinking the overlay would provide a nice, combined presentation. But this gives too many headaches.

Furthermore, it seems no mechanism is provided to merge these filesystems with one another in case of changes.

You might change a file in lowerdir and the new file ends up in upperdir, and while the overlayfs is mounted, it will work fine.

The moment you dismount, the old file will resurface in lowerdir. Copying upperdir on top of lowerdir will work, but then you need to know which are the new files.

Care must also be taken.... such a headache this!! :P .

Writing changes to upperdir (while its own fs is mounted) with a script that can distinguish between "wanted" and "unwanted" directories you may simply move "unwanted" files (that do not exist normally in upperdir fs) back to lowerdir (overwriting them) while unmounted as overlay...

An alternative is that you just provide a score of mounts, one for each directory in the top level of one filesystem, as well as one for each directory in the top level of another.... this is getting too crazy . But in that case you would end up with

/merged/A, B, C, pointing to (mount -o bind) actual dirs in some /real/A, B, C
/merged/D, E, F pointing to (mount -o bind) actual dirs in some /second/D, E, F.

any new directories get created in the root fs. Alternative is to bind D, E, F in a regular single mount of "/real".

[xennex81]

Don't even know what I wrote here exactly. Of course, this... ;-).

...
Sometimes being busy here or wasting time or people's attention (??) here just seems like the most adequate way to get ahead in my thinking and my trying with this linux system.

But I don't think anyone really objects if I use this place as a bit of a private thinking forum?

As for me, that's fine. You certainly wouldn't be the only one...
It's like sometimes I feel like I am divulging personal stuff or information that is not really relevant to other people but only to my own progress. And perhaps the stuff I write is too detailed...

[jlittle]

...
Sometimes being busy here or wasting time or people's attention (??) here just seems like the most adequate way to get ahead in my thinking and my trying with this linux system.

But I don't think anyone really objects if I use this place as a bit of a private thinking forum?

As for me, that's fine. You certainly wouldn't be the only one...

It's like sometimes I feel like I am divulging personal stuff or information that is not really relevant to other people but only to my own progress. And perhaps the stuff I write is too detailed...

Dropping the odd link, even to a wikipedia article, can help the general audience.

[xennex81]

Lol, ... deleting everything I write because I think it is improper or not good enough of humanity ;-).

I have deleted SO MANY THINGS I have created in recent times. All depression I guess.

Not thinking what you do has any worth.

Anyway.

[xennex81]

In any case, I ran into the OverlayFS madness again. I had forgotten about it. Suddenly it starts copying stuff I have downloaded into the safe which is too small for it.

It is easy enough to copy stuff back out of the safe into the store, but you first have to dismount the overlay for that.

Which can be tricky, how do you exactly dismount something that is the combination of two mount points, and both mount points are also mounted? How do you reference the in-between?.....

Well, usually what I or you would do is to dismount everything that comes close and eventually you'll get to the mounts themselves (or the overlay fs which is what you really want. Perhaps you or I could specify the overlay fs as the type to be dismounted, that might work, I have never needed to do that before.

Now because of a type I have lost 6 GB of downloaded data, it is easy enough to redownload it, but I do not have time for such things at ALL. BAH. STUPID ME.

Yes I could dismount the truecrypt.. .erm no I can't. There are no Linux tools as far as I can... actually I could. Undelete everything. Is it worth the trouble? I guess I just redownloaded most already...