Synchronizing Data and Settings Between Two kubuntu Systems

[Chopstick]

Hi,
I have a laptop and an office PC, both running kubuntu; same version and mostly same software and data. I mostly work on my office PC but sometimes also on my laptop; in addition my laptop is my primary computer for private use.

I would like to be able to synchronize both my data/text/code-files and my configuration settings for a number of programs, and ideally also most of my KDE settings.

For my actual work projects I'm using SVN for backup and synchronization. For settings and other stuff I have been looking into Unison; it mostly does what I want.

The "problem" with all (?) synchronization software is that it all boils down to syncing files and directories, and with the KDE settings and "."-directories I really don't know what I need to sync. I just added the "."-dirs of all affected programs to my Unison-profile, but there seems to be a lot of overhead involved in syncing all these files, most are probably unnecessary. (In particular firefox probably stores a lot of cached stuff in it's dot-dir.)

And KDE-setting are a real headache in this respect, because I don't want to sync everything, the reason being that my office PC has two monitors (and different hardware). Therefore I have different Desktop/Plasma arrangements and I don't want to overwrite these... the keyboard layout is also different! The laptop is German, the PC US; consequently KDE-shortcuts are also slightly different.
I also noticed that a lot of common programs have directories in the .kde/share/apps-folder... do I have to sync these as well?

To summarize, there are two questions:

1. Which programs and strategies would you recommend for synchronization?

2. Which folders and files to I have to ignore, so as to preserve my Desktop/Plasma appearance/settings and hardware-related stuff (including keyboard).

Thanks!
Chopstick

[monte48lowes]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

I have used rsync many times to transfer files.

http://www.mikerubel.org/computers/rsync_snapshots/

The linked website explains how to do backups using rsync, but also provides a lot of good information on how to use rsync. I recommend writing a simple script to complete what you want.

Mike

[Ole Juul]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

That's a great rsync page! I think I will find that useful. However, regarding the OP question, I don't see anything there which is specific to KDE. How is he going to avoid syncing the different files for monitor and keyboard settings? Quote: "I really don't know what I need to sync."

[rb0171610]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Let's talk about the data for a moment. Rather than syncing files across different computers, it would be easier to set up a shared folder on the network where you save all of your files. If you have a home network, you can set up a file server. I have found Samba to be just as effective as NFS. Samba is easier to configure and still allows access from windows computers.
If a third computer as a dedicated server is not an option, consider having the desktop computer running a file server. When working, get in the habit of saving your files primarily onto the desktop computer.
Think of one computer as a master (the desktop) and one as the slave (the laptop). When working at the laptop, mount the shared folder of the desktop into a directory you create in your home folder on the laptop. Get in the habit of saving your files there. When you walk over to the desktop, the files are physically stored on that computer. You can also use ssh to remotely login to the other computer on the network. This is much easier than storing two sets of data on two different computers and trying to always keep them in sync. Then, the only thing you need to do is back up the shared folder on occasion.
It might be easier to use a thumb drive to save the data, keeping your master copy there. This way, you can carry it back and forth between the two computers and not worry about it always being in sync. You only have one master copy! Make backups of the thumb drive on a regular basis into a folder on your desktop computer and/or your laptop. This way, you can restore a file if the thumb drive is lost, destroyed, or corrupted. This scenario also allows you to take your laptop away from home for extended periods of time to work on files independently.

Of course, this would require that you get in the habit of always saving your files into a central directory. For example, on the thumb drive, create a directory called data. Inside this directory, create subdirectories of the data types you are often saving (e.g. documents, backups, photos, spreadsheets, etc.) After saving your files into the appropriate directories, you only need to backup the parent directory, in this case the directory named data using: $ cp -r data/ (followed by the name of the folder you want to copy your backup to).

In a perfect world, you could save files in two places and they will magically become one. In a setup with only two computers and one user, it is easier to just become more organized.

As for your KDE settings, most of the data pertaining to your monitor, keyboard and mouse is not stored in your ~/.kde directory. These devices are most often managed by HAL using fdi configuration text files or they can be managed by settings in the /etc/X11/xorg.conf file.
Keeping two KDE installations synchronized on two computers of two different types is not usually an effective strategy. It makes sense in large enterprises where each computer is identical and consistency is essential. Users that are allowed to customize their desktops usually will be logging onto a centralized domain server where their home directory is stored, allowing them to remotely access their KDE desktop from any computer that they log onto.
My suggestion is to treat each computer as a separate entity. Use different backgrounds if you have to. Make changes to your KDE configuration only on the desktop first and write it down in a notebook. If you like the changes, you can always duplicate the change on your laptop KDE installation and make a checkmark in your notebook.

To backup your KDE directory, first make a backup directory called kde_backups:
$ mkdir ~/kde_backups

Save the following to a text file called backup-kde:

#/bin/bash
echo "Please wait while your KDE Backup is created . . ."
cd ~
cp -r ~/.kde ~/kde_backups && echo "Your KDE backup has been succesfully created!!!"

Execute this script from the command line:

$ sh backup-kde

To restore your backup in case of a problem, save the following text to a file called restore-kde:

#/bin/bash
cd ~
echo "Please wait while your KDE configuration is restored . . ."
cp -r ~/kde_backups/.kde ~ && echo "Your KDE configuration has been successfully restored!!!"

Execute this script from the command line:

$ sh restore-kde

These are just simple scripts that save a copy of your ~/.kde directory into your home directory. It can be customized to be saved onto a file server or thumb drive in case your hard drive crashes.

Hopefully this will give you a different perspective on things.

Rob

[monte48lowes]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

That's a great rsync page! I think I will find that useful. However, regarding the OP question, I don't see anything there which is specific to KDE. How is he going to avoid syncing the different files for monitor and keyboard settings? Quote: "I really don't know what I need to sync."

KDE settings are located in the home folder under .kde, application settings are located under .kde/share/apps configuration settings are located under .kde/share/config. As to which of the files to synchronize or not is up to the OP. It will require checking the files on each computer and excluding certain ones from rsync.

Mike

[dibl]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

1. Which programs and strategies would you recommend for synchronization?

2. Which folders and files to I have to ignore, so as to preserve my Desktop/Plasma appearance/settings and hardware-related stuff (including keyboard).

Experience has taught me that the strategy to deal with this situation begins long before time to do the synchronization. In other words, by the time the question becomes "which files" it is too late to make it simple.

If one puts data on a separate partition, (not mounted as /home, mounted as a unique data directory), and you do that on both/all computers, then data synchronization and backup is a simple matter of synchronizing only the data partitions.

So, if you install your OS in a modest-sized partition that is sufficient to permit such things as downloads of ISO files, you might choose 12GB or 15GB or something like that. Install it all there, including /home.

Your data can then be stored in a single large partition, or more than one if necessary -- depending on how much you have and how often it changes. I very creatively label it DATA. It has directories on it for MUSIC, VIDEOS, IMAGES, and DOCS. After the OS is installed in its partition, you need to make the DATA mount point in /mnt or /media, as you wish, and then edit /etc/fstab to add the partition and mount point. You use symlinks from the DATA folder to your /home/user folder, such that your /DATA/MUSIC appears is in your /home/user/Music folder, etc.

Now the question "how to sync" is obvious and easy -- you sync the DATA partitions only. The risk of messing up KDE (or Gnome or Xfce) settings is zero.

I realize this is not a direct answer to the question that was asked, but I hope it elevates the possibilities for future installations.

[rb0171610]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Unfortunately, I think passive data synchronization between two hosts without a server involved is an awful lot to expect. It would be nice to think one could easily move between different computers in the same house each with their own hard drives and the data could always sync itself, i.e. when you make changes to a file on one hard drive, it automatically changes a file of the same name on the other computer's hard drive.

The problem in this scenario is the idea of two separate hard drives with two sets of data. How does a software that syncs data on two computers know if a new file on the laptop with the same name as an older file on the desktop is actually the same file? Think of it from the standpoint of the software. "Is this the same file? Do I merge the changes in the new file with the old file to make them identical? Do I ask the user first? Decisions, Decisions." I don't the desktop OS is intended to work this way. Each computer has a separate user account using it's own data partitions/hard drives. If you want to have data, user account information, and OS configurations to be the same, you need a server that stores, manages, synchronizes and backs up this information for you in a central location.

In an enterprise network, there is usually a server involved. The user can log into any computer on the network using the same Login and Password. This user actually is logging into a server, not each individual computer. The same is true for when you log into your Yahoo or AOL account on the web. At home, if you log onto a laptop computer and then onto a desktop computer, this is not the same user, even when the accounts have the same username and password. If you are logging into a domain server by trying to login to two different computers, it will not let you. You will be logged off of the other computer, be told you are already logged in somewhere else, or even possibly locked out of your account temporarily for security reasons. In this case, you are actually always logging in as the same user with the same credentials by the virtue of a server. This server is probably not running a desktop Operating System. Servers often, but not always, require expensive hardware, a server OS, and one or more network administrators to maintain. Commercial server OS's can require a license for each user and each device on the network, which can get very expensive even if that server OS is Linux.
In contrast, the home desktop OS is installed separately on each individual computer and each computer has it's own users. So, if you want the benefits of centralized data storage and user account information at home, you will need to set up a server.

Keeping two computers at home in sync is cumbersome and difficult. It is not automatic. Often you have to rely on the rsync command, which syncs data in only one direction and for a good reason. Bi-directional syncronization requires a lot of overhead. I have wasted lots of time and accidentally lost or overwritten files doing this. I would suggest getting in the habit centrally saving all of your data manually to a shared folder, whether this be on a USB stick or a shared folder using NFS or Samba. An important side note: backups, backups, backups!!!!

As for managing two computers, you will just have to remember that they are just that -- two different computers and start treating them as such. Keeping the configurations similar is admirable, but it requires you to be organized and do the work. If you like changes you make on one computer, you will have to make the same changes on the other computer. If you find it worth the effort, it will give you some good hands-on configuration experience. Doing something twice will only making you more experienced, and in the end, more knowledgeable. And hey, be thankful, you have two really great computers!

Rob

[monte48lowes]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

The true setup for that would be a thin client.

Using rsync once or twice to copy over the current configuration shouldn't be a problem. If I understand correctly the OP is only wanting to synchronize the settings from one to the other. If more than that is needed, then a server based approach would be merited.

Mike

[rb0171610]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Thin clients are actually rarely in use any more as this refers to clients with little or no processing power of their own and often times no hard drive. Most enterprise deployments now actually use full-fledge desktop pc's that have a desktop OS installed, graphics cards and hard drives of their own. Once you log into the domain, the processing and most of the operating system relies on the desktop pc you are using. Even an application server will push most of the application over to the client pc. Servers store login information and roaming profiles, authenticate users, and provide access to shared network folders. Rarely if ever are they doing the processing for thousands of clients. Thin client is an antiquated word for terminal. So maybe Fat clients would be a better word. LOL.

I agree that rsync is a great tool to sync files every now and then. Unison is much better as it offers bi-directional syncing capablities. However, I stand by my assessment that users should become accustomed to saving files in a central location, especially when working at two different computers. A memory stick is inexpensive as well as portable.

[Chopstick]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Thanks for all your opinions!

For now I think I will stick with Unison as it seems to be more flexible than just rsync (internally Unison uses rsync anyway AFAIK).

As for using a single location to save everything. A shared drive is IMO not practical, for two reasons: I often have to work offline with my laptop (when I'm traveling), and my two computers are usually spatially separated, so that access to a shared drive may be slow.
Using some portable media may be possible, but then I would have to save almost everything there! As I said, I would like to keep certain KDE-apps in sync, too. And I would have to carry that around all the time... (I'm using an external HD for very large data sets though, also for backup).

As for what is KDE-specific: What is the difference between dot-folders in the KDE-folder and the home-folder? Some apps seem to have both... and why are not all apps in the home or the KDE folder?

Also, does anyone know where the Plasma-Dashboard-Configuration resides? Plasma widgets and so forth? Then I could configure Unison to exclude these config-files.

I also looked into Kolab to synchronize my Kontact-Stuff (Calendar, Contacts, Tasks etc.) on my office PC with Thunderbird on my laptop. But it does not seem to work well with Thunderbird... The Calendar is just empty. - Maybe I should just use Kontact on both ;-)

Chopstick

[wizard10000]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Unfortunately, I think passive data synchronization between two hosts without a server involved is an awful lot to expect. It would be nice to think one could easily move between different computers in the same house each with their own hard drives and the data could always sync itself, i.e. when you make changes to a file on one hard drive, it automatically changes a file of the same name on the other computer's hard drive...

That's exactly what I do with rsync, Rob. To answer one of your other observations the way rsync knows whether two files with the same name and datestamp are the same is by building a checksum (actually several checksums) of the files and comparing them - if the checksums match the files are the same.

Tooting my own horn but I wrote a howto here on how to do exactly what you mention.

[monte48lowes]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Also, does anyone know where the Plasma-Dashboard-Configuration resides? Plasma widgets and so forth?

These are located:

~/.kde/share/config/plasma*

Mike

[Chopstick]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

Thanks Mike! That should help.

Tooting my own horn but I wrote a howto here on how to do exactly what you mention.

This should be the link (for your convenience...):

http://kubuntuforums.net/forums/inde...opic=3106769.0

[Ole Juul]

Re: Synchronizing Data and Settings Between Two kubuntu Systems

For my own education and possible future use, I'll have a good read of wizard10000's howto, but I like the simplicity of rb0171610's general ideas. One thing to watch out for though, if you do cp -r then you lose a lot of data. I recently did this without thinking and my heart sank when I realized that all date information was lost! Depending on the type of data, this could be a huge loss. IMHO, to avoid problems I think that in most cases the -p switch needs to be included.