Use of parallelism in KDE applications

[Ofnuts]

Is it possible make some KDE apps use parallelism? In particular K3B (when ripping CDs) and Gwenview (producing contact sheets)? I don't like to wait while my processor barely hits 12% (4 cores + hyperthreading).

[Snowhog]

Linux already does this. However, prioritization of running processes is more likely to be what you are finding 'irritating'. Linux is by design, a multi-tasking OS; it doesn't wait for one process to finish before starting another.

[Ofnuts]

I know... But if the process has only one thread of execution, it will not use more than one core, leaving 7 cores idle.

[woodsmoke]

Now...THIS is a very interesting thread.

woodsmoke

[Ofnuts]

So for instance, let's take K3B. It takes 10 minutes tp re-encode a CD. It does it track by track. Only of CPU is really working. Given the amount of data read and written, it is not I/O bound. If it spawned threads or subprocesses to encode 8 tracks in parallel, the system would dispatch them to the various CPUs, all CPUs would be busy and it would take pretty close to one eighth of the time to rip the CD (OK, I could be optimistic here since hyperthreading isn't exactly a distinct CPU, but I would still expect well over a 4x improvement).

[woodsmoke]

Here is a comment from a feb 2017 thread at Ask Ubuntu

A program must be made to use multiple cores. The programmer needs to write a instruction for each core.

If the program you are running don't use it, it never will without a update/upgrade.

What can I do to speed up my program?

You can disable some cores and overclock the one that is left. But first ask yourself if the program needs that speed. Or rewrite the program if possible.

However the above is just ONE "expert"

Here is another one which, after some initial discussion describes using an application called "taskset" on an Ubuntu machine.

http://www.hecticgeek.com/2012/03/as...-ubuntu-linux/

again, interesting thread.

woodsmoke

[GreyGeek]

One can use Ksysguard to change the niceness (priority) of an application. Right mouse in the app.

[Ofnuts]

Here is a comment from a feb 2017 thread at Ask Ubuntu

A program must be made to use multiple cores.

Exactly. And my question is, are K3B and Gwenview written for this? Many application may support this with some obscure and barely documented setting (for instance, Gimp will check GEGL_THREADS in the environment).

taskset doesn't make a process run on several cores in parallel. It only ensure that a given process will be dispatched on specific cores.

If the application is written to use only one flow of execution (and therefore run on a single CPU core at a time), which is what happens by default, nothing is going to make it run on several cores. Increasing its priority is only useful when it competes against other processes for CPU.

[woodsmoke]

quite beyond my ken.

woodlikesbarbiesmoke

[wizard10000]

One thing I haven't heard mentioned is that the default CPU governor in almost all Linux kernels is the ondemand governor, which will not scale up processor frequency until CPU utilization hits 95%. Machines using the intel_pstate driver (third-generation Intel Core and later) default to the powersave governor.

Since IMO processor frequency has almost zero impact on power consumption both my laptop and home server are locked at highest frequency by using the performance governor instead of the ondemand (or in the case of my laptop, powersave) governor.

Excellent read here - https://wiki.archlinux.org/index.php...quency_scaling

You can't make a single-threaded application use more threads but you can increase CPU frequency, which will make all those threads run a little quicker.

[GreyGeek]

Exactly. And my question is, are K3B and Gwenview written for this? Many application may support this with some obscure and barely documented setting (for instance, Gimp will check GEGL_THREADS in the environment).

K3B is written using threads but the developer said it was difficult to do because he needed the threads to talk to each other. I don't know about Gwenview, although I doubt it. Not very many apps are designed to use threading or multiple processes. That's partly because most applications cannot be broken down into parallel processes that can be threaded simultaneously while allowing the app to continue accepting input from the user for processing. Take indexing, for example. A developer might be tempted to put it on a thread and let the user continue to use the app. It may improve the indexing speed but only testing will determine by how much and if the increase is worth it. But, if the index is then used before the indexing is complete, the results will not be inclusive. That's why apps throw up a modal window and do not allow continued operation until the indexing is complete. Certain classes of problems are a fit for threading and multiple processes but most are not. Programs suitable for multiple threads or processes are usually run on super computers with thousands of cores, or more, to give each thread or process one or more cores. Such programs use matrices or Taylor or McLauren Expansion series with lots of terms to create CGI output.

KDE and most of its apps are written using the Qt API. The documentation discusses threading here: http://doc.qt.io/qt-5/thread-basics.html
Qt has "Signals and Slots" technology to simplify call-backs and communication between threads and between a thread and its parent. S&S simplifies a lot of things but there are still a lot of things that can go awry and threading apps are very difficult to debug.

taskset doesn't make a process run on several cores in parallel. It only ensure that a given process will be dispatched on specific cores.

True, but running on more than one core may give better results than running on only one, but it depends on how the app was designed.

If the application is written to use only one flow of execution (and therefore run on a single CPU core at a time), which is what happens by default, nothing is going to make it run on several cores. Increasing its priority is only useful when it competes against other processes for CPU.

NO app or thread runs on a core alone. If you list the processes that are running in your system you will see that EVERY app/process is competing against THOUSANDS other apps/processes on all your cores. Changing the priority for an app will give it an advantage over apps running with a niceness of 0, or average priority. Niceness varies from -20 to +20 (High to Low Priority). Giving an app a higher priority via Ksysguard lowers its niceness. You can also use the nice command in a Konsole to do the same thing.

[Ofnuts]

K3B is written using threads but the developer said it was difficult to do because he needed the threads to talk to each other. I don't know about Gwenview, although I doubt it. Not very many apps are designed to use threading or multiple processes. That's partly because most applications cannot be broken down into parallel processes that can be threaded simultaneously while allowing the app to continue accepting input from the user for processing. [...] Certain classes of problems are a fit for threading and multiple processes but most are not.

I would put MP3-encoding a file in that category. It's complicated if you try to use several threads to encode parts of the same file, but it's easy if you are encoding several files in parallel since there is no need for the threads to communicate between themselves since they don't work on the same data.

Programs suitable for multiple threads or processes are usually run on super computers with thousands of cores, or more

Much closer to home, your garden-variety Java-based web server is multi-threaded these days.

True, but running on more than one core may give better results than running on only one, but it depends on how the app was designed.

Let rephrase that more explicitly: True, but running on more than one core at a time may give better results than running on only one at a time, but it depends on how the app was designed.

[sithlord48]

k3b is just using something like lame behind the scenes . if you want to encode N files just start N encodes using the encoder directly.

[wizard10000]

soundkonverter is a multithreaded audio conversion application built for KDE that can also apparently rip CDs (didn't know it could do CD ripping or I'd have mentioned it earlier).

I use it to convert flac to mp3 since my car doesn't do flac - and I've had it running as many as eight threads at a time.