Performance degrades over time; have to reboot every few days

[GreyGeek]

Re: Performance degrades over time; have to reboot every few days

If your system appears to be slowing down then it must also mean that
something is using more and more kernel "wakeups". Install "powertop",
open a konsole and run

sudo powertop

Let it run and watch it as you open and close various apps. It will show
you what is making all the interrupt calls and how many per second they
are making.

My 64bit Kubuntu Karmic with nothing but a konsole running powertop
shows the total number of wakeup calls to average around 130 per second.
When I am browsing with FireFox and have KCal and Thunderbird open my
average is about 450/sec.

A word of caution. Powertop gives "advice" about what to do to reduce or
eliminate the interrupt calls. BEFORE you take any of that advice talk with
folks here about what you are thinking of doing. Many interrupts are part
of the services that Kubuntu uses.. Services like the device notifier, or the
internal usb optical drive, etc... Disable those and you may kill some of the
"automatic" features that you like in Kubuntu.

[Azurescens]

Re: Performance degrades over time; have to reboot every few days

Its still running fine for now but the Xorg process is up to 207 M of memory
from the 110 it was using when I posted yesterday. Am I wrong to keep pointing
at the Xorg process? I don't know that it's part of the problem but it's the only
thing I've noticed so far that looks like it could be related.

I tried powertop but I lack the Linux experience to really make much of it.
With the system sitting idle and nothing open but the terminal window I get:

Wakeups-from-idle per second : 50.7   interval: 5.0s
no ACPI power usage estimate available

Top causes for wakeups:
 62.6% ( 62.6)    <interrupt> : nvidia
 15.8% ( 15.8)    <interrupt> : pata_amd
  8.8% ( 8.8)   <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  2.8% ( 2.8)     knotify4 : hrtimer_start_range_ns (hrtimer_wakeup)
  2.2% ( 2.2)   <kernel core> : hrtimer_start (tick_sched_timer)
  2.0% ( 2.0)   <kernel IPI> : Rescheduling interrupts
  1.0% ( 1.0)   <kernel core> : nv_kern_rc_timer (nv_kern_rc_timer)
  0.8% ( 0.8)  hald-addon-stor : hrtimer_start_range_ns (hrtimer_wakeup)
  0.6% ( 0.6)   <kernel core> : neigh_periodic_timer (neigh_periodic_timer)
  0.4% ( 0.4)      konsole : hrtimer_start_range_ns (hrtimer_wakeup)
  0.4% ( 0.4)       kwin : hrtimer_start_range_ns (hrtimer_wakeup)
  0.4% ( 0.4)  hald-addon-stor : blk_add_timer (blk_rq_timed_out_timer)
  0.4% ( 0.4)  plasma-desktop : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.2)    <interrupt> : ohci_hcd:usb2
  0.2% ( 0.2)       Xorg : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.2)      pdflush : wb_kupdate (wb_timer_fn)
  0.2% ( 0.2)   <kernel core> : dev_watchdog (dev_watchdog)
  0.2% ( 0.2)     ssh-agent : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.2)      krunner : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.2)     klauncher : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.2)     gpg-agent : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.2)  USB device 2-3.1 : Microsoft Natural Keyboard Pro ()

Then I checked it under what for me would be typical usage. I had open Evolution,
Akregator, Pidgin, Opera and Firefox with several tabs one of which was playing a
Youtube video. This resulted in:

Wakeups-from-idle per second : 518.7  interval: 10.0s
no ACPI power usage estimate available

Top causes for wakeups:
 17.3% (198.5)    <interrupt> : eth0, snd_ca0106
 17.3% (198.0)   <kernel IPI> : Rescheduling interrupts
 15.0% (171.8)   npviewer.bin : hrtimer_start_range_ns (hrtimer_wakeup)
  9.8% (111.9)  USB device 2-4 : USB-PS/2 Optical Mouse (Logitech)
  9.2% (105.6)   <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  8.9% (102.6)    <interrupt> : ohci_hcd:usb2
  8.5% ( 97.5)      firefox : hrtimer_start_range_ns (hrtimer_wakeup)
  5.5% ( 62.7)    <interrupt> : nvidia
  4.8% ( 54.6)       kwin : hrtimer_start_range_ns (hrtimer_wakeup)
  1.4% ( 15.9)    <interrupt> : pata_amd
  1.2% ( 13.7)       opera : hrtimer_start_range_ns (hrtimer_wakeup)
  0.3% ( 3.0)     knotify4 : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 2.6)      krunner : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 2.0)   <kernel core> : sk_reset_timer (tcp_delack_timer)
  0.1% ( 1.3)  plasma-desktop : hrtimer_start_range_ns (hrtimer_wakeup)
  0.1% ( 1.0)   <kernel core> : nv_kern_rc_timer (nv_kern_rc_timer)
  0.1% ( 0.8)  hald-addon-stor : hrtimer_start_range_ns (hrtimer_wakeup)
  0.1% ( 0.7)   <kernel core> : neigh_periodic_timer (neigh_periodic_timer)
  0.0% ( 0.3)  hald-addon-stor : blk_plug_device (blk_unplug_timeout)
  0.0% ( 0.3)      pidgin : hrtimer_start_range_ns (hrtimer_wakeup)
  0.0% ( 0.2)   <kernel core> : dev_watchdog (dev_watchdog)
  0.0% ( 0.2)   <kernel core> : inet_twdr_hangman (inet_twdr_hangman)
  0.0% ( 0.2)      pdflush : wb_kupdate (wb_timer_fn)

As I mentioned the system was just rebooted yesterday and is still performing fine.
I'll check again in a few days once it starts to slow down. What sort of things should
I be looking for that would indicate a problem?

[GreyGeek]

Re: Performance degrades over time; have to reboot every few days

50 wakeups for an idle system is better than average.
500 wakeups for a busy system is normal.
In a few days, if your system appears to have slowed down, it will be interesting to see what those values are.

Meanwhile, another reason why systems slow down is that memory is used by an app and then not returned to the heap when that app quites. That unreleased memory is not available for other apps so they have less available memory to use. This is the same is running with less RAM. If one or more apps continue to eat memory when they are used, eventually it would be like running on a PC with only 256MB or less. The kernel has to do a lot pageswapping with swap or the hd. Use the "free" command in a terminal to keep track of your memory usage (you can cut&paste results into a document in order to track usage). Or, you can open the System Monitor, switch to the page which shows the RAM usage, then start and stop an app and see if the RAM available returns to the previous level (used RAM returned to the heap) or not.

[Azurescens]

Re: Performance degrades over time; have to reboot every few days

Ok, I've had to reboot a few times for other reasons so it's taken a while to get it back to the point where its having some significant slow down. I ran power top again and am seeing much more wakeups when idle.
Idle:

Wakeups-from-idle per second : 259.6  interval: 10.0s
no ACPI power usage estimate available

Top causes for wakeups:
 69.4% (359.1)   <kernel IPI> : Function call interrupts
 11.9% ( 61.5)    <interrupt> : nvidia
 10.6% ( 54.8)   <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
  4.1% ( 21.1)   <kernel IPI> : Rescheduling interrupts
  1.2% ( 6.4)    <interrupt> : pata_amd
  0.6% ( 3.0)     knotify4 : hrtimer_start_range_ns (hrtimer_wakeup)
  0.4% ( 2.1)      krunner : hrtimer_start_range_ns (hrtimer_wakeup)
  0.3% ( 1.4)   <kernel core> : hrtimer_start (tick_sched_timer)
  0.3% ( 1.3)       kwin : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 1.0)  plasma-desktop : nv_kern_rc_timer (nv_kern_rc_timer)
  0.2% ( 1.0)  plasma-desktop : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.9)        adb : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 0.8)  hald-addon-stor : hrtimer_start_range_ns (hrtimer_wakeup)
  0.1% ( 0.7)   <kernel core> : neigh_periodic_timer (neigh_periodic_timer)
  0.1% ( 0.5)   ksuspend_usbd : queue_delayed_work (delayed_work_timer_fn)
  0.0% ( 0.2)     kwalletd : hrtimer_start_range_ns (hrtimer_wakeup)
  0.0% ( 0.2)       Xorg : dev_watchdog (dev_watchdog)
  0.0% ( 0.2)      pdflush : wb_kupdate (wb_timer_fn)
  0.0% ( 0.2)  hald-addon-stor : blk_add_timer (blk_rq_timed_out_timer)
  0.0% ( 0.2)     gpg-agent : hrtimer_start_range_ns (hrtimer_wakeup)
  0.0% ( 0.2)       kded4 : hrtimer_start_range_ns (hrtimer_wakeup)
  0.0% ( 0.1)     ssh-agent : hrtimer_start_range_ns (hrtimer_wakeup)
  0.0% ( 0.1)      pdflush : add_timer (commit_timeout)
  0.0% ( 0.1)     klauncher : hrtimer_start_range_ns (hrtimer_wakeup)

And under the same load as above:

Wakeups-from-idle per second : 546.5  interval: 10.0s
no ACPI power usage estimate available

Top causes for wakeups:
 18.1% (214.8)   <kernel IPI> : Rescheduling interrupts
 13.5% (159.8)   <kernel core> : hrtimer_start_range_ns (tick_sched_timer)
 13.4% (158.4)   npviewer.bin : hrtimer_start_range_ns (hrtimer_wakeup)
 12.9% (152.5)    <interrupt> : eth0, snd_ca0106
  8.0% ( 95.1)  USB device 2-4 : USB-PS/2 Optical Mouse (Logitech)
  7.9% ( 94.1)   <kernel IPI> : TLB shootdowns
  6.9% ( 82.1)    <interrupt> : ohci_hcd:usb2
  5.2% ( 61.8)    <interrupt> : nvidia
  4.7% ( 56.3)      firefox : hrtimer_start_range_ns (hrtimer_wakeup)
  3.9% ( 45.9)    <interrupt> : pata_amd
  1.4% ( 16.2)       kwin : hrtimer_start_range_ns (hrtimer_wakeup)
  1.2% ( 14.5)       opera : hrtimer_start_range_ns (hrtimer_wakeup)
  0.8% ( 9.6)      kswapd0 : io_schedule_timeout (process_timeout)
  0.3% ( 3.5)     ksysguard : hrtimer_start_range_ns (hrtimer_wakeup)
  0.3% ( 3.0)     knotify4 : hrtimer_start_range_ns (hrtimer_wakeup)
  0.2% ( 2.1)      krunner : hrtimer_start_range_ns (hrtimer_wakeup)
  0.1% ( 1.4)   <kernel core> : cfq_arm_slice_timer (cfq_idle_slice_timer)
  0.1% ( 1.4)       kwin : sk_reset_timer (tcp_delack_timer)
  0.1% ( 1.2)       opera : io_schedule_timeout (process_timeout)
  0.1% ( 1.1)      firefox : io_schedule_timeout (process_timeout)
  0.1% ( 1.0)  plasma-desktop : nv_kern_rc_timer (nv_kern_rc_timer)
  0.1% ( 1.0)  operapluginwrap : hrtimer_start_range_ns (hrtimer_wakeup)
  0.1% ( 0.9)        adb : hrtimer_start_range_ns (hrtimer_wakeup)
  0.1% ( 0.8)  hald-addon-stor : hrtimer_start_range_ns (hrtimer_wakeup)

Is it strange that the wakeups would be so much higher when idle but about the same as before when busy?

I also tried the suggestion of looking for something that keeps taking RAM and not giving it back without success. The only things I see that keep taking more and more RAM are the Xorg and plasma-desktop processes. Which is odd because I don't recall plasma-desktop being as big of an issue before. Right now Xorg is using 591.6 M and plasma-desktop is using 653.5. I only have 2 GB so those 2 processes alone are using over 1/2 my memory.

I'm growing frustrated with trying to solve this issue. Any further suggestions would be greatly appreciated but I think I may install gnome and try running it for a while. That should at least let me know if the problem is being caused by KDE or something else.

[GreyGeek]

Re: Performance degrades over time; have to reboot every few days

I also tried the suggestion of looking for something that keeps taking RAM and not giving it back without success. The only things I see that keep taking more and more RAM are the Xorg and plasma-desktop processes. Which is odd because I don't recall plasma-desktop being as big of an issue before. Right now Xorg is using 591.6 M and plasma-desktop is using 653.5. I only have 2 GB so those 2 processes alone are using over 1/2 my memory.

I see a couple problems, probably both related to your video drivers. Your xorg and plasma-desktop are 20 times larger than what they should be. Mine, for my Intel i915 driver, are 34Mb for xorg and 27Mb for the plasma-desktop, and those values are relatively constant, regardless of what I am running. The shared memory is 13.5Mb and 27Mb respectively, and all four values oscillate up and down by a few hundred Kb each.

Your very large Kernel IPI Function Calls and Interrupt rescheduling counts are probably caused by video problems and/or kernel problems associated with them. You can check this by running ksystemlog:
Run KGear --> System --> KSystemLog and click on the "Xorg Log"
On KSystemLog you can also check the kernel log, the system log and Deamon Log for a flood of reoccurring error listings.

If you see these xorg errors, or kernel errors, one thing you can do is boot a previous kernel, if you still have one or more of them in the grub menu listing. Otherwise, use synaptic to install and try previous kernels to see if one of them clears up the problem.

[Azurescens]

Re: Performance degrades over time; have to reboot every few days

I really appreciate your continued support on this issue.

I rebooted with the oldest kernel available to me, 2.6.31-14 I think it was. The Xorg and plasma-desktop processes were both using around 20Mb immediately after boot but they've crept up to around 30Mb and it's only been about 15 or 20 minutes. They're currently using 12Mb and 27Mb of Shared Memory respectively.

I briefly looked at the log files and didn't notice a significant amount of errors in any of the ones you mentioned. I did, however, see a large number of "Rejected send message" errors in the Authentication Log. I seem to get one from the dbus-daemon process almost every time I open a window. The message is always the same except for the destination part:

Rejected send message, 1 matched rules; type="method_call", sender=":1.24" (uid=1000 pid=1695 comm="kdeinit4: plasma-desktop [kdeinit]") interface="org.freedesktop.DBus.Introspectable" member="Introspect" error name="(unset)" requested_reply=0 destination=":1.49" (uid=1000 pid=2335 comm="/usr/lib/firefox-3.5.7/firefox))

Could this be related to my issue?

[GreyGeek]

Re: Performance degrades over time; have to reboot every few days

....
I rebooted with the oldest kernel available to me, 2.6.31-14 I think it was.
The Xorg and plasma-desktop processes were both using around 20Mb immediately after boot
but they've crept up to around 30Mb and it's only been about 15 or 20 minutes. They're currently
using 12Mb and 27Mb of Shared Memory respectively.

Those values appear normal -- starting at around 20-30 Mb and oscillating around some value near 30Mb as you start and stop various applications. Keep using the 31-14 kernel and see if you get the 500 - 600Mb values you mentioned in your previous post.

I briefly looked at the log files and didn't notice a significant amount of errors in any of the ones you mentioned. I did, however, see a large number of "Rejected send message" errors in the Authentication Log. I seem to get one from the dbus-daemon process almost every time I open a window. The message is always the same except for the destination part:

Rejected send message, 1 matched rules; type="method_call", sender=":1.24" (uid=1000 pid=1695 comm="kdeinit4: plasma-desktop [kdeinit]") interface="org.freedesktop.DBus.Introspectable" member="Introspect" error name="(unset)" requested_reply=0 destination=":1.49" (uid=1000 pid=2335 comm="/usr/lib/firefox-3.5.7/firefox))

Could this be related to my issue?

I don't think so. While an app may be malfunctioning and eating memory, just sending out messages doesn't mean it is, unless the message was posted by code designed to watch for such stuff. That is why I asked you to check the kernel system log and other logs. That you didn't find any repetitive messages in the logs probably means that the code surrounding the leak isn't outputting an error message because the coder didn't anticipate any problems there.

What you are looking for is an application that causes the installation to "leak" heap memory, as I pointed out in a previous post. The difference between the stack and heap memory are explained here. When memory is consumed by an application it is generally proof that the application has a bug in it, or in its reaction with firmware code like video drivers.

In your case it "appears" that your video driver and/or the plasma-desktop is leaking memory. Since others are not reporting unusual memory consumption by the plasma-desktop (typically around 30Mb) I suspect that it is your xorg video driver and its reaction with your video chip that is eating more and more of your memory as it reacts with your desktop. Changing your video driver (go up or down a version) may eliminate your problem, if the change in kernel does not. Or, you may have a defective video chip.

[Zeikcied]

Re: Performance degrades over time; have to reboot every few days

In your case it "appears" that your video driver and/or the plasma-desktop is leaking memory. Since others are not reporting unusual memory consumption by the plasma-desktop (typically around 30Mb) I suspect that it is your xorg video driver and its reaction with your video chip that is eating more and more of your memory as it reacts with your desktop. Changing your video driver (go up or down a version) may eliminate your problem, if the change in kernel does not. Or, you may have a defective video chip.

I've been having massive memory problems with Plasma starting back in KDE 4.0. Every new KDE version I keep hoping for the problem to be fixed. I'm using KDE 4.4 from the Kubuntu Backports PPA (using Kubuntu Karmic 9.10).

My plasma process is currently at 369.4 Megs and xorg is at 116.6 Megs.

I also use the proprietary NVIDIA video driver.

If you need my hardware, it's...

CPU: AMD Athlon 64 3500+
GPU: GeForce 7600 GS 215 Meg PCI-E (I don't remember the manufacturer)
RAM: 1 Gig
NVIDIA Driver Version: 185.18.36

I really don't know much else about my hardware, but I don't think anything else is really going to be an issue.

[Zeikcied]

Re: Performance degrades over time; have to reboot every few days

Well, I went to NVIDIA's website and downloaded the most recent version 195 driver, and now plasma is holding at a steady 34 Megs for the past several hours. So it seems to be an issue with the 185 series of drivers.

I hope Kubuntu Lucid offers the 195 driver, but I'm not seeing it in the Packages list.

[GreyGeek]

Re: Performance degrades over time; have to reboot every few days

Ah, so it was a video driver issue.

Open Synaptic and "pin" your current driver so the future "upgrades" don't replace it.
Select the video driver you are now using, click on the "Packages" menu option and choose "Lock Version".
That will prevent future upgrades from replacing it. "IF it ain't broken don't fix it!"