High CPU load during startup and sometimes after

[scislot]

Hi All,

I have a problem with my Kubuntu 12.10. It starts way slower than my previous Kubuntu 11.04. It seams that one CPU core is loaded in 100% while the system boots and approximately 2 minutes after login. I figured that maybe this is an issue with kernel threads but I cannot see nothing suspicious in htop or top. Surprisingly I found out about the CPU high load from the graphics indicator on a bottom bar.
This sudden high CPU load happens randomly after system boot.
At first I though that this could be the issue with Nvidia graphic drivers, but I changed them from current to experimental-310 and nothing has changed.

Any ideas on how to debug the issue?

Regards,
ToM

[SteveRiley]

My first suggestion would be to install the Boot Chart utility, which shows how long each aspect of the boot process takes.

sudo apt-get install bootchart

Once the installation has finished, reboot your PC. After you log in, you can find the chart in /var/log/bootchart -- it's the file with the ".png" extension. Attach that file to a reply here, and we can all take a look.

[oshunluvr]

My first guess - the evil trilogy: Nepomuk, Strigi, Akonadi

[SteveRiley]

http://vhanda.in/blog/2012/11/nepomuk-without-strigi/

[scislot]

Unfortunately now it's random. Sometimes the OS boots normally and sometimes with delay. Just to be clear I already turned of: Nepomuk, Strigi, Akonadi
Below is my bootchart – this is the time when it took long to load OS.

http://tscislo.rootnode.net/pub/imag...20130116-1.png

ToM

[SteveRiley]

Mind posting that on imgur.com and providing the URL? It's too small to see here.

[scislot]

http://tscislo.rootnode.net/pub/imag...20130116-1.png

[scislot]

Below:
http://tscislo.rootnode.net/pub/imag...20130116-1.png

[scislot]

Below:
http://tscislo.rootnode.net/pub/imag...20130116-1.png

[scislot]

Unfortunately now it's random. Sometimes the OS boots normally and sometimes with delay. Just to be clear I already turned of: Nepomuk, Strigi, Akonadi
Below is my bootchart – this is the time when it took long to load OS.

http://tscislo.rootnode.net/pub/imag...20130116-1.png

ToM

[scislot]

Any ideas guys? Can you please have a look at my bootchart output.

Regards,
ToM

[PraysToPan]

This sounds quite a bit like he awful issues I've had since upgrading to 12.10. Everything I have seen, on my system, points to a rather nasty kernel issue.

I'm having the problem on a slightly older Dell D-420 notebook. It has a 32-bit Core-2 U2550 @ 1.2GHz with 2GB of RAM. That being said, your system may be a generation ahead of mine, but you seem to have quite a bit more task load than I do, too.

Anyhow, I opened a bug on Launchpad: https://bugs.launchpad.net/ubuntu/+s...x/+bug/1159258

I detail everything I have noticed and the things I have tried. Currently, my system is %50-%60 faster. The difference is incredible.

It's a bit of a pain to apply the fix, though. If you can post me some stats, I can tell you if you're likely going through the issue I had.

What CPU and how much RAM are you using? I'll also need about two minutes of "vmstat 1" output, and the contents of /proc/meminfo and /proc/vmstat. Oh, and I need a breakdown of your CPU usage by user, system, wait, and system interrupts. A rough estimate from watching 'top' for a minute or two, is fine. Of course, do it all when your system seems to be at it's worst.

Jon

[SteveRiley]

Hm, someone else appears to be tinkering with these settings.

https://lkml.org/lkml/2013/4/7/12

Very neat find.

[PraysToPan]

The common practice of using much lower time slices in the hope that it improves latency for workloads such as desktops, in my case, just exasperated the problem. (ie. CONFIG_HZ=250)

When you think about it, I have a bunch of tasks which are doing lots of page allocations, on my deskotp. One of the worst, is the modern web browser. In my case, Google Chrome, is full of anticipatory memory use. Opening and closing tabs in numbers and the system is spending significant time allocating and deallocating memory.

So, we have, by default, 4k pages which can be requested in tens of megs at a time. Small pages mean huge page tables. Walking those huge tables means the translation lookaside buffers suffer more misses than hits. That's hundreds of cpu cycles wasted. Add the smaller time slice to that, and a process ends up shoved aside after getting nothing more done that walking a few megs of page tables. On a 2GB system, I would regularly have over 100MB of page tables allocated! Of course a process just spends time going from one context switch to another and you've also wasted your CPU's cache! On my system, system CPU time was at best, neck and neck with user process time and quite often, twice as high.

A perfect storm!

[SteveRiley]

Fun fun.

Here's my HP Mini 2140, running Precise as a server. Just a CLI setup, no X. It has 2 GB RAM.

root@mini2140:~# [B]cat /proc/meminfo | grep -i page[/B]
AnonPages:        486140 kB
PageTables:         4796 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

root@mini2140:~# [B]cat /sys/block/sda/queue/scheduler[/B] 
noop [deadline] cfq

root@mini2140:~# [B]hdparm -I /dev/sda[/B]
ATA device, with non-removable media
        Model Number:       TOSHIBA MK5056GSY                       
        Serial Number:      40KKT4RBT
        Firmware Revision:  LH003C  
        Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6
Standards:
        Supported: 8 7 6 5 
        Likely used: 8
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors:  976773168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                   512 bytes
        device size with M = 1024*1024:      476940 MBytes
        device size with M = 1000*1000:      500107 MBytes (500 GB)
        cache/buffer size  = 16384 KBytes
        Form Factor: 2.5 inch
        Nominal Media Rotation Rate: 7200

Only 4 MB of page tables, and a spinning disk managed by deadline, the default for server kernels in Precise. (Precise desktops sill use CFQ. Starting with Quantal, Canonical removed the separate kernel configs for servers and desktops, and set the default scheduler to deadline for all builds.)

And here's my ThinkPad T520 laptop, running Quantal desktop. It has 8 GB RAM.

steve@t520:~$ [B]cat /proc/meminfo | grep -i page[/B]
AnonPages:        994956 kB
PageTables:        37064 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

steve@t520:~$ [B]cat /sys/block/sda/queue/scheduler[/B]
[noop] deadline cfq 

steve@t520:~$ [B]sudo hdparm -I /dev/sda[/B]
ATA device, with non-removable media
 Model Number:       M4-CT256M4SSD2                          
 Serial Number:      000000001140031E608A
 Firmware Revision:  0009    
 Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
 Used: unknown (minor revision code 0x0028) 
 Supported: 9 8 7 6 5 
 Likely used: 9
Configuration:
 Logical  max current
 cylinders 16383 65535
 heads  16 1
 sectors/track 63 63
 --
 CHS current addressable sectors:   16515009
 LBA    user addressable sectors:  268435455
 LBA48  user addressable sectors:  500118192
 Logical  Sector size:                   512 bytes
 Physical Sector size:                   512 bytes
 Logical Sector-0 offset:                  0 bytes
 device size with M = 1024*1024:      244198 MBytes
 device size with M = 1000*1000:      256060 MBytes (256 GB)
 cache/buffer size  = unknown
 Form Factor: 2.5 inch
 Nominal Media Rotation Rate: Solid State Device

Currently 37 MB of page tables, and an SSD managed by noop -- I always set this on SSD-equipped machines.

Since the little server doesn't do much other than sit in my closet and deliver mail (mostly), it's hardly under any pressure at all. And since my laptop has RAM to spare (and I don't use Chrom(e)ium), I'm not running into the problems you are. It sure seems like you've found an edge case that possibly isn't all that edgy, and perhaps represents something worthy of consideration for most folks in your situation.