swap gradually filling after hibernate / resume

[rst]

Hello,
I am running 12.04 on a toshiba satellite L300, 4GB Ram. updated continuously since something like 9.10 (uname -a gives 39 kernel compilations)
I have gotten in the habit of using hibernate instead of shutdown when I'm finished with the machine.
according to the system monitor plasmoid, when I start everything up, I use about 1.5 to 2 GB of RAM and no swap. No problem. I put the machine into hibernate, and then I turn it back on, and the swap usage is quite high, but doesn't really go down. After a number of these cycles, I start to get an error message to the effect of not enough swap available.
Reboot and I'm back in business for a while.
If somebody would explain how to fix this "memory leak", point me to a man page/forum link/comment or explain to me how I don't know anything and this is expected behaviour I would appreciate it.
I usually have chrome open with 20 to 30 tabs, Kmail 2, sometimes the odd PDF, spreadsheet or terminal. A few servers running in the background.
I would be happy to provide logs or diagnostic outputs, but I'm not sure where to start and nobody wants to see 10 000 lines of "this might help".
Thanks,
Ray

[dibl]

Hibernate uses the swap space to save an image of the running system, and then upon resume that image gets loaded into memory. The fact that all is fine after a reboot is "expected behavior", since booting starts with an empty swap space.

It sounds like the image that gets saved when you hibernate is not being purged when you resume, such that you are accumulating images until swap is full. With 4 GB of RAM, I don't know why your system should ever need to swap -- the packages that you have named would not cause that. Although "servers in the background" is an interesting comment -- what are those servers doing back there?

[rst]

Hibernate uses the swap space to save an image of the running system, and then upon resume that image gets loaded into memory. The fact that all is fine after a reboot is "expected behavior", since booting starts with an empty swap space.

exactly what I expected

It sounds like the image that gets saved when you hibernate is not being purged when you resume, such that you are accumulating images until swap is full.

that's what I thought too. How do I purge the images?

Although "servers in the background" is an interesting comment -- what are those servers doing back there?

local copies of php / mysql / postgresql for testing purposes. No ports opened in the router. no other users in the house knowledgeable enough to access them. They are idle 99.9% of the time, maybe more.

Thanks,
Ray

[dibl]

How do I purge the images?

I'm away from my system at the moment, so I can't test it, but I would assume that if you issue

sudo swapoff

and then immediately

sudo swapon

that it would restart an empty swap space. I don't know whether you'll have to use the device ID as well, or not. If so, it would be

sudo swapon /dev/sda2

or whatever your swap partition is.

I think there's some problem in the 12.04 default resume configuration --- check the problem that another user is having on the 12.04 hardware forum.

[rst]

I tried swapoff, swapon with a -a flag (why do I think of old, bad commercials when I say that?) and all it did was move all the information in swap into physical memory.
Searching using these commands as keywords I came up with some suggestions to try

echo 1 > /proc/sys/vm/drop_caches
echo 2 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches

but that didn't make very much difference

vmstat -a

shows that about 2/3 of the memory being used is inactive.
Any further hints?
Ray

[dibl]

I don't think you want to mess with the /proc/sys/vm settings. Remember, the use of the swap area for the hibernate image is AFAIK totally unrelated to the rest of the memory management functions of your system. In other words, a problem related to resuming from hibernate SHOULD be totally unrelated to the virtual memory settings and configuration. At least it seems that way to me at this juncture. After you boot your computer, and before you send it into hibernation the first time, is there any evidence of any memory management issues? With 4 GB of memory, and running typical desktop packages,

free -m

should never show any usage of swap at all. Is that what you see before you hibernate it?

[rst]

I don't think you want to mess with the /proc/sys/vm settings.

according to this http://www.linuxinsight.com/proc_sys...op_caches.html these commands are non destructive. I understand that to mean you can't wreck anything

Remember, the use of the swap area for the hibernate image is AFAIK totally unrelated to the rest of the memory management functions of your system. In other words, a problem related to resuming from hibernate SHOULD be totally unrelated to the virtual memory settings and configuration. At least it seems that way to me at this juncture. After you boot your computer, and before you send it into hibernation the first time, is there any evidence of any memory management issues? With 4 GB of memory, and running typical desktop packages,

free -m

should never show any usage of swap at all. Is that what you see before you hibernate it?

swap is only ever used after hibernate that I can tell, but since after the swapoff, swapon combination, everything is now in normal memory, I thought I would see what I could find out.

[SecretCode]

My experience is that swap usage (as reported by the system load viewer plasmoid or system monitor or free -m) once used doesn't ever go down, at least not by much - unless you use that swapoff command, of course.

But also my experience - with a 4GB RAM system and frequent hibernation - is that the reported swap is unaffected by a hibernate cycle. (The hibernate process certainly uses the swap area, but I assume it's freed after resume and before the monitor plasmoid gets to see it).

What I assume is happening to you is that some process(es) is(are) hanging and getting restarted on each hibernate cycle, and reserving quite a lot of memory in so doing. One way to investigate is to look at the rss column in top or ps, e.g.

ps -e -o pid,etime,comm,%cpu,time,%mem,rss:7,vsz:7 --sort -rss | head -n 11

rss is (I think) resident set size and is the closest simple measure of "how much memory" a process is using. It's much more complicated than that because lots of bits of memory can be shared between processes. If you investigate which processes are at the top of the output after a fresh boot and after a hibernate cycle or two it should give some clues.

Also, I assume that the output of free -m confirms that free swap space is reducing?

[rst]

starting from the state mentioned in previous posts I ran:
ps command as given by SecretCode
free-m as mentioned above
and vmstat
I then rebooted (restarted), restarted the same programs and ran the same commands
I then went into hibernation and then restored and then ran the same commands
and then one more time I went into hibernation and then restored and then ran the same commands
all results are posted below.
they clearly show swap usage start at zero on clean boot, and then increment after each hibernation cycle.

I'm not sure what virtuoso-t is in the ps list. packagekitd I assume is the pending updates that were detected on reboot.
Thanks,
Ray

all swap contents moved to regular ram, due to 
#swapoff -a && swapon -a

$ ps -e -o pid,etime,comm,%cpu,time,%mem,rss:7,vsz:7 --sort -rss | head -n 11
  PID     ELAPSED COMMAND         %CPU     TIME %MEM     RSS     VSZ
18885  3-17:29:02 okular           0.0 00:01:43 11.6  457232 1047328
 2537  6-05:04:45 chrome           0.4 00:42:03  4.7  185120  870552
 2697  6-05:04:19 chrome           0.2 00:21:33  4.6  181860 1092232
 1852  6-05:06:01 plasma-desktop   0.8 01:16:23  4.4  172728 1729448
 2664  6-05:04:22 chrome           0.0 00:04:07  2.8  111144  969456
 2706  6-05:04:17 chrome           1.0 01:32:36  2.6  104864 1033436
 2416  6-05:05:02 virtuoso-t       0.0 00:05:50  2.4   95904  649960
29777    18:25:51 chrome           0.1 00:01:09  2.2   88660  924036
 2556  6-05:04:43 kmail            0.0 00:03:48  1.8   73340 2321068
 4283  6-04:32:45 chrome           0.1 00:09:48  1.8   72212  929896


$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       3386        438          0         28        562
-/+ buffers/cache:       2795       1029
Swap:         3318         22       3295


$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
 1  0  22772 446120 1881768 1363084   12   11    49    40    6    7  7  2 90  2
[note inactive =1.8G, active=1.3G]

restart system

open:
  chrome (set to reopen same tabs)
  kmail
  kate (my notes for this reply)
  Konsole
  Okaular (same file aprox. 25MB)

$ ps -e -o pid,etime,comm,%cpu,time,%mem,rss:7,vsz:7 --sort -rss | head -n 11
  PID     ELAPSED COMMAND         %CPU     TIME %MEM     RSS     VSZ
 2291       05:23 plasma-desktop   3.5 00:00:11  4.2  166064 1430760
 2711       03:06 chrome           6.5 00:00:12  2.8  112380  739012
 2656       04:02 virtuoso-t      26.6 00:01:04  2.3   91428  649820
 3418       01:40 okular           1.9 00:00:01  2.3   90164  565964
 2730       02:58 kmail            0.8 00:00:01  2.1   84060 1915084
 2845       02:34 chrome           1.0 00:00:01  2.0   80372  908052
 2282       05:32 kwin             2.7 00:00:09  1.6   65716  746424
 3266       01:50 java             1.8 00:00:02  1.6   63180 2341044
 2877       02:32 chrome           0.7 00:00:01  1.6   62848  899392
 2901       02:30 chrome           0.7 00:00:01  1.5   60472  896320
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       2973        852          0         58       1138
-/+ buffers/cache:       1776       2048
Swap:         3318          0       3318
$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
 1  0      0 874220 907236 1883704    0    0   908   388  505  898 15  4 39 43
[note inactive=0.9G, active=1.8G]


hibernate, resume


$ ps -e -o pid,etime,comm,%cpu,time,%mem,rss:7,vsz:7 --sort -rss | head -n 11
  PID     ELAPSED COMMAND         %CPU     TIME %MEM     RSS     VSZ
 2291       13:23 plasma-desktop   2.1 00:00:17  1.8   70724 1431048
 2711       11:06 chrome           2.1 00:00:14  1.7   67868  755208
 3418       09:40 okular           0.3 00:00:02  1.5   60200  565964                                                                                                                
 2845       10:34 chrome           0.3 00:00:02  1.4   56656  908052                                                                                                                
 4498       06:35 packagekitd      3.7 00:00:14  1.2   49488  472940                                                                                                                
 3266       09:50 java             0.3 00:00:02  1.1   46896 2341044                                                                                                                
 2776       10:42 chrome           0.4 00:00:02  1.0   42008  894232                                                                                                                
 2877       10:32 chrome           0.2 00:00:01  0.9   39040  899392                                                                                                                
 2947       10:22 chrome           0.3 00:00:01  0.9   37768  894048
 2901       10:30 chrome           0.2 00:00:01  0.9   35468  895296
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1789       2035          0          7        236
-/+ buffers/cache:       1545       2279
Swap:         3318        524       2793
$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
 1  1 536940 2071840 676372 954512  101  317   824   659  472  796 12  3 44 41


hibernate, resume


$ ps -e -o pid,etime,comm,%cpu,time,%mem,rss:7,vsz:7 --sort -rss | head -n 11
  PID     ELAPSED COMMAND         %CPU     TIME %MEM     RSS     VSZ
 2291       18:02 plasma-desktop   1.9 00:00:21  1.9   75584 1431108
 3418       14:19 okular           0.2 00:00:02  1.5   61248  565964
 4498       11:14 packagekitd      2.6 00:00:17  1.5   59808  505724
 2656       16:41 virtuoso-t       7.2 00:01:12  1.4   56172  649984
 2845       15:13 chrome           0.2 00:00:02  1.3   52160  908052
 3266       14:29 java             0.2 00:00:02  1.2   47136 2341044
 2776       15:21 chrome           0.3 00:00:03  1.0   40392  894232
 2711       15:45 chrome           1.5 00:00:14  0.9   37412  755200
 3453       14:16 chrome           0.1 00:00:01  0.8   32440  907088
 2330       17:35 mysqld           0.2 00:00:02  0.8   31968 1250484
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1565       2260          0          7        251
-/+ buffers/cache:       1306       2518
Swap:         3318        639       2678
$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
 2  0 654376 2314064 486644 908012  101  328   795   626  467  772 11  3 48 39

[SecretCode]

Hmm. Since your free RAM remains good, it's not a case of processes consuming lots of memory as I suspected. ... Though the fact that the rss value for Okular (and plasma-desktop and the chromes) goes right down suggests something may be still in use but no longer included in the rss value. This would be beyond my understanding of Linux memory management though.

(Having lots of chrome processes is normal, isn't it? Separate processes per tab or something?)

Unfortunately I don't really have time this week to monitor my memory while going through a few hibernate cycles!

[rst]

(Having lots of chrome processes is normal, isn't it? Separate processes per tab or something?)

I think that's a semi-new feature designed to prevent the whole program from crashing if one tab does.

Well I just did a fresh reboot, so I should be good for a week or so. I'll just leave it like this and see if anybody comes up with any good ideas.
Thanks,
Ray

[SteveRiley]

I'm not sure what virtuoso-t is in the ps list. packagekitd I assume is the pending updates that were detected on reboot.

Virtuoso is the database backend for Nepomuk.

I'm surprised that you have PackageKit on your system, it isn't normally included in a Kubuntu Precise install. Is your system upgraded from an earlier release?

[SecretCode]

I'm surprised that you have PackageKit on your system, it isn't normally included in a Kubuntu Precise install. Is your system upgraded from an earlier release?

Yes it is ...

I am running 12.04 on a toshiba satellite L300, 4GB Ram. updated continuously since something like 9.10 (uname -a gives 39 kernel compilations)

rst, do you have a free partition on this machine on which you could do a clean install?

As a simpler test, what happens if you create a new user account (under your current install) and run it through a few hibernate cycles?

[dibl]

starting from the state mentioned in previous posts I ran:
ps command as given by SecretCode
free-m as mentioned above
and vmstat
I then rebooted (restarted), restarted the same programs and ran the same commands
...

Your memory and swap figures look fine, although I'm not sure why okular was sucking so much memory. For comparison, I fired up okular and opened a small file, opened chromium-browser with about 6 tabs including a flash dislpay, and ran the same commands. This system has 6 GB of memory, running i7-950:

don@imerabox:~$ ps -e -o pid,etime,comm,%cpu,time,%mem,rss:7,vsz:7 --sort -rss | head -n 11
  PID     ELAPSED COMMAND         %CPU     TIME %MEM     RSS     VSZ
32173    03:02:39 kwin             0.6 00:01:10  2.6  163200  939248
32187    03:02:38 plasma-desktop   0.1 00:00:15  2.1  134248  919080
32021    03:02:44 Xorg             4.2 00:07:45  1.8  116024  223340
25696       00:33 chromium         7.2 00:00:02  1.5   95752  979152
25750       00:33 chromium         3.1 00:00:01  1.3   83196 1035196
25764       00:33 chromium         3.2 00:00:01  1.3   80064 1039220
25510       00:48 okular           0.8 00:00:00  1.0   64760  491936
25757       00:33 chromium         2.7 00:00:00  1.0   63084 1019804
25778       00:33 chromium         1.5 00:00:00  0.9   55032 1009276
25725       00:33 chromium         1.5 00:00:00  0.8   52032 1011300

don@imerabox:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          5966       3352       2613          0         12       1924
-/+ buffers/cache:       1415       4550
Swap:        14760          0      14760                                                                                                     

don@imerabox:~$ vmstat -a                                                                                                                    
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----                                                                
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa                                                                
 1  0      4 2658172 1216768 1819160    0    0     3     4    0   12  1  1 99  0

I have vm.swappiness turned down to 1, since I have so much memory.

[rst]

rst, do you have a free partition on this machine on which you could do a clean install?

As a simpler test, what happens if you create a new user account (under your current install) and run it through a few hibernate cycles?

I don't have a free partition, but I did try your second idea, and the results where very interesting. (free -m outputs follow body of post)
I created a new user, test, and then rebooted into that user. Even after 3 hibernate/restore cycles swap usage stayed at zero.
I then logged out of test and logged in to my usual account. swap at zero so far. I didn't start any programs I didn't need. after one hibernate/restore cycle, a little bit of swap was being used, after a second cycle, slightly less swap was being used. I then opened one program at a time, followed by a hibernate/restore cycle. no significant change until after chrome, when swap use jumped markedly.
I interpret this to mean there is a problem with either my main account, or a program loaded automatically with it, (not the system) and either chrome makes that problem worse, or chrome (or one of the tabs in it) also has the problem.
I just realized I didn't try chrome in the test account. I'll do that and post the results separately.

Your memory and swap figures look fine, although I'm not sure why okular was sucking so much memory.

It was a 25MB PDF

Thanks,
Ray

Reboot into new test user 
opened kate to take notes
Dolphin to access FTP server to store notes file
and konsole to run commands

$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1159       2665          0         27        586
-/+ buffers/cache:        544       3280
Swap:         3318          0       3318

hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1324       2500          0         31        643
-/+ buffers/cache:        650       3174
Swap:         3318          0       3318

hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1307       2517          0         31        641
-/+ buffers/cache:        634       3190
Swap:         3318          0       3318

hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1302       2522          0         31        639
-/+ buffers/cache:        631       3193
Swap:         3318          0       3318


log out of test user/ log in to reguler user
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1756       2069          0         39        914
-/+ buffers/cache:        801       3023
Swap:         3318          0       3318

hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1134       2690          0         11        338
-/+ buffers/cache:        783       3041
Swap:         3318         56       3261


hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1216       2609          0         16        400
-/+ buffers/cache:        799       3025
Swap:         3318         48       3269


hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1228       2596          0         16        411
-/+ buffers/cache:        800       3024
Swap:         3318         48       3270

open ockular
hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1322       2502          0         18        441
-/+ buffers/cache:        862       2962
Swap:         3318         47       3270

open kmail2
hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1362       2462          0         16        452
-/+ buffers/cache:        893       2931
Swap:         3318         47       3270


open chrome
hibernate/resume
$ free -m
             total       used       free     shared    buffers     cached
Mem:          3825       1710       2114          0          6        250
-/+ buffers/cache:       1453       2371
Swap:         3318        392       2925