Looking into swap usage
Looking into the machine performance I came to the question whether the system was lacking ram and abusing the swap space. I came to this question wondering if the system was lacking ram. In which case I expected to see a lot of swap usage and lot of processes waiting for memory.
My first idea was to look into how much swap memory was been used. Then I thought to look into which processes were the ones consuming more ram and swap, trying to find a responsible. It was at this point I thought that the fact that the swap is totally full does not mean that the system is running low in memory at this point. It could be that the system had a burst in the past, the swap was used and never released. Similar to linux ate my ram. Then I looked into the memory pressure and the live swap usage.
Looking into total swap usage
This can be easily done with:
free -m
total used free shared buff/cache available
Mem: 15647 10707 693 1214 4246 3381
Swap: 2047 1980 67
Here it can be seen that almost all the swap is consumed.
Looking into process memory usage with smem
smem is very simple to use. It is a tool that runs on python and the set up is rather quick.
Understanding its output
The tool reports four different columns:
- Swap. Hard drive space used as RAM.
- USS. Unique Set Size. Amount of RAM memory only used by the given process. It might be private or shared (if no other process is using it). This amount of memory will be free once the given process terminates.
- RSS. Resident Set Size. How much RAM memory this process has used recently. It includes private and shared memory.
- PSS. Proportional Set Size. USS + (total shared memory / number of processes sharing it). All of it in RAM
Commands
To check all the processes with a readable output, run the command:
smem -k -t
To check the whole system swap usage run:
smem -w -k -t
The output can be sorted by a given field. To find out the top 10 processes consuming more swap:
smem -s swap -r -k | head -n 11
To search among the output, you can pipe it directly to something like fzf
smem -s swap -r -k | fzf
This is very useful if you suspect a given process is consuming a lot of swap. You can dynamically analyze the output without the need of saving the file. So you can search in the terminal for a given process name.
For example in my case I was concern that all my snap installed packages are consuming some swap memory. So from the command above I could search for firefox, slack or pycharm. Here is how it looks to use fzf to filter for pycharm.
30594 username /snap/pycharm-professional/ 3.6M 584.0K 586.0K 852.0K
29997 username /snap/pycharm-professional/ 633.4M 3.7G 3.7G 3.7G
Once you have found a process or program that you would like to analyze, you can ask smem to filter its output for that specific name. Following the hint above, lets filter for pycharm
smem -P pycharm -k -t -s swap
PID User Command Swap USS PSS RSS
587310 username /usr/bin/python3 /usr/bin/s 0 13.7M 14.0M 17.0M
30594 username /snap/pycharm-professional/ 3.6M 584.0K 586.0K 852.0K
245818 username /usr/local/bin/node /snap/p 17.9M 735.1M 739.4M 744.2M
246042 username /usr/local/bin/node /snap/p 38.0M 134.5M 138.8M 143.8M
29997 username /snap/pycharm-professional/ 633.4M 3.6G 3.7G 3.7G
-------------------------------------------------------------------------------
5 1 692.9M 4.5G 4.5G 4.5G
Now lets compare that to the output of firefox
smem -P firefox -k -t -s swap
PID User Command Swap USS PSS RSS
601932 username /usr/bin/python3 /usr/bin/s 0 18.5M 18.8M 21.8M
365867 username /usr/lib/firefox/firefox -c 52.0K 248.4M 264.5M 330.5M
514115 username /usr/lib/firefox/firefox -c 52.0K 27.5M 44.0M 121.6M
550072 username /usr/lib/firefox/firefox -c 52.0K 77.4M 95.4M 175.0M
563710 username /usr/lib/firefox/firefox -c 52.0K 11.8M 20.7M 76.6M
563755 username /usr/lib/firefox/firefox -c 52.0K 11.8M 20.7M 76.5M
578553 username /usr/lib/firefox/firefox -c 52.0K 12.0M 21.1M 76.9M
10630 username /usr/lib/firefox/firefox -c 6.5M 1.9M 2.1M 4.9M
10665 username /usr/lib/firefox/firefox -c 8.2M 97.4M 104.5M 144.4M
10934 username /usr/lib/firefox/firefox -c 8.9M 2.6M 3.5M 9.1M
10734 username /usr/lib/firefox/firefox -c 67.0M 102.3M 108.4M 146.8M
10563 username /usr/lib/firefox/firefox 73.4M 437.3M 470.7M 553.5M
-------------------------------------------------------------------------------
12 1 164.4M 1.0G 1.1G 1.7G
This was surprising. I always assumed that firefox would be the process using the most swap, since I have an infinity number of open tabs. On the other hand, I have experienced performance problems with pycharm in the past.
Just to get the full picture, lets print the total output
smem -k -t | (head -n1 && tail -n1)
PID User Command Swap USS PSS RSS
155 2 1.7G 7.9G 8.3G 9.6G
Lets see what percentage of memory is currently been used:
smem -p -t | (head -n1 && tail -n1)
PID User Command Swap USS PSS RSS
155 2 84.98% 52.96% 55.17% 64.02%
It seems that there is large % of swap been used. However, that might not imply that the system requires more memory, since it can be inactive swap, which was filled at some point of pressure and never released.
Looking into waiting times for memory
The pressure stall files give us information about how much time the runnable processes are waiting to obtain access to a given resource.
There is pressure stall information for three different resources: cpu, memory and i/o. For each of these resources there is a file, usually found in /proc/pressure.
For the case of memory, lets look into its file:
cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=62121474
full avg10=0.00 avg60=0.00 avg300=0.00 total=49492001
These results are accumulated over periods of 10s, 60s and 300s. This hints if the usage is increasing, decreasing or stalling.
The ‘some’ line tells us the percentage of time that some tasks (processes in runnable state) had to wait for the resource (memory on this case).
On the other hand, the ‘full’ refers to the percentage of time that no runnable process at all is able to run due to lack of memory.
This waiting time is due to the processor having to swap pages in and out of the memory.
If the avg numbers were consistently large, we could deduce that the system needs extra memory.
For more information on pressure stall information see this link
Looking into swap memory activity
To look into how much swap memory is used per time interval we can use sar.
The swap activity can be printed with the -W flag. Let set the interval to 1 sec.
sar -W 1
16:10:02 pswpin/s pswpout/s
16:10:03 0.00 0.00
16:10:04 0.00 0.00
16:10:05 0.00 0.00
16:10:06 0.00 0.00
16:10:07 0.00 0.00
16:10:08 0.00 0.00
16:10:09 0.00 0.00
16:10:10 0.00 0.00
16:10:11 0.00 0.00
16:10:12 0.00 0.00
Average: 0.00 0.00
sar outputs two columns:
- pswpin/s: number of pages in to swap memory
- pswpout/s: number of pages out of swap memory
The line which print the average, shows that there is no swap memory in usage.
Putting everything together
It seems that the system was under pressure at some point and required a large amount of swap. However, at the time of running sar, there was actually not much going on.
This correlates with the user experience. Since at the time of running sar, I was just using the text editor and the terminal. While recently I have run memory intensive processes