High Swap Usage On Oracle Database Server

Situation:

When investigating one client’s Oracle database server high swap memory usage, we find the swap space usage is constantly very high on this Linux server, which causes Oracle database a big performance issue.

OS: RHEL 7.3
DB: Oracle 12.2.0.1

Findings:

1)top

Tasks: 352 total, 2 running, 350 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.4%us, 4.1%sy, 0.0%ni, 79.3%id, 2.2%wa, 0.3%hi, 0.8%si, 0.0%st
Mem: 32172820k total, 32015956k used, 156864k free, 14528k buffers
Swap: 16777208k total, 7435428k used, 9341780k free, 11129844k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
137049 oracle 15 0 16.2g 5.3g 5.3g S 20.6 17.4 10:14.17 oracle
 72457 oracle 15 0 16.2g 4.7g 4.7g S 15.3 15.3 10:50.15 oracle
...
..
.

2) pmap

$ pmap -x 137049
137049: oracleRACTEST1 (LOCAL=NO)
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 96356 11704 0 r-x-- oracle
0000000006419000 444 140 4 rwx-- oracle
0000000006488000 148 100 80 rwx-- [ anon ]
000000001966e000 532 176 92 rwx-- [ anon ]
0000000060000000 16779264 5444888 1485768 rwxs- [ shmid=0x670005 ]
00000032b6a00000 112 108 0 r-x-- ld-2.5.so
00000032b6c1c000 4 0 0 r-x-- ld-2.5.so
...
..
.
00007fff4c504000 160 136 132 rwx-- [ stack ]
00007fff4c5d2000 12 4 0 r-x-- [ anon ]
ffffffffff600000 8192 0 0 ----- [ anon ]
---------------- ------ ------ ------
total kB 16954204 5477952 1490448

3) swappiness

$ cat /proc/sys/vm/swappiness
10

4) /proc/meminfo

$ cat /proc/meminfo
MemTotal: 32172820 kB
MemFree: 158096 kB
Buffers: 12512 kB
Cached: 11124148 kB
SwapCached: 878556 kB
Active: 11885576 kB
Inactive: 376796 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 32172820 kB
LowFree: 158096 kB
SwapTotal: 16777208 kB
SwapFree: 9336848 kB
Dirty: 136 kB
Writeback: 0 kB
AnonPages: 487212 kB
Mapped: 11089784 kB
Slab: 104984 kB
PageTables: 2779136 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 24475008 kB
Committed_AS: 20456368 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 271188 kB
VmallocChunk: 34359466743 kB
HugePages_Total: 8192
HugePages_Free: 8192
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

Causes:

The huge pages are not used by Oracle database server, because the number of hugepages set-up is too small.

Solution:

1)Increase the number of  hugepages.

$ cat /etc/sysctl.conf|grep -i hugepages
vm.nr_hugepages = 8200

2) Reboot the server.

3) Check the results.

$ cat /proc/meminfo
...
..
.
SwapTotal: 16777208 kB
SwapFree:  16777208 kB
Dirty:          116 kB
Writeback:        8 kB
AnonPages:  1076372 kB
Mapped:      108400 kB
Slab:        132980 kB
PageTables:   37540 kB
NFS_Unstable:     0 kB
Bounce:           0 kB
CommitLimit: 24466816 kB
Committed_AS: 2249980 kB
VmallocTotal: 34359738367 kB
VmallocUsed:       271188 kB
VmallocChunk: 34359466743 kB
HugePages_Total: 8200
HugePages_Free: 26
HugePages_Rsvd: 19
Hugepagesize:        2048 kB


$ ipcs -ma
------ Shared Memory Segments --------
key          shmid owner perms bytes nattch status
0x00000000 5275649 root    644    80 2
0x00000000 5308419 root    644 16384 2
0x00000000 5341188 root    644   280 2
0xbb94f3d4 5373957 oracle 640 17181966336 87

------ Semaphore Arrays --------
key semid owner perms nsems
0x17a558e8 131073 oracle 640 250
...
..
.


--
-- top 
--

top - 15:15:40 up 20:13, 7 users, load average: 1.13, 0.64, 0.50
Tasks: 289 total, 3 running, 286 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.9%us, 1.4%sy, 0.0%ni, 63.0%id, 23.3%wa, 0.4%hi, 1.0%si, 0.0%st
Mem: 32172820k  total, 19801684k used, 12371136k free, 825948k buffers
Swap: 16777208k total,        0k used, 16777208k free, 931716k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 97575 oracle 16 0 16.1g 31m 25m R 16.6 0.1 0:11.88 oracle
 97973 oracle 17 0 16.1g 28m 21m D 8.6 0.1 0:01.61 oracle
...
..
..


$ pmap -x 97575
97575: oracleRACTEST1 (LOCAL=NO)
Address          Kbytes RSS Dirty Mode Mapping
0000000000400000 96356 17880    0 r-x-- oracle
0000000006419000   444   216   56 rwx-- oracle
0000000006488000   148   116  116 rwx-- [ anon ]
000000000a80a000   532   428  428 rwx-- [ anon ]
0000000060000000 16779264  0    0 rwxs- 1 (deleted)
00000032b6a00000   112   108    0 r-x-- ld-2.5.so
00000032b6c1c000    4      4    4 r-x-- ld-2.5.so
...
..
.
ffffffffff600000 8192 0 0 ----- [ anon ]
---------------- ------ ------ ------
total kB        16948952 42024 17284


$ sar 1 10
Linux 2.6.18-408.el5 (racnode1) 08/02/2017

03:18:48 PM CPU %user %nice %system %iowait %steal %idle
03:18:49 PM all 5.81 0.00 1.52 1.26 0.00 91.41
03:18:50 PM all 4.77 0.00 1.76 2.01 0.00 91.46
03:18:51 PM all 3.54 0.00 1.26 1.01 0.00 94.19
03:18:52 PM all 2.27 0.00 0.51 1.52 0.00 95.71
03:18:53 PM all 0.25 0.00 0.50 0.50 0.00 98.74
03:18:54 PM all 5.54 0.00 1.01 1.01 0.00 92.44
03:18:55 PM all 2.78 0.00 0.51 0.00 0.00 96.72
03:18:56 PM all 1.01 0.00 0.25 0.25 0.00 98.49
03:18:57 PM all 1.01 0.00 0.25 0.51 0.00 98.23
03:18:58 PM all 2.76 0.00 1.26 0.50 0.00 95.48
Average: all 2.97 0.00 0.88 0.86 0.00 95.29

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r b swpd free       buff cache     si so bi   bo in cs us sy id wa st
 0 0    0 12377776 827836 932156     0 0 1259 387  86 73 5 1 90 5 0
 0 0    0 12377776 827856 932140     0 0   15 398 3028 2395 3 2 95 1 0
 3 0    0 12373368 827856 932160     0 0  231 270 2065 2786 21 16 63 0 0
 0 0    0 12369324 827864 932140     0 0    0 168 1124 676 17 1 82 1 0
 0 0    0 12369324 827868 932160     0 0   40 112 1311 754 1 1 98 1 0
Advertisements
This entry was posted in Linux, Performance Tunning and tagged , , , , , , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s