It can be quite a learning experience for me. In fact, I find each tuning exercise to be a new experience to me, always.
This week, we have the same expert flying in from Korea to help us again. This makes the tuning exercise more fruitful. Over time, I have learnt few tips from him.
From the graph below:
1. Spikes are not good
2. Spikes are especially bad if all transactions spike at the same time
However, the above issue is easiest to solve since it implies 2 things might have gone wrong:
- OS Kernel and/or NDD settings
- Garage Collection in the JVM
If the OS kernel and NDD settings are properly tuned, you'll most likely get the graph below:
Generally, most transactions should be stable (see the blue arrow). There will be occasional spikes for some transactions, which some times we can choose to ignore if the occurrence is not frequent.
After the OS kernel and NDD settings are tuned, the next step is to slowly tweak the JVM options to eliminate/lessen garage collection.
Lesson learnt is to have a lot of patient. :)