Reinitialization for 500k+ records took too long. Customer could not wait. So I did a quick LDIF export (export-ldif) from a good directory server and imported into the out-of-sync directory server. That was much faster - around 5 mins.
After the incident, I continued to monitor the 4 directory servers replication status.
a1.servers.XXX.com.sg:4444 : 551852
a2.servers.XXX.com.sg:4444 : 551853
a3.servers.XXX.com.sg:4444 : 551851
a4.servers.XXX.com.sg:4444 : 551853
Hmm... still 1-2 entries not catching up. But exactly which entries are not the same among the 4 directory servers? Which directory server should I trust as master now, if I decide to re-sync all 4 OpenDJ again?
I have no idea until I discussed my issue with my colleague. He was reading up on CA Directory just yesterday as he was installing CA SiteMinder (which uses CA Directory as the configuration store) for a customer of ours.
He told me there is a utility which I can use - ldifdelta.
Use the ldifdelta tool to calculate the change, or delta, between two LDIF files. The ldifdelta program is an offline directory synchronization tool based on the LDAP directory interchange format. You can use ldifdelta to fully or partially synchronize directories.
Bingo! Exactly what I need.