Porting embedded OpenDJ in OpenAM to external OpenDJ - Part III

I almost went to hell last week during a migration exercise where we moved the embedded OpenDJ servers to a pair of external OpenDJ servers in a production site for one of our customers.

In fact, I had previously verified that the migration is risk-free in our test labs. I was too confident. I forgotten the fact that there are multiple OpenAM servers deployed in customer's production environment. And thus, there are multiple embedded OpenDJ servers as well.

I tested on a standalone OpenAM with an embedded OpenDJ. Serve me right!

So, what went wrong?

In fact, as per what I have previously blog here (Part I) and here (Part II), everything worked per planned. I restarted JBoss application servers twice and the bootstrap files were updated accordingly. (The bootstrap files should be updated to point to the external OpenDJ servers, instead of the embedded ones)

Upon JBoss AS restart, I was able to see a bind message in the external OpenDJ servers. But that was it! When I log-in to the OpenAM console and navigated from tabs to tabs, the OpenDJ access logs were not moving. How could that be? I knew configuration data was stored in OpenDJ servers, and thus when the tabs were accessed, OpenAM must retrieved the appropriate information from the external OpenDJ.

And to my horror, when I debugged further, I saw that the OpenAM servers were still accessing information from the embedded OpenDJ servers. The replication between the multiple embedded OpenDJ servers was still functioning. 

I tried to add/modify some policies via the OpenAM console, and I saw that the data was written to the embedded OpenDJ servers! The external OpenDJ servers were a pair of white elephants!

The whole debugging process took a long while. In fact, we were working 1-2 hours beyond the planned schedule. 

In the end, we found the culprit to be a parameter in the Advanced tab - com.sun.identity.sm.sms_object_class_name.

By default, the value com.sun.identity.sm.ldap.SMSEmbeddedLdapObject comes pre-set in OpenAM release.

This value has to be changed to com.sun.identity.sm.ldap.SMSLdapObject in order for the multiple OpenAM servers to start communicating with the external OpenDJ servers. Not only bind access, but all types of LDAP operations.

Really stressful night. Stupid me for not testing in a multi-servers environment prior to the actual migration.

.

PS: Now, the weird thing is I still have the standalone OpenAM server running in my labs and it is able to communicate with the external OpenDJ server, even though the value com.sun.identity.sm.ldap.SMSEmbeddedLdapObject has not been modified. I also confirmed both OpenAM (our test labs and our customer's site) are running on OpenAM 10.0.0.

Why? I do not know. Strange!


.