There has been a lot of discussions on how to configure SSL-enabled OpenAM servers to communicate with one another via self-signed certificates.
Actually, honestly speaking, it's not an OpenAM issue. It's a JDK key-store issue. One needs to understand how SSL works. As mentioned in my blog some times back, I found this link a good start.
These days, I am using SSLPoke pretty often. This is the most wonderful tool to have. I'll make sure SSLPoke pass before I continue to configure the 2nd and subsequent OpenAM server.
Or you can set the following JVM-option:
This will show why SSL handshake fails.