Porting embedded OpenDJ in OpenAM to external OpenDJ - Part II

In my previous blog on Porting embedded OpenDJ in OpenAM to external OpenDJ, I mentioned that the bootstrap file will be automatically updated whenever there is a change in Directory Configuration tab.

Well... it actually happened to my test labs which is using Tomcat for OpenAM deployment. I do not see it happening in an environment that is utilizing JBoss Application Server in my customer's site.

A restart of JBoss will update the bootstrap file. However, the OpenAM is still pointing to the old directory server (since it was booted with the old bootstrap file). Another restart of JBoss will do the trick!

.

Solaris EOL Dates

Some of our customers are still running Solaris 10 (yes, a lot more have already migrated out to RHEL).

.

Inbound closed before receiving peer's close notify: possible truncation attack?

I was just trying to close a OpenDJ Control Panel gracefully by clicking on the EXIT from the menu, but was very surprised when the access log displayed the following:

 Why? Pretty misleading...

.

Porting embedded OpenDJ in OpenAM to external OpenDJ

This week, my team is tasked to migrate out all embedded configuration data store (OpenDJ) to a pair of external OpenDJ configured with multi-master replication (MMR).

How do we go about migrating from embedded to external OpenDJ?


Step 1: Add a new instance of Directory Server to Directory Configuration

You'll notice that the bootstrap file will be automatically updated.

Step 2: Remove the old instance of Directory Server from Directory Configuration

The old entry in the bootstrap file will be automatically deleted.

Step 3: Export the data in Embedded OpenDJ into a LDIF file

[azlabs@cdemo ~]$ apache-tomcat-7.0.35/bin/shutdown.sh  (this will shut down OpenDJ since it is embedded into OpenAM)


[azlabs@cdemo ~]$ cd /home/azlabs/opensso/opends/bin
[azlabs@cdemo bin]$ ./export-ldif --includeBranch dc=opensso,dc=java,dc=net --backendID userRoot --ldifFile /home/azlabs/embed.ldif
:
[22/May/2013:13:39:31 +0800] category=JEB severity=NOTICE msgID=8847447 msg=Exported 417 entries and skipped 0 in 0 seconds (average rate 866.9/sec)



Step 4: Configure External OpenDJ for OpenAM Schema

[azlabs@cdemo bin]$ cd /home/azlabs/OpenDJ-2.4.6/bin

[azlabs@cdemo bin]$ ./stop-ds

Stopping Server...

[azlabs@cdemo ~]$ tar -cvf OpenDJ-2.4.6.CLEAN.tar OpenDJ-2.4.6/* (never be sorry! do a backup first.)

[azlabs@cdemo ~]$ cd /home/azlabs/OpenDJ-2.4.6/config

[azlabs@cdemo config]$ cp config.ldif config.ldif.CLEAN  (never be sorry! do a backup first.)

[azlabs@cdemo config]$ vi config.ldif

  ds-cfg-single-structural-objectclass-behavior: warn

  ds-cfg-allow-pre-encoded-passwords: true

[azlabs@cdemo ~]$ cd /home/azlabs/OpenDJ-2.4.6/config/schema

[azlabs@cdemo schema]$ cp /home/azlabs/opensso/opends/config/schema/99-user.ldif .

Step 5: Import data into External OpenDJ from LDIF file

[azlabs@cdemo ~]$ cd /home/azlabs/OpenDJ-2.4.6/bin

[azlabs@cdemo bin]$ ./import-ldif --includeBranch dc=opensso,dc=java,dc=net --backendID userRoot --ldifFile /home/azlabs/embed.ldif

:

:

[22/May/2013:13:56:42 +0800] category=JEB severity=NOTICE msgID=8847454 msg=Processed 417 entries, imported 417, skipped 0, rejected 0 and migrated 0 in 3 seconds (average rate 134.0/sec)

[22/May/2013:13:56:42 +0800] category=JEB severity=NOTICE msgID=8847536 msg=Import LDIF environment close took 0 seconds

Step 6: Start External OpenDJ

[azlabs@cdemo bin]$ ./start-ds

:

[22/May/2013:13:48:56 +0800] category=CORE severity=NOTICE msgID=458887 msg=The Directory Server has started successfully

Step 7: Start OpenAM

[azlabs@cdemo ~]$ apache-tomcat-7.0.35/bin/startup.sh



Step 8: Reconfigure OpenAM

Access Control > / (Top Level Realm) > Authentication > LDAP

Access Control > Services > Policy Configuration

Access Control > Data Stores > embedded (or you can remove this and create a new data store)

Step 9: Restart OpenAM

[azlabs@cdemo ~]$ apache-tomcat-7.0.35/bin/shutdown.sh

[azlabs@cdemo ~]$ apache-tomcat-7.0.35/bin/startup.sh

Cool!


PS: Of course, you need to set ds-cfg-allow-pre-encoded-passwords back to false again for better security.

I'm tempted to remove the embedded OpenDJ binary from the OpenAM totally, as I keep seeing the following in the embedded OpenDJ access log during OpenAM start-up:


[22/May/2013:14:25:46 +0800] SEARCH REQ conn=1 op=3 msgID=4 base="cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config" scope=baseObject filter="(objectClass=*)" attrs="1.1"
[22/May/2013:14:25:46 +0800] SEARCH RES conn=1 op=3 msgID=4 result=0 nentries=1 etime=3
[22/May/2013:14:25:46 +0800] SEARCH REQ conn=1 op=4 msgID=5 base="cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config" scope=baseObject filter="(objectClass=*)" attrs="objectclass"
[22/May/2013:14:25:46 +0800] SEARCH RES conn=1 op=4 msgID=5 result=0 nentries=1 etime=7
[22/May/2013:14:25:46 +0800] SEARCH REQ conn=1 op=5 msgID=6 base="cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config" scope=baseObject filter="(objectClass=*)" attrs="ds-cfg-enabled,ds-cfg-java-class,ds-cfg-num-update-replay-threads"

But I do not know whether or not removing the "opends" directory will bomb OpenAM. Will try when I have the time.


.

AM Session Failover Statistics

I blogged some time back that there is a Statistics for AM Session Failover. That statistics is actually coming from Sun GlassFish(tm) Message Queue 4.4 perspective. 

We know there is a layer (AMSFO) on top of the Sun GlassFish(tm) Message Queue 4.4. How about some statistics from this layer?

Use the "-i"!

[cheechong@cdemo bin]$ diff amsfo amsfo.BAK.20130517
171c171
<         $AMEXECUTABLE -a $CLUSTER_LIST -u $USER_NAME -f $PASSWORDFILE -b $DATABASE_DIR $AMSESSIONDB_ARGS -m $CONF_FILE -i 1 >> $LOG_DIR/amsessiondb.log &
---
>         $AMEXECUTABLE -a $CLUSTER_LIST -u $USER_NAME -f $PASSWORDFILE -b $DATABASE_DIR $AMSESSIONDB_ARGS -m $CONF_FILE >> $LOG_DIR/amsessiondb.log &amp;

.

Server Connection Closed

I blogged about Connection issue from OpenAM to MS Active Directory some time back. The issue was not able to be reproduced in each and every MS Active Directory, I said.

I just came back from a customer's site in Thailand. They just switched to a new MS Active Directory for their development environment. And right away, they encountered the same issue!!

From the Authentication log, this is what you will observed:


amAuthAD:05/17/2013 10:54:14:922 AM ICT: Thread[ajp-172.19.194.75-8009-11,5,jboss]
WARNING: Search for User error:
org.forgerock.opendj.ldap.ErrorResultIOException: org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed:
        at org.forgerock.opendj.ldif.ConnectionEntryReader.hasNext(ConnectionEntryReader.java:278)
        at com.sun.identity.authentication.modules.ldap.LDAPAuthUtils.searchForUser(LDAPAuthUtils.java:883)
        at com.sun.identity.authentication.modules.ldap.LDAPAuthUtils.authenticateUser(LDAPAuthUtils.java:466)
        at com.sun.identity.authentication.modules.ldap.LDAP.process(LDAP.java:526)
        at com.sun.identity.authentication.spi.AMLoginModule.wrapProcess(AMLoginModule.java:998)
        at com.sun.identity.authentication.spi.AMLoginModule.login(AMLoginModule.java:1103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.identity.authentication.jaas.LoginContext.invoke(LoginContext.java:208)

.

Secure Cookie

In Security Best Practices when deploying OpenAM, there is this 1-liner tip: "When using https use secure cookies ."

What exactly is "when using https"?

The documentation states the following:



Secure Cookie 
If yes, then OpenAM sets the cookie in secure mode such that the browser only returns the cookie if a secure protocol such as HTTPS is used. 
Default: No property: com.iplanet.am.cookie.secure

To be more precise, assume a Web Policy Agent is installed on a Apache Server (HTTPS is not enabled) where a protected application is installed. And assume Secure Cookie is enabled on OpenAM server (web container is running HTTPS).

1. A user attempts to access the protected application (residing on Apache Server).
2. The access is "intercepted" by the Web Policy Agent.
3. There is no existing session found. The Web Policy Agent sends redirection to user's browser.
4. The user's browser is being redirected to OpenAM server for user authentication
5. User keys in valid user name and password.
6. OpenAM server now generates a cookie in secured mode. A "goto" redirection takes place.
7. The user's browser now contains the secure cookie sent from OpenAM server
8. Now, the user's browser is about to perform a redirection back to the protected application.
9. But, BOMB!

Why?

In secure cookie mode, the browser is instructed not to send the cookie to Apache server, since it is not running over HTTPS. You will only be able to use this setting if Apache server is also set up to use HTTPS.


Conclusion

In order to enable secure cookie, the OpenAM server has to be running on HTTPS protocol first. Then, the web containers of all protected applications have to be running on HTTPS protocol as well.

How often do you see this kind of deployment? Seldom.

.

Session cookie

The following is an extract from OpenAM mailing list which I think is very useful in understanding the structure of OpenAM Session Cookie.

> The session cookie is made up from two parts:
>
> Session ID
> Server Identifier
>
> The Session ID
>
> The session ID is a SHA1-PRNG generated secure hash. This
> hash references the session on the authoritative server. 
> In order to forge the cookie they would need to guess a 
> PRNG that matched exactly to a valid session with the same
> hash on a server. This would be next to impossible, it 
> would be easier to try and steal the cookie value rather
> than try and generate a valid one from scratch.
>
> The Server Identifier varies depending on if the server 
> is in a site and if the site is running session failover. 
> The whole server identifier is base64 encoded. It is made 
> up like this
>
> S101|SI10|SK34329478
>
> S1 is server instance, the server where the session resides
> SI is the Site Identifier, the site where the server resides
> SK is the storage key, used during session failover.

.

Securing OpenAM

There are guides describing how to secure OpenAM. Read Security Best Practices when deploying OpenAM and Securing OpenAM.

Both guides talk about URIs Restriction:

• Restrict access to URIs that you do not use, and prevent internal endpoints such as /sessionservice from being reachable over the Internet. 
• Restrict access to URIs not used by your use-cases: internal endpoints SHOULD NOT be reachable from the Internet (/sessionservice and accompanies).

What exactly is "such as /sessionservice"? What exactly is "/sessionservice and accompanies"?

Don't you feel annoyed when you read such hardening guide? Which endpoint(s) are to be restricted?

Luckily, there is this Security Advisory from ForgeRock.

Vulnerable endpoints: 
/openam/authservice
/openam/loggingservice
/openam/namingservice
/openam/policyservice
/openam/profileservice
/openam/sessionservice

How do we go about restricting the endpoints in Apache HTTPD server if it is deployed as a reverse proxy to the OpenAM server?

The above only allow access to the endpoints from a fixed range of internal IP address.

If we want to further harden by disallowing REST APIs calls, we can add "identity" to LocationMatch.


Nice.

Session Activation Failed

I must admit I have not seen this error message "Session Activation Failed" before I encountered it yesterday.

How did it happened?

Background
In my customer's site, we have a couple of sites. Within each site, we have multiple OpenAM servers and we have AM Session Failover configured.

Now, each OpenAM server stores its configuration data in the embedded OpenDJ server.

In total, we have 8 OpenAM servers. Thus, we have 8 corresponding embedded OpenDJ servers. And we know that Multi-Master Replication (MMR) is auto-magically configured during installation. (It is automatically taken care of by the OpenAM installation and configuration script)


OpenAM 6 running port 8180 - Server ID 09
InternetSSO site - Server ID 10

OpenAM 7 running port 8180 - Server ID 11

OpenAM 8 running port 8180 - Server ID 12

What was encountered?
When the 3 final OpenAM servers for Internet was added and InternetSSO site was created, all OpenAM servers were restarted one after another.

When I performed a log-in via the load-balancer (https://www.abc.com/openam/UI/Login), I would intermittently encountered "Session Activation Failed" error!

So, I shut down OpenAM 6, 7 and 8. And started them one at a time. I realized the error came from OpenAM 6 and 7. Everything was OK with OpenAM 8.

This is super strange!

Analysis
This took me quite a while and it was my colleague who found out the root cause. By the way, the logging level has to be switched to MESSAGE. Otherwise, there's no way to detect the problem.

On OpenAM 6 server, the Session debug log was showing :

amSession:05/15/2013 12:18:53:118 PM ICT: Thread[main,5,jboss]

UserName=amuser : clusterServerList=09 : connectionMaxWaitTime=5000 :jdbcDriverClass=com.iplanet.dpro.session.jdbc.HADBConnectionImpl : jdcbURL=172.2.3.4:7676,172.2.3.3:7676 : minPoolSize=8 : maxPoolSize=32

amSession:05/15/2013 01:34:11:710 PM ICT: Thread[main,5,jboss]

UserName=amuser : clusterServerList=09 11 : connectionMaxWaitTime=5000 :jdbcDriverClass=com.iplanet.dpro.session.jdbc.HADBConnectionImpl : jdcbURL=172.2.3.4:7676,172.2.3.3:7676 : minPoolSize=8 : maxPoolSize=32

amSession:05/15/2013 02:45:42:465 PM ICT: Thread[main,5,jboss]

UserName=amuser : clusterServerList=09 11 : connectionMaxWaitTime=5000 :jdbcDriverClass=com.iplanet.dpro.session.jdbc.HADBConnectionImpl : jdcbURL=172.2.3.4:7676,172.2.3.3:7676 : minPoolSize=8 : maxPoolSize=32

The same is observed on OpenAM 7 server.


However, on OpenAM 8 server (the good behaving server), the Session debug log was showing: 

amSession:05/15/2013 12:19:58:447 PM ICT: Thread[main,5,jboss]

UserName=amuser : clusterServerList=09 11 12 : connectionMaxWaitTime=5000 :jdbcDriverClass=com.iplanet.dpro.session.jdbc.HADBConnectionImpl : jdcbURL=172.2.3.4:7676,172.2.3.3:7676 : minPoolSize=8 : maxPoolSize=32

Something is really strange. The configuration data on OpenAM 6 and 7 did not seem to be in-sync with OpenAM 8. But wasn't MMR auto-magically configured?


Resolution

A few more restart of OpenAM 6 and 7 did the trick!

The Session debug logs were finally showing the correct clusterServerList!!

UserName=amuser : clusterServerList=09 11 12 : connectionMaxWaitTime=5000 :jdbcDriverClass=com.iplanet.dpro.session.jdbc.HADBConnectionImpl : jdcbURL=172.2.3.4:7676,172.2.3.3:7676 : minPoolSize=8 : maxPoolSize=32

Finally the configuration data were in-sync.


Luckily, we are moving away from embedded OpenDJ servers next week. We'll have 2 dedicated standalone OpenDJ servers (with MMR configured) to serve the 8 OpenAM servers (and more to be added few months down the road).

*phew*

Updated on 27th May 2013:
Here is a good tips on syncing all embedded OpenDJ servers.


.

SFO is not supported across sites

"SFO is not supported across sites" - What does this statement mean? (Read here)

I get to personally experienced it this week while configuring AM Session Failover (on 2 different sites) for a customer of mine.

Theoretically, each site that is participating in SFO will need a Message Queue Broker Cluster where a minimal of 2 message queue servers are installed.

However, I was plain lazy and was trying my luck. In "Secondary Configuration Instance", I configured the same configuration for both InternetSSO and IntranetSSO instances.

In addition, we have quota constraints enabled and the session quota exhaustion behavior is set to DESTROY_OLD_SESSIONS.

Lastly, the Active User Sessions count is set to 1.

So, what will happen when a user has already logged in to Intranet site and attempt to log in to Internet site?

1. The quota constraint kicks into effect (1 active user allowed at any single point of time).
2. However, the session quota exhaustion behavior will not kick into effect (OpenAM is not able to destroy the old session).


Why?

amSession:05/15/2013 03:24:13:315 PM ICT: Thread[ajp-172.19.176.115-8109-3,5,jboss]

Failed to destroy the next expiring session.

com.iplanet.dpro.session.SessionException: java.lang.NullPointerException

at com.iplanet.dpro.session.Session.refresh(Session.java:1437)

at com.iplanet.dpro.session.Session.getSession(Session.java:1097)

at org.forgerock.openam.session.service.DestroyAllAction.action(DestroyAllAction.java:57)

at com.iplanet.dpro.session.service.SessionConstraint.checkQuotaAndPerformAction(SessionConstraint.java:183)

at com.iplanet.dpro.session.service.InternalSession.activate(InternalSession.java:1053)

at com.sun.identity.authentication.service.LoginState.activateSession(LoginState.java:1193)

at com.sun.identity.authentication.service.AMLoginContext.runLogin(AMLoginContext.java:611)

:

:

Caused by: java.lang.NullPointerException

at com.iplanet.dpro.session.service.ClusterStateService.checkServerUp(ClusterStateService.java:321)

at com.iplanet.dpro.session.service.ClusterStateService.checkServerUp(ClusterStateService.java:220)

at com.iplanet.dpro.session.service.SessionService.checkServerUp(SessionService.java:2081)

at com.iplanet.dpro.session.Session.getSessionResponse(Session.java:1688)

at com.iplanet.dpro.session.Session.doRefresh(Session.java:1454)

at com.iplanet.dpro.session.Session.access$300(Session.java:113)

at com.iplanet.dpro.session.Session$3.run(Session.java:1426)

at com.sun.identity.session.util.RestrictedTokenContext.doUsing(RestrictedTokenContext.java:86)

at com.iplanet.dpro.session.Session.refresh(Session.java:1423)

:

:

Caused by: com.iplanet.dpro.session.SessionException: Session state is invalid. AQIC5wM2LY4Sfcx9Tij6-6PzKZBBdJNzYNJpZPkutzgpdMk.*AAJTSQACMTAAAlNLAAstMjA3NzY4NTU4OAACUzEAAjA5*

at com.iplanet.dpro.session.service.SessionService.checkSession(SessionService.java:1267)

at com.iplanet.dpro.session.service.SessionService.getSessionInfo(SessionService.java:1226)

at com.iplanet.dpro.session.Session.doRefresh(Session.java:1450)

at com.iplanet.dpro.session.Session.access$300(Session.java:113)

at com.iplanet.dpro.session.Session$3.run(Session.java:1426)

at com.sun.identity.session.util.RestrictedTokenContext.doUsing(RestrictedTokenContext.java:86)

at com.iplanet.dpro.session.Session.refresh(Session.java:1423)

Simply put - The session object belongs to Intranet Site. When DESTROY_OLD_SESSIONS action is executed, OpenAM is not able to destroy it in Internet Site. 

.