OpenSSO with embedded OpenDS for high-volume traffic - Part II

I managed to capture 2 graphs taken during the load-testing sessions.

With embedded OpenDS in a 12-nodes OpenSSO environment, there will be consistent spikes happening at 15-20 mins interval and subsequently 25 mins ... (that was when I issued a thread dump and realized there were numerous threads related to OpenDS replication service)

Ideally, we would prefer the following graph. Consistently flat. Spikes at 15-min and 26-mins though, but minor enough to ignore. Still trying to further tune.

PS: Getting the almost flat graph above is not totally contributed by moving the embedded OpenDS to external Sun Directory Server. There were some other tuning made. 

.

OpenSSO with embedded OpenDS for high-volume traffic

I was performing another round of performance load-testing for a customer in Singapore last week. The target was to achieve 10,000 concurrent hit per second (yes, it is a tall order) and the load must be sustained for 1 hour.

The setup was a pretty old version OpenSSO 8 Update 2 Patch 4. The embedded OpenDS was even older. It was still OpenDS 1.0.0. Why no upgrade? Well, better dun ask. Political lah.


We have, in fact, performed many rounds of tuning - OS kernel, TCP stack, Web Servers (reverse proxy), Application Servers, JVM options - prior to last week's activity.

We were almost there, except failing for 1-2 secs at times. Spikes kept coming after 15-20 mins; went away; came back again 10-15 mins later ...


So during one of the spikes, I executed a thread dump. And the following was observed:

At that point of time, we had 12 x OpenSSO servers running, with embedded OpenDS. Me bad, I know. Lazy bum me!

No choice, I made a decision to switch out from 12 x embedded OpenDS to 3 x external Sun Directory Server 7. Political again, dun ask me why you never this.. never that.. :)

Immediately, we could see the effect. The NLWP (Number of Light-Weight Processes) decreased from high-300 to mid-200 upon application servers restart. Pretty good improvement.

Subsequent load-testings also yield better result with less spikes. The NLWP never increased beyond 650 at peak (previously it could easily reached 900+).

At least, the graphs looked flatter.

We are now on our last mile... I think the reverse-proxy servers are over-loaded. Adding more hardware at the moment... Hopefully, we can achieve the expected result in a week or 2!



PS: My friends at ForgeRock also confirm that a deployment of  >4 OpenAM instances with external OpenDJ as the configuration store yields better performance. It is largely due to the meshed replication setup. Also, using the latest version of OpenDJ (2.6.0 at the moment) will help as well. OpenDS is far too old. :)


.

Splunk SSO with OpenAM

I have just integrated SSO between OpenAM and Splunk in our intranet. I must admit this is the simplest integration I have performed so far. 

 

I took references from the following 2 links by Indira.

1. Splunkweb SSO

2. SSO Story: Splunkweb and OpenAM/OpenSSO

(Note: The instruction was not that clear though. A little tweaking is required. Minor though)

What really impressed me was the SSO debug page.

It was able to clearly indicate what SSO settings have been configured and what HTTP headers have been passed over from the OpenAM via the Policy Agent.

This makes debugging a breeze! Cool tool!

.

OpenAM with HOTP and OATH and Microsoft SharePoint

If Microsoft SharePoint is integrated with OpenAM, we know that the Password Replay feature is required. 

There are numerous links on how to achieve Single Sign-On between OpenAM and SharePoint server:

0. Enable IIS 7 Basic Authentication & Password Replay Support

1. OpenSSO Sharepoint 2010 Policy Agent Installation (Explained)

2. Support for OWA and SharePoint filters/auth modules

Another thing to note is that Authentication Chaining (e.g. AD followed by OATH) will break the Password Replay support.

This is because the password from the AD module is not carried over to OATH module. And thus the OATH module cannot pass over the password to the Password Relay Post Authentication Processing module. 

You'll get a HTTP 403: Forbidden error when trying to access a SharePoint page.

There is a workaround though. 

Step 1: Do not use Authentication Chaining

Step 2: In each HOTP and/or OATH module, set the Authentication Level to higher than 0. e.g. 10

Step 3: Create a Condition - Authentication Level in the SharePoint Policies. 

Step 4: Set the Authentication Level to 10.

Now, of course, this implies all policies have to be configured the same.


.

OpenAM with HOTP and OATH

This week, I have a little free time of my own and am able to continue my exploration with OpenAM. I know that HOTP (HMAC-based One Time Password) and OATH (Open Authentication) features are available in OpenAM 10.1-Xpress. 

These 2 features are what customers have always been looking for, especially the financial institutions. It's great that the features are now supported out-of-the-box!

So, I went ahead to implement these new features into our existing OpenAM infrastructure.

All staff has to key in their user name and password first.

Subsequently, they will be challenged again for 2FA. 2 choices are given - HOTP which will fire an email with the OTP code; OATH where staff is to configure Goggle Authenticator on his/her mobile device.

It works like a charm!

.

OpenAM OATH Authentication - java.util.NoSuchElementException

There is a new authentication module in OpenAM 10.1-Xpress - OATH authentication.

And there is a nice wiki which illustrates how to integrate OATH authentication module with Google Authenticator. (Read here) That gets me interested! I could implement this for our company's intranet portal which is protected by OpenAM.

As mentioned in the wiki,

The key attributes we need to set for Google Authenticator are:

• Auth Level.  This is a higher strength multi-factor module, so we assign a value of 10 here. 
• One Time Password Length: This is the length of the OTP that will be displayed by the Google Authenticator application. Six is the default for Authenticator. 
• Minimum secret key length: I used 8 for this example, which is too short for production. This is the length (in hex characters) of the encoded secret. 
• Secret Key Attribute: This is the name of the ldap attribute where the secret key is stored. For this example I am using the "title" attribute. This isn't a great choice, and for production you would extend your ldap schema with a dedicated attribute. 
• OATH Algorithm:  TOTP for Google Authenticator
• Last Login Time Attribute:  The OATH TOTP module needs to store the last login timestamps (UNIX long time) in this attribute.  I am using "description" but again you should extend your schema with a dedicated attribute.

Now, one thing to take note is: besides keying in the Secret Key attribute into the LDAP server ("title" is used in the example), one has to key in a default value for Last Login Time attribute ("description").

Otherwise, you'll hit into the same error as what I had encountered:

amAuthOATH:09/11/2013 03:57:01:432 PM SGT: Thread[ajp-apr-192.168.0.89-8009-exec-10,5,main]
ERROR: OATH.checkOTP() : checkOTP process failed :
java.util.NoSuchElementException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:796)
        at java.util.HashMap$KeyIterator.next(HashMap.java:828)
        at org.forgerock.openam.authentication.modules.oath.OATH.checkOTP(OATH.java:577)
        at org.forgerock.openam.authentication.modules.oath.OATH.process(OATH.java:322)
        at com.sun.identity.authentication.spi.AMLoginModule.wrapProcess(AMLoginModule.java:1000)
        at com.sun.identity.authentication.spi.AMLoginModule.login(AMLoginModule.java:1170)
       :
       :

This will always happen for the initial login after OATH is configured because the attribute is empty. The code could have been better written to take care of this special scenario.

.

Replication Gateway - Part II

I mentioned about Replication Gateway feature from OUD.

Is this the only way to replicate data from Sun Directory Server Enterprise Edition to OpenDJ or OUD?

Definitely no.

There is this free tool called LDAP Synchronization Connector.

Ldap Synchronization Connector synchronize data from any data source including databases, LDAPdirectories or files by reading, transforming and comparing these data between the source and the target referentials. These connectors can then be used to continuously synchronize a data source to a directory, for a one shot import or just to compare differences by outputting CSV or LDIF format reports.

In Azimuth Labs, we have used LSC on production sites when we migrated our customers' data from Sun Directory Server to ForgeRock OpenDJ. It works like a charm!

.

OpenAM Roadmap - Updated

The planned OpenAM 10.2 release has been dropped. Instead, OpenAM 11 will be released in this quarter.

What can one look forward to? The detail is here.

To most of my customers, the following 3 items are their top priorities:

1. OpenAM Server IPv6 Support
2. OpenAM Java 7 Support
3. REST APIs Enhancements

.

ForgeRock Business Model

This question keeps repeating the ForgeRock user forum. I think I might as well paste it here in case my customers ask me the same.

To be honest, this is fair model. The engineers also need to feed their families. We can't expect every single piece of source code be made publicly available.

.

OpenDJ 2.6.0 Control Panel

Most of the time when I deployed OpenDJ server and wanted to use OpenDJ Control Panel to connect to it, I'll run the control panel from my laptop or desktop.

And I do not usually install a OpenDJ server on my laptop/desktop because there is really no need to. So, I'll download the OpenDJ ZIP for a particular version of OpenDJ, unzip it, go to the bin or bat folder and run the control-panel executables. And I'm done!

Not with OpenDJ 2.6.0 .... there is no response when I run the control-panel executable.

After figuring it out for a while, I then realized I need to run the setup executable first. Once that's done, the control-panel will then work.

.