Unbreakable (backward-compatible) OpenAM Login Page

I am currently migrating OpenAM 9.5.3 to OpenAM 11.0.2 for a customer based in Singapore. In fact, I migrated Sun Access Manager 7.0 to OpenAM 9.5.3 few years back for the same customer. How time flies? Getting older. :)

The migration is pretty smooth so far … with a little surprise and confused me for a while.

We know the UI for OpenAM 11 looks like the one below:

But after I ported customized codes and JSP pages over, I was constantly redirected to the old OpenAM server! How can it be?!! The below was what I saw...

It took me (and my colleague too) quite a while to realize that we are in fact landing at OpenAM 11 Login Page.


See the differences below? ForgeRock logo and footer are different in OpenAM 11.

Ok lah, from backward-compatible point of view, well done man! Good job!


By the way, I ended last month with a 50-km trail race - The North Face Singapore.

Awesome run!

.

Message:Service does not exist : sunAMAuthApplicationService

This is a super misleading error message in OpenAM Authentication debug log. If you turn on debug level to Message and restart OpenAM, it will appear almost 100% of the time.

And last Saturday, I was at a customer's site which had a farm of OpenSSO 8U1R2 running. I was trying to debug on a node and switch to Message level, and I was seeing the same misleading error!

Wow! This has been there for years! This post is to remind myself to ignore it the next time I see it again.

amAuthConfig:10/18/2014 03:48:25:480 PM SGT: Thread[pool-1-thread-11,5,main]
Error retrieving service schema
Message:Service does not exist : sunAMAuthApplicationService
        at com.sun.identity.sm.ServiceSchemaManagerImpl.isValid(ServiceSchemaManagerImpl.java:135)
        at com.sun.identity.sm.ServiceSchemaManagerImpl.(ServiceSchemaManagerImpl.java:115)
        at com.sun.identity.sm.ServiceSchemaManagerImpl.getInstance(ServiceSchemaManagerImpl.java:589)
        at com.sun.identity.sm.ServiceSchemaManager.(ServiceSchemaManager.java:112)
        at com.sun.identity.sm.ServiceSchemaManager.(ServiceSchemaManager.java:77)
        at com.sun.identity.authentication.config.AMAuthLevelManager.updateGlobalAuthLevelMap(AMAuthLevelManager.java:697)
        at com.sun.identity.authentication.config.AMAuthLevelManager.initialize(AMAuthLevelManager.java:199)
        at com.sun.identity.authentication.config.AMAuthLevelManager.(AMAuthLevelManager.java:97)
        at com.sun.identity.authentication.config.AMAuthLevelManager.getInstance(AMAuthLevelManager.java:109)
        at com.sun.identity.authentication.config.AMConfiguration.addServiceListener(AMConfiguration.java:770)
        at com.sun.identity.authentication.config.AMConfiguration.getModuleBasedConfig(AMConfiguration.java:624)
        at com.sun.identity.authentication.config.AMConfiguration.newConfiguration(AMConfiguration.java:213)
        at com.sun.identity.authentication.config.AMConfiguration.getAppConfigurationEntry(AMConfiguration.java:691)
        at com.sun.identity.authentication.service.AuthUtils.isPureJAASModulePresent(AuthUtils.java:1528)
        at com.sun.identity.authentication.service.AMLoginContext.executeLogin(AMLoginContext.java:315)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:435)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:371)
        at com.sun.identity.authentication.AuthContext.runLogin(AuthContext.java:675)
        at com.sun.identity.authentication.AuthContext.login(AuthContext.java:572)
        at com.sun.identity.authentication.AuthContext.login(AuthContext.java:524)
        at com.sun.identity.authentication.AuthContext.login(AuthContext.java:383)
        at com.sun.identity.security.SystemAppTokenProvider.getAppSSOToken(SystemAppTokenProvider.java:78)
        at com.sun.identity.security.AdminTokenAction.getSSOToken(AdminTokenAction.java:295)
        at com.sun.identity.security.AdminTokenAction.authenticationInitialized(AdminTokenAction.java:143)
        at com.sun.identity.authentication.UI.LoginLogoutMapping.initializeAuth(LoginLogoutMapping.java:103)
        at com.sun.identity.authentication.UI.LoginLogoutMapping.init(LoginLogoutMapping.java:65)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1100)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1023)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4969)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:5376)
        at com.sun.enterprise.web.WebModule.start(WebModule.java:345)
        at com.sun.enterprise.web.LifecycleStarter.doRun(LifecycleStarter.java:58)
        at com.sun.appserv.management.util.misc.RunnableBase.runSync(RunnableBase.java:304)
        at com.sun.appserv.management.util.misc.RunnableBase.run(RunnableBase.java:341)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

.

Open Web Application Security Project - Top 10 for 2013

I came across OWASP Top 10 for 2013 and found it to be informative.

The OWASP Top 10 focuses on identifying the most serious risks for a broad array of organizations. For each of these risks, we provide generic information about likelihood and technical impact using the following simple ratings scheme, which is based on the OWASP Risk Rating Methodology.

So, here's the link to Top 10 for 2013. Quite alarming that Injection and XSS are still on top of the list year in year out.



.

What's the reasons to Move from OpenSSO to OpenAM?

If you still have Sun OpenSSO running and need reasons to move to OpenAM, wait no more … Sign up for the webinar - 100,000 Reasons to Move from OpenSSO to OpenAM …. Steve, with his ever in-depth knowledge of Sun Access Manager, Sun OpenSSO and ForgeRock OpenAM, will explain why!

The most important point he'll touch on is:

ForgeRock provides a very simple upgrade so that you can take your existing Oracle OpenSSO deployment -- now EOL'd with no significant updates since 2010 -- and immediately transform it into a next-generation access management platform.

.

Where to store Password Policy?

Any Single Sign-On product in the market will need to talk to a back-end User Store for authentication purpose. A User Store can be a traditional LDAP server or a modern Microsoft Active Directory Server.

Now, the SSO product fronts the end-users and this is where Password Policy is enforced. (Of course, if a corporate environment is using Microsoft platform, then password policy can be enforced during the Windows logon process)

Today, I talked to a Product Pre-Sales engineer and was pretty pissed off. She knew my team is managing the SSO product, as well as the LDAP server for a customer. We have a migration exercise going on at the moment.

My customer wants us to make use of the out-of-the-box features from the SSO product as much as possible - to the extend of making sure the Password Policy implemented on the LDAP server can be seamlessly integrated with this SSO product.

We spent a few days reading up and setting up a POC in our labs. We confirmed that this SSO product is not able to "interact" with any underlying LDAP server with regard to Password Policy management. It is only able to implement its own password policy.

She insisted that the RIGHT WAY of doing thing is for the SSO product to enforce password policy for the corporate.

This is "vendor-lock-in" strategy. You think customer is stupid?

I have implemented many SSO projects. In Sun's days, when the old Sun Access Manager and OpenSSO server could not integrate seamlessly with Microsoft Active Directory, we did not tell our customers that was the wrong way of enforcing password policy and they should use Sun AM/OpenSSO built-in password policy instead.

We stripped the codes and implemented custom authentication module for our customers.

User Store (LDAP server or Microsoft Active Directory Server) should be the authoritative password policy enforcer.

Wake up, lady!

.

OS Patching and the Impact

I encountered a Priority 1 case in a customer site 2 weeks ago with an OpenSSO deployment for one of our government ministries. 

What really happened was the Single Sign-On infrastructure became unstable after a Solaris Patch Cluster was applied, especially during high-load.

No one realised the issue right after patching. There was even a round of internal testing conducted and the system was given the go-ahead by the administrator.

But on the 1st working day after the patching, the help-desk received numerous calls that authentication was "sometimes OK, sometimes not OK". They thought the farm of OpenSSO servers were restarting one at a time. But no one touched the OpenSSO servers at all.

We observed that CPU consumptions were on the high side and connection timeouts were often encountered.

I was consulted. As I was helping customers with tuning in another environment where our Solaris servers were recently patched and encountered poor performance, I thought it was a good bet to roll back the Solaris Patch Cluster.

As the fire was on the Production side, a decision was made to perform the roll-back on the servers in the Production environment first. That solved the issue!

Root cause? I'll let the Oracle experts tell us since they are paid to service us.

Once the fire was put off, we also subsequently roll-back the patch on our QAT environment and we were back in business.

Lesson learnt is OS Patching does have impact on the performance of the software that is installed on top of it. Do not ignore this fact. Load testing after patching will be ideal if time permits. (well, I would say no one does this 90% of the time. :> )

.

Scaling for Modern Web

I saw this slide in ForgeRock European Summit this week in France.

This is the type of scale OpenAM is built to support the Modern Web. Yes.

How many products out there can claim to perform at the same scale? Of course, please compare apple-to-apple. Play fair and not compare by adding in enormous number of high-end gigantic hardwares.

PS: By the way, 100M users only requires roughly 2,500 AuthN/sec.

What am I doing here?

• OpenSSO with embedded OpenDS for high-volume traffic
• OpenSSO with embedded OpenDS for high-volume traffic - Part II

I already told my customer 10,000 AuthN/sec is almost 99.999% unlikely to happen in their environment. They insisted. *Sigh*

But it again proves that OpenSSO/OpenAM is built to be highly scalable.

.

OpenSSO with embedded OpenDS for high-volume traffic - Part II

I managed to capture 2 graphs taken during the load-testing sessions.

With embedded OpenDS in a 12-nodes OpenSSO environment, there will be consistent spikes happening at 15-20 mins interval and subsequently 25 mins ... (that was when I issued a thread dump and realized there were numerous threads related to OpenDS replication service)

Ideally, we would prefer the following graph. Consistently flat. Spikes at 15-min and 26-mins though, but minor enough to ignore. Still trying to further tune.

PS: Getting the almost flat graph above is not totally contributed by moving the embedded OpenDS to external Sun Directory Server. There were some other tuning made. 

.

OpenSSO with embedded OpenDS for high-volume traffic

I was performing another round of performance load-testing for a customer in Singapore last week. The target was to achieve 10,000 concurrent hit per second (yes, it is a tall order) and the load must be sustained for 1 hour.

The setup was a pretty old version OpenSSO 8 Update 2 Patch 4. The embedded OpenDS was even older. It was still OpenDS 1.0.0. Why no upgrade? Well, better dun ask. Political lah.


We have, in fact, performed many rounds of tuning - OS kernel, TCP stack, Web Servers (reverse proxy), Application Servers, JVM options - prior to last week's activity.

We were almost there, except failing for 1-2 secs at times. Spikes kept coming after 15-20 mins; went away; came back again 10-15 mins later ...


So during one of the spikes, I executed a thread dump. And the following was observed:

At that point of time, we had 12 x OpenSSO servers running, with embedded OpenDS. Me bad, I know. Lazy bum me!

No choice, I made a decision to switch out from 12 x embedded OpenDS to 3 x external Sun Directory Server 7. Political again, dun ask me why you never this.. never that.. :)

Immediately, we could see the effect. The NLWP (Number of Light-Weight Processes) decreased from high-300 to mid-200 upon application servers restart. Pretty good improvement.

Subsequent load-testings also yield better result with less spikes. The NLWP never increased beyond 650 at peak (previously it could easily reached 900+).

At least, the graphs looked flatter.

We are now on our last mile... I think the reverse-proxy servers are over-loaded. Adding more hardware at the moment... Hopefully, we can achieve the expected result in a week or 2!



PS: My friends at ForgeRock also confirm that a deployment of  >4 OpenAM instances with external OpenDJ as the configuration store yields better performance. It is largely due to the meshed replication setup. Also, using the latest version of OpenDJ (2.6.0 at the moment) will help as well. OpenDS is far too old. :)


.

Policy Agent error >> find_active_login_server(): Library not initialized

Sometimes, it can be a pain looking at how the IT staff in customers' sites attempt to debug Policy Agent related issues.

Usually, what I observed is that they will keep on restarting the web containers where the Policy Agents are deployed. Then they will keep hitting the browsers and start to complain "It still cannot work! Why? Why?".

Well, if you are too lazy to take a look at the policy agent debug log, you'll never know why. Some do not even know where is the debug log located. *sigh*



2013-08-13 11:40:15.199   Error 11762:9a41700 PolicyEngine: am_policy_is_notification_enabled: InternalException in PolicyEngine::isNotificationEnabled with error message:Invalid policy handle. and code:invalid argument
2013-08-13 11:40:15.200   Error 11762:9a41700 all: find_active_login_server(): Library not initialized.
2013-08-13 11:40:19.376   Error 11762:9a417e0 PolicyEngine: am_policy_is_notification_enabled: InternalException in PolicyEngine::isNotificationEnabled with error message:Invalid policy handle. and code:invalid argument


The issue today is the Policy Agent is not able to contact OpenSSO server when the web container starts. And the IT staff never even attempt to restart the web container.

Ensure the OpenSSO server is started, then restart the web container where the Policy Agent is deploy will do the trick!


.

Configuring System Failover and Session Failover for High Availability

This chapter (Configuring System Failover and Session Failover for High Availability) in Oracle Documentation site is still a must-read.

.

Cross Domain Single Sign On - What's it?

The concept of why a cookie created specifically for a particular domain cannot be accessed by another domain has been explained repeatedly year over year.

A lot of articles tried to explain, but in words, which can get a little complicating. (ok, maybe to me :>)

The following diagram explains it all. (yes, OpenSSO articles is still the to-go place for reading-up concepts)

Clear and precise. Why talk so much when a simple diagram can do the job? (maybe technical writers just like to write, but doesn't like to draw)


To introduce CDSSO, each SSO product implement its proprietary mechanism to support single sign-on across multiple domains. 


OpenAM / OpenSSO introduced Cross-Domain Controller Servlet (CDCServlet) to handle CDSSO.

SiteMinder introduced Cookie Provider to handle the same scenario.

.

OpenAM Lineage

I have told customers multi-times - OpenAM is not a new product, even though ForgeRock is only 2. In fact, OpenAM is around 13 years old now. The history of OpenAM dates back to early 2000 when Sun Microsystems started this as a Directory Server Access Management Edition (DSAME).

Interestingly, after Oracle took over Sun OpenSSO, the product was "chopped" into several non-compatiable products - each sold differently :).

.

No such Organization found

There was a discussion in OpenAM mailing list few days ago with regard to an error - "No such Organization found". 

When such an error occurs, there are a few possibilities. One of it is a mis-configuration in Realm/DNS Aliases, such that an identical alias is created in different Realm.
 
The following shows an alias "idp.azlabs.sg" being created in Top Level Realm.

The same alias "idp.azlabs.sg" is also being created in Realm "testrealm".

When such a setting is configured, "No such Organization found" error will be displayed when a user or administrator attempts to login via OpenAM login page. As such, you are stuck if you are an administrator! There is no way you can login via the GUI.

How to resolve this issue?

* Use a LDAP browser to explicitly delete the duplicate alias.

The following is the alias which we want to keep:

The following is the duplicate alias which we need to delete:

Delete the entry "sunxmlKeyValue=sunidentityrepositoryservice-sunOrganizationAliases=idp.azlabs.sg" will do the trick! Remember to restart OpenAM server.

By the way, the following error is captured in Authentication debug log - "Multiple mappings found for organization identifier: idp.azlabs.sg".

This is where a lot of people failed to check before posting their questions in the mailing list. If one could look at the Authentication debug log in detail, the root cause is pretty obvious.

.

Password Reset Capability in OpenAM

Why do I like OpenAM that much after all these days (from Sun Access Manager, to Sun OpenSSO, to ForgeRock OpenAM)?

The answer is simple: it's highly flexible.

For automatic generation of a random password after a reset, there is always a default implementation - PasswordGenerator that comes out-of-the-box.

So what if this default implementation does not suit what your customers want? Write your own implementation! That's what I like about it - the flexibility to hook in my own plugin.

For password reset notification, the default implementation - NotifyPassword uses email medium to alert users.

If your customers require a SMS notification, what do you do? Write your own implementation! That's what I like about it. :)

Well, maybe not for those who do not like to code though. :>

.

How to enable old password prompt?

I'm preparing a POC for a customer in Thailand to showcase the capability of OpenAM integrated with their Microsoft Active Directory.

One of the requirement is to be able to change password, but the user must be prompted to enter his old password first.

The default Change Password UI from OpenAM has the filed greyed out.

To enable old password prompt, go to Configuration -> Console -> Administration. Enable "Prompt user for old password".

It's enabled now. Nice!

.

LDAP Error 53: The LDAP server is unable to perform the specific operation

I was trying to configure a AD Data Store. 

It was pretty straight-forward to get the AD Data Store configured. If configured properly, all AD users will be displayed according in the Subjects tab.

So, I went ahead to create a new user. (OK, just for testing purpose. I have never used OpenAM to provision users in production before. There are far better tools in doing the same.)

BOMB! I received "LDAP Error 53: The LDAP server is unable to perform the specific operation" when I clicked OK.

What could have gone wrong?

LDAPv3Repo: Create called on IdType: user: forgerocker attrMap = {uid=[forgerocker], unicodePwd=xxx..., sn=[Rocker], inetuserstatus=[Active], givenname=[], cn=[Forge Rocker]}

:

:

LDAPv3Repo:06/18/2012 09:51:10:065 PM SGT: Thread[http-apr-8180-exec-1,5,main]

    : before ld.add: eDN=cn=forgerocker,cn=users,DC=az-ex,DC=sg

LDAPv3Repo:06/18/2012 09:51:10:207 PM SGT: Thread[http-apr-8180-exec-1,5,main]

ERROR: LDAPv3Repo.create failed. errorCode=53  0000001F: SvcErr: DSID-031A11E5, problem 5003 (WILL_NOT_PERFORM), data 0

LDAPv3Repo:06/18/2012 09:51:10:207 PM SGT: Thread[http-apr-8180-exec-1,5,main]

LDAPv3Repo.create failed

com.sun.identity.shared.ldap.LDAPException: error result (53); 0000001F: SvcErr: DSID-031A11E5, problem 5003 (WILL_NOT_PERFORM), data 0

at com.sun.identity.shared.ldap.LDAPConnection.checkMsg(LDAPConnection.java:5523)

at com.sun.identity.shared.ldap.LDAPConnection.add(LDAPConnection.java:3234)

at com.sun.identity.shared.ldap.LDAPConnection.add(LDAPConnection.java:3255)

at com.sun.identity.shared.ldap.LDAPConnection.add(LDAPConnection.java:3181)

at com.sun.identity.idm.plugins.ldapv3.LDAPv3Repo.create(LDAPv3Repo.java:2100)

at com.sun.identity.idm.server.IdServicesImpl.create(IdServicesImpl.java:442)

at com.sun.identity.idm.AMIdentityRepository.createIdentity(AMIdentityRepository.java:384)

A google search solves the issue.. the connection from OpenAM to AD must be in SSL mode. Read here and here for detailed explanation. This is due to the attribute unicodePwd in AD. 

So I went ahead to enable my AD for SSL and re-configure the AD Data Store in OpenAM to connect via SSL.


Well, the following error is ever so common ... "PKIX path building failed: .... unable to find valid certification path to requested target"

Always happen without fail if self-signed certificate is used. :) Well, importing the Root CA cert of the self-signed certificate into the cacerts store in JDK will resolve the issue.

.

Ignore User Profile in OpenAM

There are 4 ways to control whether a user profile is required for authentication to be successful in OpenAM.

The default is Required and the next popular is Dynamic. I have never tried Ignored before. 

So I went ahead to test what to expect if Ignored is chosen.

1. I created a AD authentication module

2. I purposely ignored creating a corresponding AD data store

3. I even went to the extend of deleting the default OpenDJ data store

The following is the screen when my AD user successfully authenticated - "You've logged in".

The end user console is skipped from displaying.

A session is still created when you navigate to the Session tab in OpenAM Admin Console.

.

OpenAM Active Directory Data Store

If you create an Active Directory Data Store post-installation, then you'll come across this screen. The instruction (see here) was clearly stated:

Before saving these settings ensure the 'Load schema when finished' check-box at the top of the page is selected, as this should install the OpenAM schema into Active Directory.

The LDIF file that contains the AD User schema can be found in [openam-config-directory]/ldif/ad/ad_user_schema.ldif.


So where will these schema be populated in the AD? I had a hard time finding them as I am not familiar with ADSI on Windows machine.

The trick was to select "Schema" for well known Naming Context!

There you go ...

.

OpenAM Logging

In OpenAM 10.0.0, there is an enhancement to the logging subsystem.

Previously, the log files in %BASE_DIR%/%SERVER_URI%/log/ are accumulated. And they can get really huge if no proper housekeeping is in place.

So the enhancement in this release is to implement log rotation (-1 implies no rotation).

To turn on log rotation, key in a value greater than -1 (1440 in minutes == 1 day). There is no need to restart OpenAM server.

Nice! The logs are now rotated every day.

However, the nightmare remains.... the logs in debug and stats directories remain unrotated. The next improvement should be the log files in these 2 directories. For the time being, nightly cron is still a must! :)


Updated on 21th Jun 2012

I was wrong about saying logs in debug directory remain unrotated. See OPENAM-41. It's there, but super not obvious only.

.