OpenAM Debug Files Mapping

Have you ever tried looking at OpenAM source code and attempted to debug from the log files? And subsequently got lost? :> 

There are numerous debug logs. Which one to zoom in?

[cheechong@cdemo debug]$ ls

Authentication  Configuration  CoreSystem  IdRepo  Session  amUpgrade

Well, there is a mapping file actually. It's debugfiles.properties.


[cheechong@cdemo classes]$ pwd
/home/cheechong/opt/apache-tomcat-7.0.35-am1/webapps/am10/WEB-INF/classes
[cheechong@cdemo classes]$ ls -alt debugfiles.properties 
-rw-rw-r-- 1 cheechong cheechong 3439 Apr 13  2012 debugfiles.properties


[cheechong@cdemo classes]$ head debugfiles.properties
AuthAgents=Authentication
FAMSTSAuthorizationProvider=WebServices
LDAPConnectionPool=CoreSystem
LDAPv3EventService=IdRepo
LDAPv3Repo=IdRepo
PLLClient=CoreSystem
PossibleLocales=CoreSystem
ReplayPasswd=Authentication
SAE=Authentication
SystemTimer=CoreSystem
:
:
:


.

End of Service Life Announcement for OpenAM 9.x, OpenDJ 2.4.x, OpenDJ 2.5

I just received an announcement from ForgeRock Support regarding the end of service life for the various Open Identity Stack products.

Time to upgrade! :)

.

Defense In Depth

I came across this article from CA that talks about security threats and attacks.

At the end of the article, it mentioned briefly on how to defense in depth against threats and attacks. I find it a good reminder for everyone of us.

• Least privilege access
 – to help ensure that users have the minimum access to do their job

• Decouple security from administration
 – don’t allow non-security personnel to change security settings

• Fine-grained control over admin access
 – helps contain the damage if a system is breached

• Shared account management
 – to eliminate shared passwords and improve accountability

• User activity auditing
 – track and audit all access to help identify potential attack attempts

• Information protection
 – control not only access to information, but its use

Very common-sense advice, but how many of us really practice it at all times? :)

.

Why a Write Op on a traditional LDAP server is expensive?

I was doing some reading up on Sun Directory Server Enterprise Edition 7 (Sun DSEE) and came across a section on using a Directory Proxy Server to distribute loads, especially Writes.

The documentation went further by explaining why a Write Operation is resource intensive. I find it refreshing to revisit technical details.

Write operations are resource intensive. When a client requests a write operation, the follow sequence of events occurs on the database: 

•  The backend database is locked

•  The entry is locked in the database cache

•  The access control check plug-in is called

•  Any backend pre-operation plug-ins are called

•  The database transaction begins

•  The database files are updated

•  The old entry cache is replaced with new data

•  The database transaction is committed

•  Any backend post-operation plug-ins are called

•  The backend database is unlocked

Because of this complex procedure, an increased number of writes can have a dramatic impact on performance.

.

OpenDJ Replication Enable Error

I was trying to establish a replication agreement between 2 OpenDJ 2.4.6 servers.

[dsadm@appr1 opendj]$ bin/dsreplication enable --host1 appr1.abc.com --port1 4444\
>   --bindDN1 "cn=directory manager" \
>   --bindPassword1 password --replicationPort1 8989 \
>   --host2 appr2.abc.com --port2 4444 --bindDN2 "cn=directory manager" \
>   --bindPassword2 password --replicationPort2 8989 \
>   --adminUID admin --adminPassword password --baseDN "o=openam" -X -n

Establishing connections ..... Done.
Checking registration information ..... Done.
Configuring Replication port on server appr1.true.th:4444 ..... Done.
Configuring Replication port on server appr2.true.th:4444 ..... Done.
Updating replication configuration for baseDN o=openam on server appr1.true.th:4444 ..... Done.
Updating replication configuration for baseDN o=openam on server appr2.true.th:4444 ..... Done.
Updating registration configuration on server appr1.true.th:4444 ..... Done.
Updating registration configuration on server appr2.true.th:4444 ..... Done.
Updating replication configuration for baseDN cn=schema on server appr1.true.th:4444 ..... Done.
Updating replication configuration for baseDN cn=schema on server appr2.true.th:4444 ..... Done.
Initializing registration information on server appr2.true.th:4444 with the contents of server appr1.true.th:4444 .....
Error during the initialization with contents from server
appr1.true.th:4444.  Last log details: [19/Jul/2013:15:24:12 +0700]
severity="NOTICE" msgCount=0 msgID=9896349 message="Initialize From Replica
task quicksetup-initialize1 started execution".  Task state: STOPPED_BY_ERROR.
Check the error logs of appr1.true.th:4444 for more information.
See /tmp/opends-replication-4857551577835267301.log for a detailed log of this
operation.

There was not any useful detail in /tmp/opends-replication-4857551577835267301.log. No luck.

So I started to debug ... By looking at the above error stack, I knew that the problem occurred only during initialisation. There was nothing wrong with replication configuration etc ... Then I realised the system administrator has iptables running! 

Bingo! Port 8989 (used by the OpenDJ Replication Server) is not opened. 

The error message was not useful at all. :)

.

New Gmail Features!

Gmail surprises me again today!

The Inbox is now organized into categories - Primary, Social, Promotions, Updates and Forums.

Notice that "Social" is 1 of the 5 identified categories? Yes, social media is here to stay. Do not forget, everybody. :)

Another feature I like is the Red Highlighter that stays right at the left of the Inbox. It moves as you switch from one label to the other. Pretty neat!

Well done!

.

AMSFO Installation on Windows Platform

This is the 1st time I tried a full installation of OpenAM and AM Session Failover on Windows platform. No happy beginning though. :(

There are 2 executables in the "bin" directory which are causing problem for me while I was trying out AM Session Failover. They are amsfopassword.bat and amsessiondb.bat.

When I tried to generate a password file via the amsfopassword.bat, it kept complaining that 'C:\Program' is not recognized as an internal or external command.

Taking a look at the .bat file, I realized there is a space between "Program Files".

Anyway, I myself also think space is evil. So, I went ahead to move the default JDK to my own OpenAM directory, instead of "Program Files".

That did the trick!

.

The Directory Server jar file ._snmp-mib2605.jar in directory ../lib/extensions cannot be loaded

I tried to install the latest release of OpenDJ 2.6 on my customer's environment in Thailand. (Download OpenDJ 2.6 here)

Somehow, I just could not get it to work! I kept getting the following error:


Jul 10, 2013 2:26:17 PM org.opends.quicksetup.Application$ApplicationPrintStream println
INFO: An error occurred while attempting to process the Directory Server
configuration file /data/webdata/opendj/config/config.ldif:  The Directory
Server jar file ._snmp-mib2605.jar in directory
/data/webdata/opendj/lib/extensions cannot be loaded because an unexpected
error occurred while trying to open the file for reading:  error in opening
zip file (ZipFile.java:-2 ZipFile.java:127 JarFile.java:135 JarFile.java:99

This is very strange. The environment which I was trying to install is a RHEL OS. And I have installed OpenDJ many times without any error.

A google brought me to this OpenDJ bugster OPENDJ-628. But this was marked as "Not a defect" and it happened only on Mac OS.

Hmmm... what's the relationship then? I went to the directory on the server where the OpenDJ tar file was un-tarred.

Did I say tar file? Bingo!

Before I traveled to Bangkok, I download OpenDJ 2.6 onto my Macbook Air. And my Safari auto-magically unzipped for me.

And so before I uploaded to the server in my customer's environment, I had to do a tar first. Yes, on my Macbook Air. 

And thank you Apple! See the diagram above.. The tar file contained tons of ._ files!!

OpenDJ doesn't like it during installation and setup.

So, I had to download OpenDJ again. But I had to make sure the "Open safe files after downloading" is unchecked on my Safari first.

Now, the downloaded OpenDJ remained in its ZIP format. Uploaded to the server again; unzipped; execute ./setup --cli.

Done!

.

Common UI for Open Identity Stack

Moving forward, there will be a common UI for the 3 products from ForgeRock. (Well, 4 if we count Identity Bridge in :> )

The UI is much cleaner, unlike those inherited from Sun/Oracle.

The main emphasis will be on end-user pages. e.g. Login page, Registration page, User Self-Help page. They will be highly customizable without breaking any codes during patches or upgrades. 

And of course, REST is the way moving forward. 

All 3 products will expose REST endpoints. This will be how the UI communicates with the underlying products.


.

OpenIdM + Brainwave - Perfect Match!

During the Open Identity Summit, I saw a presentation which got me pretty excited.

It was a presentation by Brainwave.

While OpenIdM aims to be the market leader in Identity Management, Brainwave wants to be the market leader in Identity Intelligence. It just won the Gartner 2013 Cool Vendor of the year!

So how do the 2 products complement each other? A simple diagram below serves to illustrate the relationship.

Architecturally, they are also a good fit. Both are running on top of OSGi framework.

Now, when it comes to Reports and Insight, this is where the WOW! comes.

I must say they are very comprehensive. Basically, they cover:

1. What are my risks?
2. What needs to be fixed?
3. Am I compliant?

In a very comprehensive way, I must say again. And there are multiple views to look at the same sets of data. Look here for a list of benefits from Brainwave.

Overall, I am pretty impressed with Brainwave and my team has initiated an internal detailed study on how to best integrate OpenIdM with Brainwave. This will better prepared ourselves to introduce an alternative solution to our existing Sun IdM customers.

Again, I must say big complex products out there in the marketplace will not necessarily win the battle. Yes, their salesmen are preaching hard these days. But I still believe lean deployment is the way to go.

Do you not agree with me that most customers buy a complex product usually to only use about less than 40-50% of the features? Then, why pay so much?

.

OpenDJ 2.6.0 is shipped!

The much anticipated OpenDJ 2.6.0 is finally released on the 4th of July 2013. I think the team purposely choose this date for shipping. :)

There is a whole bunch of new features and bug fixes with this release. Take a look at the long release notes if you can.

Of the many new features, the followings are my favorites:

• OpenDJ now provides native RESTful access over HTTP to directory data
• OpenDJ now lets you delegate authentication to another LDAP directory service, such as Active Directory. The feature is called pass through authentication (PTA)
• OpenDJ now lets you filter access and audit logs to focus on messages that interest you
• The OpenDJ dictionary password validator can now check whether a password value contains dictionary words as substrings
• OpenDJ now uses Berkeley JE 5, which brings many performance improvements
• OpenDJ now lets you execute control-panel as any user, not only the user who installed OpenDJ

There are really many new features in this release. Bug fixes as well. 

I do not think the other company can catch up with the same old code base, given the pace ForgeRock is going. :)

Well done, guys in ForgeRock!

.

Unable to parse LDIF entry starting at line 25

So, I had an OpenAM-10.1.0-Xpress running in one of my test environment. All was good (I think I have run it for almost 3 months), until this morning when I restarted the Tomcat.

Before I restarted the Tomcat, I tried to manually start OpenDJ. But it took a long while, and the following error was shown:

[03/Jul/2013:13:35:02 +0800] 0 caught error thread={main(1)} method={loadSchemaFile(SchemaConfigManager.java:499)} caught={org.opends.server.util.LDIFException: Unable to parse LDIF entry starting at line 25 because the line "-" does not include an attribute name}
[03/Jul/2013:13:35:20 +0800] category=CONFIG severity=SEVERE_WARNING msgID=3276992 msg=Schema configuration file cts-add-schema.ldif in directory /home/azlabs/var/openam1/opends/./config/schema cannot be parsed because an unexpected error occurred while trying to read its contents as an LDIF entry: Unable to parse LDIF entry starting at line 25 because the line "-" does not include an attribute name (id=5439494)

So I went to the OpenDJ schema folder. What is cts-add-schema.ldif?

This is mad! Why is there a LDIF modify script in the schema folder?

This was reported as OPENAM-2272. A similar JIRA was raised as OPENAM-1975. Both are reported for OpenAM-10.1.0-Xpress.

My personal thought is this is an ugly way of extending the OpenDJ schema. 

Firstly, the naming convention is already wrong. 

Secondly, cts-add-schema.ldif is rightly not a schema file. Why should it be in a schema folder at all?

Same goes to 99-oauth2attributes.ldif. 

And worst thing is, the original 99-user.ldif pre-10.1.0 has become:

This is so ugly... *sigh* 

I hope there is a clean up prior to release of OpenAM 10.2 stable.

There is 1 thing I always tell my team: Do not think you know everything. And if things work, it does not always mean we have done it the proper way.

We are only specialists in certain domains. If we are not an expert in a particular domain, we should always seek opinion from the domain experts.  

.

New Paradigm for the Modern Web - Part II

I blogged about the New Paradigm for the Modern Web last week after I attended ForgeRock Open Identity Summit in California.

So, ForgeRock defined its own definition of what it thinks the modern web is all about.

Yesterday, I attended the 1st Compuware APM User Conference in Singapore. Guess what? I saw something similar.

Cloud, Mobile, Social is not going to go away anytime soon. They will just get bigger, and we should better prepared ourselves for them.


.

Goodbye Postini

We have been using Postini Message Security way before Google bought Postini over. It has been a long ride.

Comes Jun 2013, it's time to say goodbye. We are not keen to transit over to Google Apps Platform, so are our customers.

Hello, SpamHero. We have been running trial for 2 months. So far, we are satisfied with the services provided. Well, except that it is not able to filter ADV messages that well. (Postini has this feature where it can aggressively filter off any message that contains advertisement. I like that feature! Just me.)

.

Agent Session Idle Timeout

By default, OpenAM Policy Agent has an non-expiring token after it first authenticated with OpenAM server.

How do we change this value?

com.iplanet.am.session.agentSessionIdleTime


Nice.


. 

Changing Bind DN/Password in Directory Configuration

I blogged twice on how to port the embedded OpenDJ in OpenAM to an external OpenDJ. Meanwhile, I also discovered something weird when changing the Bind DN/Password in Directory Configuration.

Let's say you ignore the porting of the embedded OpenDJ to an external one. Let's only assume you want to only change the Bind DN and Password of the default "cn=Directory Manager" for security reason.

Ludo has a good article on how to create Multiple Directory Administrators. Of course, you can create a more restrictive proxy administrator for OpenAM. 



Now, if you are on OpenAM 10.0.1, then no problem. You simply change the default "cn=Directory Manager" and its corresponding password to one which you have newly created.

A restart of OpenAM will modify the bootstrap file to reflect the new Bind DN. (Of course, I still do not know when OpenAM will auto-magically modify the bootstrap file as-and-when changes are made via the OpenAM console; and when it will require a restart in order for the bootstrap file to be modified)

And now, we are done! That's OpenAM 10.0.1.


However, if you are on OpenAM 10.0.0, then no luck. Sorry. No matter how you change the Bind DN/Password via the OpenAM console, a restart of OpenAM will flush away the entries and replaced with the original ones. The bootstrap file is also not modified with the new pair of Bind DN/Password.


Why? I spent some time trying to figure out. In the end, I found out that the problem was with the LDAP update when entries are modified on OpenAM console.

I opened OpenDJ control panel and observed ou=com-sun-identity-servers, ou=default, ou=GlobalConfig, ou=1.0, ou=iPlanetAMPlatformService, ou=Services, ....

When I was modifying the entries in OpenAM 10.0.1, I did managed to see that the DirDN was changed from "cn=Directory Manager" to "cn=Admin2, dc=cdemo, dc=sg" when I clicked SAVE on OpenAM console.


Before

After

The same was not happening when I did the experiment on OpenAM 10.0.0. Do you have a clue? I dun.


.

Multi-Site Deployment Issue

When I attended the Open Identity Summit, I took away with me a great tip for OpenAM when Steve Ferris presented his OpenAM Survival Tips.

In Production environment for large deployment, sometimes we have to configure multi-sites for customers. (Multiple sites => multiple login URLs)

If I could illustrate with a diagram, the following is the recommended deployment architecture for multi-sites. Each site will point to its own pair of MQ servers. 

The reasons are clearly documented:

Multiple-Site configuration 

In a multiple-site configuration, two or more OpenSSO Enterprise servers are configured in each site. A multiple-site configuration is useful when you need to centrally manage OpenSSO Enterprise servers located in distant geographical locations. Multiple-site configuration is usually used in WAN environments, or where sites are managed as separate operational units within a LAN environment. Each site can have one or more load balancers. 

While system failover can be configured among all sites in the deployment, session failover is possible only within each site. WAN environments are subject to speed, network latency, firewall, and bandwidth issues. 

For these reasons, OpenSSO Enterprise session failover is not supported across multiple sites within a LAN or WAN environment. 

The following are typical reasons to use a multiple-site configuration: 

• Close proximity of OpenSSO Enterprise servers in a LAN environment 
• Underlying network infrastructure limitations exist. 
• Operational domains are managed as independent units. 
• OpenSSO Enterprise servers span across network boundaries as in the case of a WAN environment.

Why is it so? What's the problem technically?

It is due to the way Database URL list is keyed in for a site.

The following diagrams show what is happening behind the scene. All OpenAM instances will attempt to connect to the 1st MQ server in the Database URL list. If an OpenAM instance happens to be in another data-center from the 1st MQ server, no luck! It will still attempt to connect to it via WAN.

This, of course, causes latency and subsequently performance degradation. As such, session failover across WAN is never supported, even if you have a subscription from Oracle.


Well, not if you have a valid ForgeRock subscription. There is now an AM patch available for ForgeRock customers.

And thus, the following architecture is now made possible!

Just in time for one of my customers' multi-sites deployment.

.

Scheduled release dates for OpenAM, OpenIdM and OpenDJ

• OpenAM 10.2 Q3/2013
• OpenIDM 2.2 Q4 2013 
• OpenDJ 2.6.0 End of June 2013
• OpenDJ 3.0 Early 2014, withProxy services

Download link here.

.

OpenAM Mobile Application

Late last year, I blogged about Oracle Mobile Access and Social Management. I stated clearly why I think Cloud, Mobile, Social is the way going forward.

I'm glad OpenAM now has a solution for mobile application access. It is made possible by the latest release of OAuth2 Provider feature in OpenAM.

With the OAuth2 Provider in OpenAM, we also need to aware that OpenAM has REST endpoints exposed to its Core engine.

The REST endpoints are very important here. They make mobile integrations clean, without having to deploy custom APIs into the mobile applications.

Below is the sequence diagram for a OpenAM mobile application demo:

Pretty cool. Isn't it?


.