OpenDJ and OpenAM compatibility

Recently, due to the strict PDPA requirement from the PDPC (Personal Data Protection Commission Singapore), we are to ensure the user profiles stored in OpenDJ are kept totally safe. I blogged before that one of my customers was exploring Data at Rest Encryption Solutions from Gemalto. That was almost a year ago.

I met customer a month ago. I told him that OpenDJ Data Confidentiality feature can be enabled on a per database backend basis to encrypt LDAP entries before being stored to disk in OpenDJ 3.x. There's a blog by Ludo that explains the feature in detail.

However, customer is still on OpenAM 11.0.3. There might be compatibility issue.

Lucky am I. I just saw an article in ForgeRock Backstage.


Embedded OpenDJ

External OpenDJ

In short, customer cannot proceed to integrate OpenAM 11.0.3 with OpenDJ 3.5.

By the way, saw that last line? "It is strongly recommended that you always upgrade to the latest maintenance releases for whichever versions of OpenAM and OpenDJ you have deployed."


Yes, easier said than done. There is always a tech-refresh cycle and a cost attached to each refresh. It's really not as simple as upgrade to the latest release as and when it's available.


.

ELK . OpenAM . OpenDJ - Part 3

Today, my customer called to alert me that their SSO infrastructure is "not stable". We had OAuth2 Provider configured for this customer. OAuth2 clients' authentication hit into unauthorized error. A little while later, he called me again telling me the Login Page went into an infinite loop. 

But ... but ... everything went back to normal after a while. 

OpenAM/OpenDJ are very stable products from our experience dated way back to Sun Microsystems. They can't go into crash mode out of the sudden. 

In the end, we found out that the network team is tech-refreshing the network. Wow! During office hours? Yes. Well done!

Anyway, until we found out the root cause, the first tool I used was to quickly go to the ELK which we have set up for this customer. Nothing unusual. 

And I blogged about how my team use ELK platform to monitor ForgeRock Open Identity Stack here and here.

"Invalid Password Server Trend Live" - This tracks the user invalid password events.

In my previous blog, I wrote that we noticed huge Invalid Password spike and subsequently identified that it was due to a malfunction automated application. 

Below is a trend for the past 30 days. It was obvious the application went haywire on 29th April and was subsequently rectified on 11th May.

I went to the same dashboard and selected the trend for past 7 days. As observed, on a normal day, the max Invalid Password should be in the 65 range (per OpenAM server).

This is something we are working on right now. With this threshold on mind, we are building a Notification Service to trigger proactive alert to the Ops team. 

.

ELK . OpenAM . OpenDJ - Part 2

I blogged about how we use ELK to monitor trends, especially abnormal ones, on ForgeRock Identity Stack. 

"Login Failed Server Trend Live" - This tracks the user login failure events. 

Just a few days ago, we observed that the number of Login Failed events had increased.

So, we zoomed in and found out there were many lines of the following errors:

"2016-05-17 00:09:00"   "Login Failed|module_instance|Application"      "Not Available" golfdigest 202.xx.xx.xx   INFO    o=xxx.sg  AUTHENTICATION-268     "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" Application     202.xx.xx.xx"


I know if module_instance is Application, then this is not a user authentication. Most likely, it is a Policy Agent in action. By the way, every Policy Agent will need to authentication with OpenAM in order to pull the policies.

So based on the IP address (202.xx.xx.xx), we found out the owner of the application. Ah! This is a defunct site. The SSO administrator has already removed the Policy Agent "golfdigest" from OpenAM as part of the sunsetting process, but the network team has not disabled/removed the Policy Agent on the Apache web server yet.

We have many more useful dashboard in Kibana, which we use for operational purpose.  I'll share later.


.

Ansible . OpenAM . OpenDJ . OpenIDM

For some of our larger deployment of ForgeRock Identity Stack, we will usually request to install Ansible on the development node.

For pure Linux environment, it works like a charm once SSH keys are exchanged during initial setup. (For Windows, it is supported but slightly more complicated to set up)

What do we use Ansible for? Almost every operational task.

Change configuration files; Update custom codes; Update JSP/UI pages; Deploy patches from ForgeRock; Restart Apache/Tomcat servers ... anything.

We used to make human errors occasionally when we managed a farm of over 10 OpenAM and 6 OpenDJ servers. Now that everything is pushed from development node, there is hardly any human error.


I just saw Ansible Tower on Ansible website.

No urgent need to use this feature. Our current Ansible already has logging/audit in place.

Anyway, once a playbook script is tested, there is hardly any error when it is being executed.

.

ELK . OpenAM . OpenDJ

We have ELK (Elasticsearch, Logstash and Kibana) deployed for a long time OpenAM/OpenDJ customer of ours some time back. The idea is not new. Similar solutions have been deployed by some other ForgeRock folks/partners.

We know ELK can only keep trends, but not send notification. (Ok ok, Elastic does offer Watcher for its commercial version) What we intend to do is to add Notification Service side-by-side with ELK. No, we do not intend to keep all data from OpenAM/OpenDJ in Elastic and to trigger alert from there. Some data are not useful to keep in Elastic (e.g. total entries count from all OpenDJ to determine if replication is operating optimally). We just need a simple cache layer (e.g. Ehcache) to keep these types of "data-in-transit" in order to trigger alerts to administrators/operators.

I'll talk more about this next time.

But so far, how useful is ELK to customer? Pretty good feedback.


"Login Failed Server Trend Live" - This is a live trend where by Logstash agents send "live" data from all OpenAM servers by monitoring amAuthentication.error logs. This tracks the user login failure events. 

If the user login failure count is high for a particular day for a particular OpenAM node, we can zoom into amAuthentication.error log to find out more.

"Invalid Password Server Trend Live" - This tracks the user invalid password events. 

This trend is different from the previous. An Invalid Password event happens when a user id is correct, but password is invalid. 

"2016-05-10 13:00:43"   "Invalid Password"      "Not Available" UID=ntustc001,ou=XXXX,o=xxx.sg    202.83.xx.xxx   INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"       "Not Available" LDAP    202.83.xx.xxx

A quick zoom into amAuthentication.error log reveals a particular user was attempting to log in with an invalid password.

[amuser@f1]$ cat amAuthentication.error.20160510 | wc -l
18475

[amuser@f1]$ cat amAuthentication.error.20160510 | grep -i ntustc001 | wc -l
18039


A total of >18k invalid login attempts. That's quite unusual.

This is where the customer service personnel can call up their paying customer to find out what exactly happened and if he/she requires a password reset service.

Proactive customer engagement model!



By the way, if you look at amAuthentication.error in-depth, you might see some Chinese characters like 登录失败  (Login Failed) and 无效密码 (Invalid Password). These are traffic from Chinese locale browsers.


"2016-05-10 07:00:33"   登录失败        "Not Available" "Not Available" 118.176.xx.xxx  INFO    o=xxx.sg  AUTHENTICATION-200      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    118.176.xx.xxx

"2016-05-10 08:56:22"   无效密码        "Not Available" uid=A480,ou=xxx,o=xxx.sg  
     202.4.xxx.xx    INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    202.4.xxx.xx

LDIF Delta Utility

This summary is not available. Please click here to view the post.

OpenDJ with Data at Rest Encryption Solutions Part II

I blogged about OpenDJ with Data at Rest Encryption Solutions few weeks back. The POC was carried out with OpenDJ 2.6.

OpenDJ 3.0 has recently been released and the test equipment (SafeNet Data at Rest Encryption appliance) is still in our office, why not let's run a test again OpenDJ 3.0? :)

In OpenDJ 3.0, there are 2 local DB backends to choose from: JE (Oracle Berkeley DB Java Edition - 2.5 version) and PDB.


The throughtput for search operation is faster in OpenDJ 3.0 as compared to version 2.6. The response times are also slightly faster.


Encrypted partition: 11229 (JE) & 10488 (PDB) vs 10210 (JE)   ~10% diff
Non-encrypted partition: 11305 (JE) & 11235 (PDB) vs 10410 (JE)  ~ 8.5% diff

For write operations, the difference in throughput is not big. In fact, PDB performs worse. (Hmmm... Maybe need to ask my colleague to re-run the test again) ..

Encrypted partition: 2926 (JE) & 2758 (PDB) vs 2750 (JE)   ~6.5% diff
Non-encrypted partition: 3282 (JE) & 2824 (PDB) vs 3199 (JE)  ~ 2.5% diff


The difference in response time is also not big. Again, I might want to re-run the test for PDB again. Or maybe some tuning is required. Not sure, it's something new to me. :)

Encrypted partition: 8.2 (JE) & 8.70 (PDB) vs 8.72 (JE)
Non-encrypted partition: 7.3 (JE) & 8.4 (PDB) vs 7.5 (JE)

.

OpenDJ with Data at Rest Encryption Solutions

We just completed a POC for a customer few weeks ago. This is a long time customer using Sun Directory Server and subsequently ported over to ForgeRock OpenDJ.

Recently, due to the strict PDPA requirement from the PDPC (Personal Data Protection Commission Singapore), we are to ensure the user profiles stored in OpenDJ are kept totally safe.

So, few weeks ago, we worked with Gemalto engineers to deploy OpenDJ local DB backend on a partition that is protected by SafeNet Data at Rest Encryption.

Worked like a charm!

I asked my colleage to run a little load-test (6 Connections / 4 Threads) using OpenDJ SDK Toolkit for comparison.

The response time for search operation looks pretty similiar. 

 Now, when it comes to write operations, then there is quite a bit difference in the response time.

Need to be careful in deployment sizing when customer decides on this solution.

.

OpenDJ Naming Conflicts nsuniqueid=xxx

Just realized I took a long break from blogging. So, I took a long holiday in December to make a family trip to Taiwan. Nice place to be in! In fact, I visit Taiwan frequently. This time round, we went to the eastern side of Taiwan - Taitung area. 

So, we have a long time customer who is using OpenDJ as a centralized user store. Few weeks back, he emailed to notify me that he found some "weird" user entries with DN as such:

dn: nsuniqueid=85c95685-21eb11df-80d5c084-e9b4da2d+uid=stageon238,ou=members,o=XXX.sg

A similar entry was found in the same OpenDJ:

dn: uid=stageon238,ou=members,o=XXX.sg


This is a naming conflict that only happens for replayed operations in a MMR setup.

I am not able to find any documentation from OpenDJ. Here are 2 links that talk about this naming conflict:

1. Resolving Naming Conflicts
2. Solving Naming Conflicts

In short, remove the entry which is older.


.

Install OpenDJ Directory Server With a Properties File

We were working with some AWS engineers on creating puppet scripts for a Open Identity Stack installation.

OpenDJ installation with a properties file was what we tackled first since it is fairly easy to get OpenDJ up and running, with command line.

There is a section in ForgeRock Documentation that describes how to Install OpenDJ Directory Server With a Properties File.

A sample properties file was given, but it was unclear what is the use of ldifFile?

As I didn't know what was that ldifFile parameter for, I commented it during installation. All went fine and the installation was completed successfully. But ...

When we started to configure OpenAM, we encountered error - "Invalid Suffix".

So I fired up OpenDJ Control Panel to take a look, and I discovered why. 

There was no entry at all. 

Usually when I install OpenDJ via command line, I would be given an option to choose like the following:

And I usually choose Option 1. I needed an equivalent via the Properties File installation method.

The documentation was not clear enough what we should do in such a situation. So I played around with the ldifFile parameter. I should define a base entry in the ldif File. And that will do.

dn: dc=cdemo,dc=sg
dc: cdemo
objectClass: top
objectClass: domain

And that really solved the issue! A check with OpenDJ Control Panel showed everything was in place after the successful configuration of OpenAM.

Nice!

.

OpenAM 12 : Changes & Deprecated Functionality

We kick-started an Open Identity Stack (OpenAM, OpenIdM, OpenDJ) project yesterday. * feeling excited *

So for this project, we intend to use OpenAM 12.0.1 - the latest release to 12 series.


In Additional New Features, I saw the following which my customers asked for:

• Audit Logging to Syslog
• CORS Support for OpenAM APIs
• Session Failover Across Sites
• Whitelist for Custom Login URIs
• Policy Support for Common HTTP Operations

Personally, since I'm very much into the internal working of OpenAM, my favourite will be:

• Reduced Cross-Talk

In OpenAM 12, there are quite a number of important changes which we need to be aware of. Read Chapter 3. OpenAM Changes & Deprecated Functionality.

The Policy Evaluation engine has undergone huge changes, so one has to be careful here when performing an upgrade. Luckily this is a green field project!

There is also a lot of fixes - read 4.1. Key Fixes.


.

ForgeRock Professional Service

These days, when ForgeRock sells subscriptions, it has an option to purchase Professional Service as a bundle. Free or not? I'm not too sure. It's part of price negotiation, I believe. 

The ForgeRock Professional Service can be utilized as follows: 

I think the original intend is to get customers up-to-speed with Open Identity Stack (OIS) deployment. This is especially helpful/applicable to customers who are more technical-savvy and would like to deploy OIS by themselves. High-level Architecture Workshop will be helpful.

Now, what if a SI who is well-trained in OIS is involved?

Customers, being customers, will want to cut cost. Always.

The worst case scenario is for a customer to ask the SI to cut-off a few man-days from implementation cost, since the customer can utilize Product Specialist Field Assistance. Wow! Unbelievable, isn't it? Nay, this should not happen at all and it really doesn't make sense. If customer insists, I think the best way is to walk away.


I think this Professional Service should be best utilized in Production, especially On-Site troubleshooting and Performance Analysis. There will be situations (or there had been?) where a site is badly deployed by a so-so System Integrator (SI) and is now causing production issues. It would be real hard for support engineers to provide remote support. An on-site troubleshooting comes in handy.


.

Open Identity Stack - Common Audit Framework

I'm looking forward to the Common Audit Framework in the next release of Open Identity Stack (OpenAM, OpenIDM, OpenDJ and OpenIG). 

At present, each component is logging in its own format and to different targets (e.g. file-system, database, syslog). Thus making debugging across components tedious.

This is definitely a good move!

.

Online export of OpenAM Configuration data from OpenDJ

We all know that the OpenAM configuration which we managed via the OpenAM Administration console is stored in either an embedded OpenDJ server (by default) or an external OpenDJ server.

For production systems, it would be recommended to backup the OpenAM configuration data nightly, specifically ou=services,o=[your organization]. 

I'm not too sure about ou=opensso adminusers,o=[your organization], but I'll backup a copy just to play safe.

Operationally, how huge will this OpenAM configuration data be? I can safely say it's very small. Will it have any performance hit on the OpenDJ server? Not likely. If the OpenDJ server is embedded which shares the same JVM as OpenAM, will it affect the functioning of OpenAM server? Not likely.

Anyway, we are talking about backup during off-peak hours, ain't we?

.

Porting embedded OpenDJ in OpenAM to external OpenDJ - Part IV

I blogged a few times on the procedures to port embedded OpenDJ in OpenAM to external OpenDJ. See Part I, Part II and Part III.

So we are happily done with the migration from embedded to external OpenDJ. What's next?

There is a last mile -- clean up!

Remember the old embedded OpenDJ is started whenever OpenAM is started? It stays in the same JVM as OpenAM (you'll get it if you are in the Java world).

So this is to say even if OpenAM is configured to point to external OpenDJ, the old embedded OpenDJ will always be started whenever OpenAM is started. Simple logic.

So, how can we stop the old embedded OpenDJ from starting?

.../[openam-configuration-directory]/opends is where the embedded OpenDJ stays. 

Remove or rename the folder will do the trick. (I usually rename, never remove in case I want to roll back)


Done.

.

OpenDJ Replication Architecture

The way OpenDJ replicates data from one node to the other is slightly different from Sun/Orace Directory Server. OpenDJ utilizes Replication Servers for data replication as shown below:

I was reading OpenDJ Mailing List the other day and came across a great response from Chris Ridd:

The key difference in OpenDJ is that replication is not configured between pairs of servers like it often is in other products.

Instead, OpenDJ logically consists of a number of replication servers that are “fully meshed”, i.e. every replication server is connected to every other replication server. The replication servers hold the changelogs, but no LDAP data. OpenDJ also has directory servers that hold LDAP data that connect to replication servers in order to send and receive changes.

(You can configure an OpenDJ server to be a pure directory server (a DS) or a pure replication server (an RS) or both at the same time (DS+RS). The setup tool will create all of these combinations for you.)

When you use “dsreplication enable”, you are effectively adding a new node (--host1) to the existing fully meshed topology (--host2 can be any host in the topology). As Jean-Noël mentioned, the replication servers exchange views of the topology, so when you add a new replication server to the topology, it automatically becomes connected to every other replication server in the topology.

Good read!

Configuration Data Store Indexes for External OpenDJ

In production, the Configuration Data Store is usually an external OpenDJ server. In fact, most usually a pair of OpenDJ servers.

There is a section in OpenAM documentation - Preparing an External Configuration Data Store, which is useful . Read here.

The most important item is to set indexes on OpenDJ during configuration. See below:

In a recent deployment, I have forgotten to set the above indexes and the machine where OpenDJ servers were running on always hit 90-100% CPU. Very top-ish through the day.

After the indexes were set, the OpenDJ servers went back to normal.  Proper indexing does help.


.

ForgeRock has set up APJ HQ in Singapore

Finally, it's official. ForgeRock announced that it has set up its Asia-Pacific Japan (APJ) regional headquarters in Singapore. Full press release here.

In fact, we have been working closely with Sumal & team since December 2014. Customers in the Asia region, especially those in Singapore, have been asking for ForgeRock's presence years ago. This has finally become a reality.

In addition, support engineers will also be stationed in Singapore to better serve customers in this region.

This is a great start to the year!


By the way, Singapore Press Holdings is Azimuth's customer. :)

.

Insufficient Access Rights: You do not have sufficient privileges to perform an unindexed search

In my previous post, I mentioned that we are currently migrating OpenAM 9.5.3 to OpenAM 11.0.2 for a customer based in Singapore. At the same time, we are migrating a very old OpenDJ 2.4.3 to OpenDJ 2.6.1.

And of course, we do have customized schema in OpenDJ and customized codes in OpenAM. Which customer will not have customized requirement?

Anyway, when we run an unit test against the new setup, we encountered the error below:

11:24:50.514 [http-apr-7070-exec-1] DEBUG DEBUG - [LdapConnection - getAttributes] Getting attributes for filter (sgloginid=USERA)

11:24:50.532 [http-apr-7070-exec-1] ERROR DEBUG - Check devices command exception: Proxy User ID: USERA  User ID: USERA, Facebook ID: null, M2GW IP: x.x.x.x

org.forgerock.opendj.ldap.ErrorResultIOException: org.forgerock.opendj.ldap.AuthorizationException: Insufficient Access Rights: You do not have sufficient privileges to perform an unindexed search

        at org.forgerock.opendj.ldif.ConnectionEntryReader.hasNext(ConnectionEntryReader.java:235) ~[opendj-ldap-sdk-3.0.0-SNAPSHOT.jar:na]

One of my colleagues googled and directed us to OpenDJ Administration Guide Indexing Attribute Values - Determining What Needs Indexing.

A quick fix is to grant unindexed-search privilege to the account that binds to OpenDJ. But should we?

No, I do not think so.

The article says it all, in order of preference to resolve unindexed search issue:

1. If directory users tell you their client applications are getting this error, then you can work with them either to help them make their search filter specific enough to use existing indexes, or to index attributes they need indexed in order to perform their searches.

2. If you do need to allow some applications to perform unindexed searches, because they need to retrieve very large numbers of entries for example, then you can assign them the unindexed-search privilege.


In our case, we should index the attributes as sgloginid is used very frequently in OpenAM for authentication and in another customized application.

.

POODLE SSL Bug and the impact on OpenDJ

Ludo has posted a blog on POODLE SSL Bug and OpenDJ.

Good read!