Why IAM Projects Fail? Part II

I woke this morning and received an email from SailPoint. It is the leader in identity governance (IGA) market as named by Gartner for Q2, 2016.

By the way, the underlying engine of SailPoint is Waveset from Sun Microsystems. My team is still supporting a customer not yet migrated out from Waveset. We have a long history implementing products from Sun Microsystems.

I'm fairly surprised Dell One Identity solutions are in the Leaders quadrant. It's not picking traction in this Asia South region. My team is trained in this product as well. Still waiting for deal-on. Ha!


But then, so what if a product is in the top quadrant? Will it warrant implementation success?

I said before and I am repeating now:

In the market, there is really not much difference between the various IDM/IDG products. I can safely say their features are almost 85-90% similar. It's the implementors & customers' key stakeholders ("People") that makes the difference between a successful and failure IAM project.


The magic quadrant is usually used by top-decision makers to cover their backside. Simply put.


I talked about the Pain Points previously. 

Again (coincidentally?), the biggest pain is People. 

• No ownership/main-driver (no full-time PM) 
• Not trained
• Not really know what they really need (Keep changing requirement) 
• Not enough support from application teams 
• No well thought-of test plans & not following test plans

If I may, I would like to add on 2 points which I recently observed.

• Design Documents were signed-off with no intention of following through (this is happening especially frequent in this region & usually causes huge delay)
• Test plans were not vetted and tests were carried out without support from internal team

These were caused by poor leadership. When a system is not well tested, especially the edge cases, things will break during operational time. And it's common. To blame the product, the implementers and the testers is easy. But do remember, when you point a finger at others, at least 3 other fingers are pointing back to yourself.


On point 2 - "Not trained", this could also mean the implementor is not well-trained as well. Oh really? Yes, it does happen. And it happened to us recently!

But being a responsible implementer, do we want it to happen? No. And do we want to redeem ourselves if given a chance? Definitely.

I still remember many years back when I was tasked by Sun Microsystems to debug a Sun Access Manager (the grandfather of OpenAM btw) issue for a teleco in Vietnam. I was given 22 days. In the end, it took me 4 long months! I was new to Sun Access Manager then. I was exploring. I'm not stupid by the way. When you are new to a product, you need time and patient.

Most importantly, my customer believed in me. He did not doubt me. He did not complain to Sun Microsystems behind my back. I felt bad at the delay but he was understanding. With help from support & product team, the issue was finally resolved. Usually, that's what the support & product team requires, an eye onsite to provide accurate feedback to them.


I keep repeating this story to my team these days, especially now when the morale is low.

In computing, there is no such thing called "not-fixable". Any bug can be fixed, it's just a matter of time. But of course, customers need to give you chance.

Otherwise,

To my team-mates: I handpicked you and I trained you, so I believe in you.


.

ELK . OpenAM . OpenDJ - Part 2

I blogged about how we use ELK to monitor trends, especially abnormal ones, on ForgeRock Identity Stack. 

"Login Failed Server Trend Live" - This tracks the user login failure events. 

Just a few days ago, we observed that the number of Login Failed events had increased.

So, we zoomed in and found out there were many lines of the following errors:

"2016-05-17 00:09:00"   "Login Failed|module_instance|Application"      "Not Available" golfdigest 202.xx.xx.xx   INFO    o=xxx.sg  AUTHENTICATION-268     "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" Application     202.xx.xx.xx"


I know if module_instance is Application, then this is not a user authentication. Most likely, it is a Policy Agent in action. By the way, every Policy Agent will need to authentication with OpenAM in order to pull the policies.

So based on the IP address (202.xx.xx.xx), we found out the owner of the application. Ah! This is a defunct site. The SSO administrator has already removed the Policy Agent "golfdigest" from OpenAM as part of the sunsetting process, but the network team has not disabled/removed the Policy Agent on the Apache web server yet.

We have many more useful dashboard in Kibana, which we use for operational purpose.  I'll share later.


.

Ansible . OpenAM . OpenDJ . OpenIDM

For some of our larger deployment of ForgeRock Identity Stack, we will usually request to install Ansible on the development node.

For pure Linux environment, it works like a charm once SSH keys are exchanged during initial setup. (For Windows, it is supported but slightly more complicated to set up)

What do we use Ansible for? Almost every operational task.

Change configuration files; Update custom codes; Update JSP/UI pages; Deploy patches from ForgeRock; Restart Apache/Tomcat servers ... anything.

We used to make human errors occasionally when we managed a farm of over 10 OpenAM and 6 OpenDJ servers. Now that everything is pushed from development node, there is hardly any human error.


I just saw Ansible Tower on Ansible website.

No urgent need to use this feature. Our current Ansible already has logging/audit in place.

Anyway, once a playbook script is tested, there is hardly any error when it is being executed.

.

ELK . OpenAM . OpenDJ

We have ELK (Elasticsearch, Logstash and Kibana) deployed for a long time OpenAM/OpenDJ customer of ours some time back. The idea is not new. Similar solutions have been deployed by some other ForgeRock folks/partners.

We know ELK can only keep trends, but not send notification. (Ok ok, Elastic does offer Watcher for its commercial version) What we intend to do is to add Notification Service side-by-side with ELK. No, we do not intend to keep all data from OpenAM/OpenDJ in Elastic and to trigger alert from there. Some data are not useful to keep in Elastic (e.g. total entries count from all OpenDJ to determine if replication is operating optimally). We just need a simple cache layer (e.g. Ehcache) to keep these types of "data-in-transit" in order to trigger alerts to administrators/operators.

I'll talk more about this next time.

But so far, how useful is ELK to customer? Pretty good feedback.


"Login Failed Server Trend Live" - This is a live trend where by Logstash agents send "live" data from all OpenAM servers by monitoring amAuthentication.error logs. This tracks the user login failure events. 

If the user login failure count is high for a particular day for a particular OpenAM node, we can zoom into amAuthentication.error log to find out more.

"Invalid Password Server Trend Live" - This tracks the user invalid password events. 

This trend is different from the previous. An Invalid Password event happens when a user id is correct, but password is invalid. 

"2016-05-10 13:00:43"   "Invalid Password"      "Not Available" UID=ntustc001,ou=XXXX,o=xxx.sg    202.83.xx.xxx   INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"       "Not Available" LDAP    202.83.xx.xxx

A quick zoom into amAuthentication.error log reveals a particular user was attempting to log in with an invalid password.

[amuser@f1]$ cat amAuthentication.error.20160510 | wc -l
18475

[amuser@f1]$ cat amAuthentication.error.20160510 | grep -i ntustc001 | wc -l
18039


A total of >18k invalid login attempts. That's quite unusual.

This is where the customer service personnel can call up their paying customer to find out what exactly happened and if he/she requires a password reset service.

Proactive customer engagement model!



By the way, if you look at amAuthentication.error in-depth, you might see some Chinese characters like 登录失败  (Login Failed) and 无效密码 (Invalid Password). These are traffic from Chinese locale browsers.


"2016-05-10 07:00:33"   登录失败        "Not Available" "Not Available" 118.176.xx.xxx  INFO    o=xxx.sg  AUTHENTICATION-200      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    118.176.xx.xxx

"2016-05-10 08:56:22"   无效密码        "Not Available" uid=A480,ou=xxx,o=xxx.sg  
     202.4.xxx.xx    INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    202.4.xxx.xx

LDIF Delta Utility

This summary is not available. Please click here to view the post.

Software Support

Recently, we hit into a production issue and were not able to resolve by ourselves. Thus we raised a support ticket on behalf of our customers. The support engineer needed the core dump when the issue re-surfaced again. So, he suggested to use Process Explorer or Debug Diagnostic tool.

Sure, no problem. But he added: "... If you encounter problem on dump capture, can you please engage Microsoft to assist as both process explorer and debug diagnostic tool are provided by Microsoft to capture dump."

Oh well, you were the one who asked customer to use tools from Microsoft to help you with debugging. Now, if that suggested tool(s) has issue, we need to raise a separate ticket with Microsoft?


Few weeks ago, we raised another ticket to the same engineer. By the way, he is support engineer for Product A in this company. The ticket was about how to integrate Product A with Product B from the same company.

To our astonishment, he responded with the following:

"It seems like some configuration issue on Product B. I suggest you open a ticket with Product B to check the configuration. Beside, from Product A documentation that you pointed out, I didn't find the OAuth provider can be Product B (Facebook and Google are the OAuth Provider mentioned). Therefore, I'm not sure if this can be done with Product B. If you have any additional documentation or details on how to integrate Product A with Product B as OAuth Provider, please share with me."

Hello, who is the customer? If you are not sure, you jolly well walk over (or skype/email) to Product B support team and find out more from them. You asking customer to bridge the communication gap between your two support teams? This is embarrassing.

You better wake up!


Side note: Besides ForgeRock products, my team delivers IAMS products from other principles as well.

OpenAM & Facebook Business Manager

We know OpenAM supports OAuth 2.0/OpenID Connect authentication module. OpenAM provides a wizard for configuring common OAuth 2.0/OpenID Connect authentication providers, such as Facebook, Google, and Microsoft.

In most use cases, for example Facebook, customers will go to Facebook Developers and create a new app for the company. 

The whole company will then use a common Facebook App ID.

Now, what if a company has multiple customer-facing websites? Each business unit owner would like to have their own Facebook Analytics for Apps to understand how people access their websites. 

This is where Facebook Business Manager comes in handy.

 

2 things to change:
1. Create 1 OAuth 2.0/OpenID Connect authentication module for each sub-account
2. Change FB Login icon on each website to authenticate with the approriate authentication module created in Step 1.


.

Policy Request and Response

A customer of mine has an issue with Policy Evaluation, so I was asked to help him debug.

We know OpenAM Web Policy Agent is PEP (Policy Enforcement Point), while OpenAM Server is PDP (Policy Decision Point). So, an web agent will send a Policy Request to OpenAM server and OpenAM server will send back a Policy Response, all in XML format.

The logs are captured in Policy debug log.


If the policy evaluation returns success, then "allow" will be returned as part of ActionDecision.

This maps well with what is configured in OpenAM Administration Console.

However, do take note that when policy evaluation returns failure, then "deny" is not returned as part of ActionDecision.

Customer thought something was wrong with OpenAM policy evaluation engine because he was expecting DENY to be returned from OpenAM server. 

. 

Why IAM Projects Fail?

I was presenting to a potential Identity Management & Governance customer the other day. I talked about Why IAM Projects Fail and the Pain Points from our field experience.

Gartner has this nice diagram to explain Why IAM Projects Fail. A summary can be found here. 

See the proportion of People vs Products? In the market, there is really not much difference between the various IDM/IDG products. I can safely say their features are almost 85-90% similar. It's the implementors & customers' key stakeholders ("People") that makes the difference between a successful and failure IAM project.

A good People combination will spend more time in defining Principles, Practices, Policies and Processes. 

The Pain Points are from our field experience.

Again (coincidentally?), the biggest pain is People.

• No ownership/main-driver (no full-time PM) 
• Not trained 
• Not really know what they really need (Keep changing requirement) 
• Not enough support from application teams 
• No well thought-of test plans & not following test plans

Sounds familiar? Well, the fact is we still encounter them in almost all IDM/IDG projects.


.

OpenAM - http://null:8080/auth/XUI/#login/

At times, like today, I do encounter http://null:8080/auth/XUI/#login/ error when trying to log in to OpenAM.

A quick workaround is to close the browser and open a new browser or go into Incognito mode.

PS: On the browser that I encountered the above issue, I did have another OpenAM session running. But it was pointing to a totally different domain. No time to think straight yet. :) Just a quick note here since I have captured the logs for further investigation.




Updated on 10th March 2016

Peter was referring to Configuration > Authentication > Core.

I tested in my environment. It worked! Thank you! Thank you!


.

OpenAM Fresh Install Issue - HTTP Status 500 - AMSetupFilter.doFilter

Out of laziness, I reused an existing Tomcat instance that had OpenAM 12 installed and configured.

It's fast! 

Stop Tomcat; Remove OpenAM 12 war file and the exploded OpenAM 12 directory; Deploy OpenAM 11 was file; Start Tomcat. 

Bomb! As soon as Tomcat started, I encountered HTTP Status 500 error.

Took me a while to realize that I have not yet cleaned up the hidden OpenAM configuration.

[ec2@am0 ~]$ cd .openamcfg/
[ec2@am0 .openamcfg]$ ls -altr
total 16
-rw-rw-r--.  1 ec2:ec2   25 Mar  1 18:55 AMConfig_data_opt_am1200_webapps_auth_

This file pointed to a configuration directory where OpenAM 12 was previously installed. Now that OpenAM 11 was installed, the above AMSetupFilter.doFilter error appeared.

Stop Tomcat.

[ec2@am0 .openamcfg]$ rm -fr AMConfig_data_opt_am1200_webapps_auth_

Start Tomcat again.

.

OpenDJ with Data at Rest Encryption Solutions Part II

I blogged about OpenDJ with Data at Rest Encryption Solutions few weeks back. The POC was carried out with OpenDJ 2.6.

OpenDJ 3.0 has recently been released and the test equipment (SafeNet Data at Rest Encryption appliance) is still in our office, why not let's run a test again OpenDJ 3.0? :)

In OpenDJ 3.0, there are 2 local DB backends to choose from: JE (Oracle Berkeley DB Java Edition - 2.5 version) and PDB.


The throughtput for search operation is faster in OpenDJ 3.0 as compared to version 2.6. The response times are also slightly faster.


Encrypted partition: 11229 (JE) & 10488 (PDB) vs 10210 (JE)   ~10% diff
Non-encrypted partition: 11305 (JE) & 11235 (PDB) vs 10410 (JE)  ~ 8.5% diff

For write operations, the difference in throughput is not big. In fact, PDB performs worse. (Hmmm... Maybe need to ask my colleague to re-run the test again) ..

Encrypted partition: 2926 (JE) & 2758 (PDB) vs 2750 (JE)   ~6.5% diff
Non-encrypted partition: 3282 (JE) & 2824 (PDB) vs 3199 (JE)  ~ 2.5% diff


The difference in response time is also not big. Again, I might want to re-run the test for PDB again. Or maybe some tuning is required. Not sure, it's something new to me. :)

Encrypted partition: 8.2 (JE) & 8.70 (PDB) vs 8.72 (JE)
Non-encrypted partition: 7.3 (JE) & 8.4 (PDB) vs 7.5 (JE)

.

OpenAM Policy Configuration - High-Availability and Load-Balancing

I realized I blogged this topic before last year - OpenAM Policy Configuration - HA Issue.

One of my customers encountered policy evaluation failure last week due to a fault with one of their OpenDJ servers.

The Policy Configuration (Access Control ++ / (Top Level Realm) ++ Services ++ Policy Configuration) is seldom in use, unless a LDAP Filter condition is configured in Policies (Access Control ++ / (Top Level Realm) ++ Policies)

I'm trying my luck again today to find out a good HA solution for Policy Configuration.


1) The following instructs all OpenAM nodes to communicate with a single OpenDJ server. This is not ideal.

2) The following will not work.

OpenAM debug log will complain with failed to get LDAP server name. If you enter more than one server name in the policy config service's Primary LDAP Server field, please make sure the ldap server name is preceded with the local server name".

3) For multi OpenAM servers, it is better to distribute the load equally to different OpenDJ servers in the backend. However, of course, this doesn't solve the original high-availability/failover issue.

I'm thinking of having 2 entries of the same local server name, each append with a different OpenDJ server. No luck!

Tested. OpenAM only takes the last entry if multiple identical local server name appear. In the above case, only "am2.cdemo.sg:3389" is mapped to "am2.cdemo.sg" (local server name).


The underlying code is such that a local server name can only return a LDAP SERVER entry.

com.sun.identity.policy.PolicyConfig

com.sun.identity.shared.datastruct.CollectionHelper 

Why isn't there a Secondary LDAP Server configuration?


For now, no choice, I think we need a LB in front of the OpenDJ servers.

.

OpenDJ with Data at Rest Encryption Solutions

We just completed a POC for a customer few weeks ago. This is a long time customer using Sun Directory Server and subsequently ported over to ForgeRock OpenDJ.

Recently, due to the strict PDPA requirement from the PDPC (Personal Data Protection Commission Singapore), we are to ensure the user profiles stored in OpenDJ are kept totally safe.

So, few weeks ago, we worked with Gemalto engineers to deploy OpenDJ local DB backend on a partition that is protected by SafeNet Data at Rest Encryption.

Worked like a charm!

I asked my colleage to run a little load-test (6 Connections / 4 Threads) using OpenDJ SDK Toolkit for comparison.

The response time for search operation looks pretty similiar. 

 Now, when it comes to write operations, then there is quite a bit difference in the response time.

Need to be careful in deployment sizing when customer decides on this solution.

.

OpenAM Security Advisory #201601

ForgeRock has just released a security advisory #201601 last night and it is quite a long list - a total of 15 vulnerabilites found in OpenAM 12.0.2, 12.0.1, 11.0.3 and 10.0.2.

I have customers using 12.0.2, 11.0.3 and 10.0.2. Not really good news to me. :(

Details can be found here. Start patching!

OpenDJ Naming Conflicts nsuniqueid=xxx

Just realized I took a long break from blogging. So, I took a long holiday in December to make a family trip to Taiwan. Nice place to be in! In fact, I visit Taiwan frequently. This time round, we went to the eastern side of Taiwan - Taitung area. 

So, we have a long time customer who is using OpenDJ as a centralized user store. Few weeks back, he emailed to notify me that he found some "weird" user entries with DN as such:

dn: nsuniqueid=85c95685-21eb11df-80d5c084-e9b4da2d+uid=stageon238,ou=members,o=XXX.sg

A similar entry was found in the same OpenDJ:

dn: uid=stageon238,ou=members,o=XXX.sg


This is a naming conflict that only happens for replayed operations in a MMR setup.

I am not able to find any documentation from OpenDJ. Here are 2 links that talk about this naming conflict:

1. Resolving Naming Conflicts
2. Solving Naming Conflicts

In short, remove the entry which is older.


.