OpenAM 13.5

ForgeRock released OpenAM 13.5 few weeks ago. Well, to be politically correct, ForgeRock delivered a Mid-Year Platform Minor release few weeks ago. Marketing gimmick! :)

Behind the hook, every single product (OpenAM, OpenDJ, OpenIDM, OpenIG) is still installed individually. Versioning for each product is still not aligned - OpenAM 13.5, OpenDJ 3.5, OpenIDM 4.5, OpenIG 4.5. Why .5 and not the usual increment to .2, then .3? Marketing gimmick! :)

For the first time, ForgeRock is delivering a Mid-Year Platform Minor release that is available only to subscription customers. Non-subscribing community customers can get access to these new features in our next major release. From this point forward, we will ship product 2x per year to get you faster access to the latest features.

As this is a minor release it includes several new and important features that customers can now get access to via BackStage, including: 

• Push Authentication for Passwordless Login and Frictionless Second Factor Authentication 

• Stateless OAuth Token and Server Support 

• Identity Relationship Visualisation 

• Common Audit Event Handlers for ElasticSearch, JMS 

• API Protection (Rate Limiting) 

• Encrypted Database Entries

Let's zoom into OpenAM 13.5. Release Notes here.

The Push Authentication is a nice feature to have. Again, this is the trend these days. An Access Management product without this feature will not look attractive. Technical detail of how Push Authentication works here.

I saw a demo last week and managed to capture a few screens.

The following is a quick overview of how Push Authentication works.

Nice feature! :)

.

RSA SecurID Access Part II

This is what RSA position SecureID Access moving forward - "Secure & convenient access for any user, from anywhere to anything".

The trend these days cannot run away without Adaptive Risk (Risk-based Identity Assurance in RSA SecureID Access) and Cloud integration.

Risk-based Identity Assurance in RSA SecureID Access.

.

RSA SecurID Access

Last week, RSA announced the release of RSA SecurID Access. RSA SecurID Access starts with the RSA SecurID solution at its heart and it combines the feature of the old RSA VIA Access.

By the way, the whole VIA Access product line is now rebranded under SecurID Access.

The plus point is the Multifactor Authentication (MFA) Service to protect cloud and SaaS applications with a variety of mobile optimized authentication methods, which include push notifications, biometrics and FIDO coupled with Identity Assurance.

Identity Assurance is similar to the commonly known Adaptive Risk Authentication, except Identity Assurance currently is not able to track history of users' profiles.

The engineer who demonstrated to me was expecting me to respond with a "Wow! This is innovative!".

But ... I saw something like this few years ago - Salesforce + ForgeRock = Salesforce Identity Connect and ForgeRock Releases Breakthrough Identity Bridge for Cloud Service Providers.

Architecturally, this is nothing new to me.

.

Authentication - MFA and Risk AuthN

I came across an article on Multi-factor Authentication: Best Practices for Securing the Modern Digital Enterprise from Ping Identity Office of the CTO.

There is this nice diagram illustrating the various authentication mechanism.

1. Something you know (for example, a password or a PIN). 
2. Something you have (for example, a mobile phone or a token). 
3. Something you are (for example, a fingerprint or other biometric data).

Generally, combining multiple authentication factors results in a higher Level of Assurance (LoA) that the individual attempting to authenticate is actually the individual in question. Because even if one of the factors has been compromised, the chances of the other factor also being compromised are low. 

Authentication mechanisms can also be distinguished by whether they use the same channel where the user accesses the application, or a separate channel that’s dedicated for authentication.

In the market today, there is yet another type of authentication which is picking up traction - Risk Authentication.

I have this nice diagram from CA Risk Authentication data sheet.

CA Risk Authentication can detect suspicious activity for consumer and enterprise online services without burdening users. This multi-channel risk assessment solution transparently detects and prevents fraud before losses occur.

This usually works together with MFA solutions for Step-Up Authentication.

Technical aside, what's the business case for using MFA and Risk Authentication?

Compliance regulations and industry guidelines are increasing their emphasis on stronger authentication to protect data. Organizations do not want to deploy overbearing authentication systems that require repetitive user interaction because of the negative affect on user experience, which impacts both the adoption of online services and customer loyalty.  

The overall challenge is to detect and block fraudulent activity before losses occur with minimal impact to users.

.

Data Access Governance

Walking the ground helps to keep me abreast of the latest "wants" from customers, especially the ministries. There seems to be a need for Data Access Governance (DAG), in particularly unstructured data.

Interesting!

In most ministries these days, there are a lot of Microsoft SharePoint deployments and File Sharing is mostly using Microsoft technology. Thus there is a serious gap in knowing who has what access to which files/documents.
 

Currently, RSA, SailPoint and Dell One Identity have DAG features in their Governance offering.


.

Quick look at SSO through the ages

I found this article few days back... An interesting map of the road I have travelled so far... :)

Let’s take a quick look at SSO through the ages.

• It started with password synchronization, but that soon became too cumbersome, too labor-intensive, and required too much integration to be a true ”enterprise” solution. 
• Next we had the concept of enterprise SSO where all credentials were stored and the appropriate fields were automatically filled in when login was required. But ESSO doesn’t leverage more modern SSO concepts and is still difficult to implement and manage. 
• Finally we arrived at “true” SSO for Windows with the advent of Active Directory (AD), where a single account and a single credential provides universal access without any synchronization or form-filling. The problem is it only works for Microsoft stuff or things that you can get to play nice with AD, leaving many critical systems out in the cold. 
• Today we have the concept of federation, which is “true” SSO for web applications, but only if those applications talk the right standards, leaving lots of legacy web applications and all thick client apps out of the equation.

Never ending story if there is no common standard. By the way, there won't be any common standard in an ideal world. Contradicting? Yes, my take. 

.

Just a reminder ...

In today's news, 77th Street has announced that it will end its business. Its last shop will close by end of July. 

Founded in 1988, 77th Street was popular among youngsters for its range of streetwear and accessories. Its first outlet was a small shop at Far East Plaza. At its peak, the streetwear chain had 16 stores across Singapore.

"Well, you know, the dream during that time was to open up to 24 or 30 outlets all around Singapore. The rental was then affordable. But as the rental became higher and higher and higher, we find that we're just working for the landlord to pay off the rental,” said the founder.

On the other hand, a convenience shop that sells bargain products near my house has also just decided to call it a day.

It's a timely reminder for my team - Azimuth Labs is .com, but not .org. We are not a charitable organization.

We usually avoid getting into a price war.

Some customers, especially those in this region, have no perception of what they should pay for getting specialist services. They compare us with generic coders/developers that can be found everywhere. The further they source, the cheaper they get.

Some even behave like what they'll do when they go to a wet market.

Sometimes, when I quoted a Change Request for 5 MD (we do not practice putting in buffer in our quotations), they'll negotiate for 3 MD. Other times, when I told them a Change Request is required for certain configuration change which is not quoted in the signed-off SOW, they'll argue that there shouldn't be any additional charge because "no code change what. only configuration change. click, click, click only".

My usual response will be: "Oh, then you do it yourself."

I always illustrate this to my team with "General Practitioner (GP) vs Heart/Eye Specialist". What you pay is what you get. If you are strong enough to not go to a specialist due to the high cost involved, so be it. It's your health. Do not complain if your health deteriorates. But if you do decide to pay a visit to a specialist, you need to pay him/her what he/she is worth.

As simple as that.

PS: In this region, even when I hold my head high up telling people "I am a specialist". They will laugh it off. People who are into IT are not well respected here. This is life. :)




 .

Connectors for Identity Provisioning

We have an intern working for us before he reports for National Service next month. So we gave him some exploration tasks, one of it being to automate our onboarding process using OpenIDM.

New staff on boarding procedures for our office:
1. Create an AD Account
2. Create Zimbra Email Account
3. Create VPN Account
4. Create SVN Account
5. Create Helpscout Account
6. Create Slack Account
7. Create JIRA User
8. Add user to Google calendar

Simple task. Should be able to finish within a month.

No.

Tasks like creating a Zimbra email account, VPN account and SVN account are command-lines operations, thus we need a SSH connector. Took a quick look at https://forgerock.org/openicf/docs/connectors/. Happy! We thought we can workaround with the Solaris Connector.

But no again... the link was broken.

Raised a ticket with Support and was told a new SSH Connector is going to be released soon for OpenIDM 4.0.

Before that happens, what can we do? Generate a flat file using CSV Connector; Run a script to execute the commands on target systems. Less than ideal.

So sometimes, things like this do happen. When you look back the design after implementation, you would like : "OMG! What the hell? Who did that?" Mostly not asking what was the limitation when a design decision was made.

Side track a bit... So we were looking deep into RSA Via Lifecycle & Governance. One thing we like about their architecture was a Mulesoft engine was embedded, which provides the connectivity to external systems. Mulesoft has tons of connectors out there (https://www.mulesoft.com/exchange#!/?types=connector). Ok, not all are Mulesoft-certified, but I would think if one is certified, it would also mean the cost will be different. :)

So, one of my Identity Specialist went crazy and wanted to integrate OpenIDM with Mulesoft. It has what we want - SSH, Slack, JIRA.

Will try. :)

.

Token for Application Access

When dealing with CA API Gateway (aka Layer 7) and CA Single Sign-On (aka SiteMinder), especially how to gain access to the backed protected applications, one has to differentiate the types of token required.

CA API Gateway accepts OAuth2 tokens, while CA SiteMinder traditionally accepts SM Session cookies.


In the mobile world, if there is a hybrid app that needs to communicate with an API-enabled application behind CA API Gateway and with a web application behind CA SiteMinder, it would be easier if both PEP (Policy Enforcement Point) "talk" OAuth2.

It saves the application developers time to handle SM Session cookies.

Is this achievable? Yes, a bit painful.

1) Need a new authentication scheme on SiteMinder to act as OAuth2 client.
2) This OAuth2 client needs to communicate with an OAuth2 Provider, which is the CA API Gateway

* Note: CA SiteMinder cannot act as OAuth2 Provider. Mobile deployment aside, if one has a pure web deployment and requires OAuth2 integration, CA SiteMinder is not able to fulfill the requirement singly. It has to integrate with some other OAuth2 Providers out there.

If customers choose to have everything under the same family, CA API Gateway will satisfy the need. However, it would be a real white elephant if there is no API Management requirement. :)

By the way, my team tested. CA SiteMinder OAuth2 authentication scheme is able to integrate with OpenAM whereby OpenAM acts as the OAuth2 Provider. Works like a charm!

.

Expired CTS Token not cleaned up in OpenDJ

Just came back from a short holiday to Perth. Winter time, so weather was super good!

During the trip, good news come one after another. Today, the greatest good news arrived! The 2nd half of this year will be super busy! :)

Anyway, back to the topic ....

In a clean-up exercise which happened prior to my holiday, we realized expired CTS tokens were not cleaned up in OpenDJ for one of our customers' deployment. OpenAM 11.0.2 was deployed.

From the graph above, it is very obvious that whenever OpenAM is restarted, expired CTS tokens will be cleared up nicely by a Java class in OpenAM - CTS Reaper. However, there is an issue with this class and thus subsequently, expired CTS tokens keeps building up.

I raised a support ticket with ForgeRock and received a prompt response:

I think the more likely culprit for this could be: OPENAM-3283 If so, upgrading to 11.0.3 will resolve this. .... You could diagnose this further by looking at the Session debug log, and also at the CTS Reaper behaviour in your OpenDJ access logs. If you're seeing OPENAM-3283, you would see the CTS Reaper searches (by default once per minute) stop occurring in the access logs. You may also see an exception in the Session log. With full message level debug logging you would be able to see that the CTS Reaper is no longer waking up.

Once I patched OpenAM to 11.0.3, the graph becomes "stable" for the past few days. And it will be for subsequent days to come. :)

PS: Support ticket was raised on 17th May. Response and resolution were given within 2 hours. The graph was for production which I was not given the green light to patch until 11th June.

Personally, I have deal with Support teams from various product principles. Most will route you to a Call Center (some are even outsourced). So the original tickets will then be routed to backend Engineering. This takes forever to resolve an issue. And by the way, before they route you to Engineering, they'll read the product documentation first to determine if your configuration is supported at all. OMG!

ForgeRock Support never fails my high expectation. These guys are awesomely technical. Thank you!

Stable Release

Some customers have this notion that they should start deploying the latest stable release as and when it is made available, even to the extend of making an upgrade during project implementation.

These are the type of customers with no operational experience. They also happen to be the group who has very little product development background. 

On the other hand, I have a customer who is still running OpenAM 10.0.2. We talked about upgrading last November. I told him OpenAM 13.0.0 was scheduled to be released in December. 

Straight-away, he made the decision to request for upgrade budget from Management, but the upgrade job is only to be carried out this August. Why? 

"Never go for the initial release of a major release. I am going to wait for the next minor release to 13.0.0." Wise man! I like working with this type of customer. Same frequency. Some customers question/doubt you when you tell them from your field experience. :)

By the way, OpenAM 13.5 is on the way soon.... 

• Summer 2016 
• Push Authentication offering password-less operation 
• Further Admin Console ease of use work 
• Stateless OAuth2/OIDC

.

ELK . OpenAM . OpenDJ - Part 3

Today, my customer called to alert me that their SSO infrastructure is "not stable". We had OAuth2 Provider configured for this customer. OAuth2 clients' authentication hit into unauthorized error. A little while later, he called me again telling me the Login Page went into an infinite loop. 

But ... but ... everything went back to normal after a while. 

OpenAM/OpenDJ are very stable products from our experience dated way back to Sun Microsystems. They can't go into crash mode out of the sudden. 

In the end, we found out that the network team is tech-refreshing the network. Wow! During office hours? Yes. Well done!

Anyway, until we found out the root cause, the first tool I used was to quickly go to the ELK which we have set up for this customer. Nothing unusual. 

And I blogged about how my team use ELK platform to monitor ForgeRock Open Identity Stack here and here.

"Invalid Password Server Trend Live" - This tracks the user invalid password events.

In my previous blog, I wrote that we noticed huge Invalid Password spike and subsequently identified that it was due to a malfunction automated application. 

Below is a trend for the past 30 days. It was obvious the application went haywire on 29th April and was subsequently rectified on 11th May.

I went to the same dashboard and selected the trend for past 7 days. As observed, on a normal day, the max Invalid Password should be in the 65 range (per OpenAM server).

This is something we are working on right now. With this threshold on mind, we are building a Notification Service to trigger proactive alert to the Ops team. 

.

OpenAM Security Advisory #201604

I slept at almost 2:00 AM last night. I was busying upgrading OpenAM 11.0.2 to OpenAM 11.0.3 for a customer. I woke up receiving an email sent from ForgeRock at 6:47 AM regarding OpenAM Security Advisory #201604.

 

This advisory affects OpenAM 11.0.3 and I'm supposed to patch it! Bang my head. I could have delayed upgrading till next week (since today is Vesak Day). :>

Why IAM Projects Fail? Part II

I woke this morning and received an email from SailPoint. It is the leader in identity governance (IGA) market as named by Gartner for Q2, 2016.

By the way, the underlying engine of SailPoint is Waveset from Sun Microsystems. My team is still supporting a customer not yet migrated out from Waveset. We have a long history implementing products from Sun Microsystems.

I'm fairly surprised Dell One Identity solutions are in the Leaders quadrant. It's not picking traction in this Asia South region. My team is trained in this product as well. Still waiting for deal-on. Ha!


But then, so what if a product is in the top quadrant? Will it warrant implementation success?

I said before and I am repeating now:

In the market, there is really not much difference between the various IDM/IDG products. I can safely say their features are almost 85-90% similar. It's the implementors & customers' key stakeholders ("People") that makes the difference between a successful and failure IAM project.


The magic quadrant is usually used by top-decision makers to cover their backside. Simply put.


I talked about the Pain Points previously. 

Again (coincidentally?), the biggest pain is People. 

• No ownership/main-driver (no full-time PM) 
• Not trained
• Not really know what they really need (Keep changing requirement) 
• Not enough support from application teams 
• No well thought-of test plans & not following test plans

If I may, I would like to add on 2 points which I recently observed.

• Design Documents were signed-off with no intention of following through (this is happening especially frequent in this region & usually causes huge delay)
• Test plans were not vetted and tests were carried out without support from internal team

These were caused by poor leadership. When a system is not well tested, especially the edge cases, things will break during operational time. And it's common. To blame the product, the implementers and the testers is easy. But do remember, when you point a finger at others, at least 3 other fingers are pointing back to yourself.


On point 2 - "Not trained", this could also mean the implementor is not well-trained as well. Oh really? Yes, it does happen. And it happened to us recently!

But being a responsible implementer, do we want it to happen? No. And do we want to redeem ourselves if given a chance? Definitely.

I still remember many years back when I was tasked by Sun Microsystems to debug a Sun Access Manager (the grandfather of OpenAM btw) issue for a teleco in Vietnam. I was given 22 days. In the end, it took me 4 long months! I was new to Sun Access Manager then. I was exploring. I'm not stupid by the way. When you are new to a product, you need time and patient.

Most importantly, my customer believed in me. He did not doubt me. He did not complain to Sun Microsystems behind my back. I felt bad at the delay but he was understanding. With help from support & product team, the issue was finally resolved. Usually, that's what the support & product team requires, an eye onsite to provide accurate feedback to them.


I keep repeating this story to my team these days, especially now when the morale is low.

In computing, there is no such thing called "not-fixable". Any bug can be fixed, it's just a matter of time. But of course, customers need to give you chance.

Otherwise,

To my team-mates: I handpicked you and I trained you, so I believe in you.


.

ELK . OpenAM . OpenDJ - Part 2

I blogged about how we use ELK to monitor trends, especially abnormal ones, on ForgeRock Identity Stack. 

"Login Failed Server Trend Live" - This tracks the user login failure events. 

Just a few days ago, we observed that the number of Login Failed events had increased.

So, we zoomed in and found out there were many lines of the following errors:

"2016-05-17 00:09:00"   "Login Failed|module_instance|Application"      "Not Available" golfdigest 202.xx.xx.xx   INFO    o=xxx.sg  AUTHENTICATION-268     "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" Application     202.xx.xx.xx"


I know if module_instance is Application, then this is not a user authentication. Most likely, it is a Policy Agent in action. By the way, every Policy Agent will need to authentication with OpenAM in order to pull the policies.

So based on the IP address (202.xx.xx.xx), we found out the owner of the application. Ah! This is a defunct site. The SSO administrator has already removed the Policy Agent "golfdigest" from OpenAM as part of the sunsetting process, but the network team has not disabled/removed the Policy Agent on the Apache web server yet.

We have many more useful dashboard in Kibana, which we use for operational purpose.  I'll share later.


.

Ansible . OpenAM . OpenDJ . OpenIDM

For some of our larger deployment of ForgeRock Identity Stack, we will usually request to install Ansible on the development node.

For pure Linux environment, it works like a charm once SSH keys are exchanged during initial setup. (For Windows, it is supported but slightly more complicated to set up)

What do we use Ansible for? Almost every operational task.

Change configuration files; Update custom codes; Update JSP/UI pages; Deploy patches from ForgeRock; Restart Apache/Tomcat servers ... anything.

We used to make human errors occasionally when we managed a farm of over 10 OpenAM and 6 OpenDJ servers. Now that everything is pushed from development node, there is hardly any human error.


I just saw Ansible Tower on Ansible website.

No urgent need to use this feature. Our current Ansible already has logging/audit in place.

Anyway, once a playbook script is tested, there is hardly any error when it is being executed.

.

ELK . OpenAM . OpenDJ

We have ELK (Elasticsearch, Logstash and Kibana) deployed for a long time OpenAM/OpenDJ customer of ours some time back. The idea is not new. Similar solutions have been deployed by some other ForgeRock folks/partners.

We know ELK can only keep trends, but not send notification. (Ok ok, Elastic does offer Watcher for its commercial version) What we intend to do is to add Notification Service side-by-side with ELK. No, we do not intend to keep all data from OpenAM/OpenDJ in Elastic and to trigger alert from there. Some data are not useful to keep in Elastic (e.g. total entries count from all OpenDJ to determine if replication is operating optimally). We just need a simple cache layer (e.g. Ehcache) to keep these types of "data-in-transit" in order to trigger alerts to administrators/operators.

I'll talk more about this next time.

But so far, how useful is ELK to customer? Pretty good feedback.


"Login Failed Server Trend Live" - This is a live trend where by Logstash agents send "live" data from all OpenAM servers by monitoring amAuthentication.error logs. This tracks the user login failure events. 

If the user login failure count is high for a particular day for a particular OpenAM node, we can zoom into amAuthentication.error log to find out more.

"Invalid Password Server Trend Live" - This tracks the user invalid password events. 

This trend is different from the previous. An Invalid Password event happens when a user id is correct, but password is invalid. 

"2016-05-10 13:00:43"   "Invalid Password"      "Not Available" UID=ntustc001,ou=XXXX,o=xxx.sg    202.83.xx.xxx   INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"       "Not Available" LDAP    202.83.xx.xxx

A quick zoom into amAuthentication.error log reveals a particular user was attempting to log in with an invalid password.

[amuser@f1]$ cat amAuthentication.error.20160510 | wc -l
18475

[amuser@f1]$ cat amAuthentication.error.20160510 | grep -i ntustc001 | wc -l
18039


A total of >18k invalid login attempts. That's quite unusual.

This is where the customer service personnel can call up their paying customer to find out what exactly happened and if he/she requires a password reset service.

Proactive customer engagement model!



By the way, if you look at amAuthentication.error in-depth, you might see some Chinese characters like 登录失败  (Login Failed) and 无效密码 (Invalid Password). These are traffic from Chinese locale browsers.


"2016-05-10 07:00:33"   登录失败        "Not Available" "Not Available" 118.176.xx.xxx  INFO    o=xxx.sg  AUTHENTICATION-200      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    118.176.xx.xxx

"2016-05-10 08:56:22"   无效密码        "Not Available" uid=A480,ou=xxx,o=xxx.sg  
     202.4.xxx.xx    INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    202.4.xxx.xx

LDIF Delta Utility

This summary is not available. Please click here to view the post.

Software Support

Recently, we hit into a production issue and were not able to resolve by ourselves. Thus we raised a support ticket on behalf of our customers. The support engineer needed the core dump when the issue re-surfaced again. So, he suggested to use Process Explorer or Debug Diagnostic tool.

Sure, no problem. But he added: "... If you encounter problem on dump capture, can you please engage Microsoft to assist as both process explorer and debug diagnostic tool are provided by Microsoft to capture dump."

Oh well, you were the one who asked customer to use tools from Microsoft to help you with debugging. Now, if that suggested tool(s) has issue, we need to raise a separate ticket with Microsoft?


Few weeks ago, we raised another ticket to the same engineer. By the way, he is support engineer for Product A in this company. The ticket was about how to integrate Product A with Product B from the same company.

To our astonishment, he responded with the following:

"It seems like some configuration issue on Product B. I suggest you open a ticket with Product B to check the configuration. Beside, from Product A documentation that you pointed out, I didn't find the OAuth provider can be Product B (Facebook and Google are the OAuth Provider mentioned). Therefore, I'm not sure if this can be done with Product B. If you have any additional documentation or details on how to integrate Product A with Product B as OAuth Provider, please share with me."

Hello, who is the customer? If you are not sure, you jolly well walk over (or skype/email) to Product B support team and find out more from them. You asking customer to bridge the communication gap between your two support teams? This is embarrassing.

You better wake up!


Side note: Besides ForgeRock products, my team delivers IAMS products from other principles as well.

OpenAM & Facebook Business Manager

We know OpenAM supports OAuth 2.0/OpenID Connect authentication module. OpenAM provides a wizard for configuring common OAuth 2.0/OpenID Connect authentication providers, such as Facebook, Google, and Microsoft.

In most use cases, for example Facebook, customers will go to Facebook Developers and create a new app for the company. 

The whole company will then use a common Facebook App ID.

Now, what if a company has multiple customer-facing websites? Each business unit owner would like to have their own Facebook Analytics for Apps to understand how people access their websites. 

This is where Facebook Business Manager comes in handy.

 

2 things to change:
1. Create 1 OAuth 2.0/OpenID Connect authentication module for each sub-account
2. Change FB Login icon on each website to authenticate with the approriate authentication module created in Step 1.


.