ELK . OpenAM . OpenDJ - Part 3

Today, my customer called to alert me that their SSO infrastructure is "not stable". We had OAuth2 Provider configured for this customer. OAuth2 clients' authentication hit into unauthorized error. A little while later, he called me again telling me the Login Page went into an infinite loop. 

But ... but ... everything went back to normal after a while. 

OpenAM/OpenDJ are very stable products from our experience dated way back to Sun Microsystems. They can't go into crash mode out of the sudden. 

In the end, we found out that the network team is tech-refreshing the network. Wow! During office hours? Yes. Well done!

Anyway, until we found out the root cause, the first tool I used was to quickly go to the ELK which we have set up for this customer. Nothing unusual. 

And I blogged about how my team use ELK platform to monitor ForgeRock Open Identity Stack here and here.

"Invalid Password Server Trend Live" - This tracks the user invalid password events.

In my previous blog, I wrote that we noticed huge Invalid Password spike and subsequently identified that it was due to a malfunction automated application. 

Below is a trend for the past 30 days. It was obvious the application went haywire on 29th April and was subsequently rectified on 11th May.

I went to the same dashboard and selected the trend for past 7 days. As observed, on a normal day, the max Invalid Password should be in the 65 range (per OpenAM server).

This is something we are working on right now. With this threshold on mind, we are building a Notification Service to trigger proactive alert to the Ops team. 

.

ELK . OpenAM . OpenDJ - Part 2

I blogged about how we use ELK to monitor trends, especially abnormal ones, on ForgeRock Identity Stack. 

"Login Failed Server Trend Live" - This tracks the user login failure events. 

Just a few days ago, we observed that the number of Login Failed events had increased.

So, we zoomed in and found out there were many lines of the following errors:

"2016-05-17 00:09:00"   "Login Failed|module_instance|Application"      "Not Available" golfdigest 202.xx.xx.xx   INFO    o=xxx.sg  AUTHENTICATION-268     "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" Application     202.xx.xx.xx"


I know if module_instance is Application, then this is not a user authentication. Most likely, it is a Policy Agent in action. By the way, every Policy Agent will need to authentication with OpenAM in order to pull the policies.

So based on the IP address (202.xx.xx.xx), we found out the owner of the application. Ah! This is a defunct site. The SSO administrator has already removed the Policy Agent "golfdigest" from OpenAM as part of the sunsetting process, but the network team has not disabled/removed the Policy Agent on the Apache web server yet.

We have many more useful dashboard in Kibana, which we use for operational purpose.  I'll share later.


.

ELK . OpenAM . OpenDJ

We have ELK (Elasticsearch, Logstash and Kibana) deployed for a long time OpenAM/OpenDJ customer of ours some time back. The idea is not new. Similar solutions have been deployed by some other ForgeRock folks/partners.

We know ELK can only keep trends, but not send notification. (Ok ok, Elastic does offer Watcher for its commercial version) What we intend to do is to add Notification Service side-by-side with ELK. No, we do not intend to keep all data from OpenAM/OpenDJ in Elastic and to trigger alert from there. Some data are not useful to keep in Elastic (e.g. total entries count from all OpenDJ to determine if replication is operating optimally). We just need a simple cache layer (e.g. Ehcache) to keep these types of "data-in-transit" in order to trigger alerts to administrators/operators.

I'll talk more about this next time.

But so far, how useful is ELK to customer? Pretty good feedback.


"Login Failed Server Trend Live" - This is a live trend where by Logstash agents send "live" data from all OpenAM servers by monitoring amAuthentication.error logs. This tracks the user login failure events. 

If the user login failure count is high for a particular day for a particular OpenAM node, we can zoom into amAuthentication.error log to find out more.

"Invalid Password Server Trend Live" - This tracks the user invalid password events. 

This trend is different from the previous. An Invalid Password event happens when a user id is correct, but password is invalid. 

"2016-05-10 13:00:43"   "Invalid Password"      "Not Available" UID=ntustc001,ou=XXXX,o=xxx.sg    202.83.xx.xxx   INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"       "Not Available" LDAP    202.83.xx.xxx

A quick zoom into amAuthentication.error log reveals a particular user was attempting to log in with an invalid password.

[amuser@f1]$ cat amAuthentication.error.20160510 | wc -l
18475

[amuser@f1]$ cat amAuthentication.error.20160510 | grep -i ntustc001 | wc -l
18039


A total of >18k invalid login attempts. That's quite unusual.

This is where the customer service personnel can call up their paying customer to find out what exactly happened and if he/she requires a password reset service.

Proactive customer engagement model!



By the way, if you look at amAuthentication.error in-depth, you might see some Chinese characters like 登录失败  (Login Failed) and 无效密码 (Invalid Password). These are traffic from Chinese locale browsers.


"2016-05-10 07:00:33"   登录失败        "Not Available" "Not Available" 118.176.xx.xxx  INFO    o=xxx.sg  AUTHENTICATION-200      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    118.176.xx.xxx

"2016-05-10 08:56:22"   无效密码        "Not Available" uid=A480,ou=xxx,o=xxx.sg  
     202.4.xxx.xx    INFO    o=xxx.sg  AUTHENTICATION-201      "cn=dsameuser,ou=DSAME Users,o=xxx.sg"    "Not Available" LDAP    202.4.xxx.xx