OpenDJ vs OUD - Part II

I came across this comparison chart which clearly illustrates how the ForgeRock Open Identity Stack has evolved after Sun Microsystems sun-set.

The following chart attempts to compare OpenDJ against Directory Services offering from Oracle.

Oracle has 3 different directory products to choose from. The Sun OpenDS code base provides the foundation for both Oracle Unified Directory and OpenDJ which means all the advantages of the Oracle product can be found in OpenDJ as well - Internet scalability, HA, and support for use cases for the enterprise and cloud - except OpenDJ is 100% open source with an actively and rapidly contributing community and the flexibility to customize code.

Not forgetting the fact which I have mentioned in my previous blog - OpenDJ vs OUD - Which do you need? - key team members from OpenDS projects are now in ForgeRock.

So who will have the more superior product over time?

The choice is really yours...


.

Replication Gateway - Part III

I blogged about OUD providing a Replication Gateway (note: this is different from Replication Server) to help replicate data two-ways from Sun DSEE to Oracle OpenDS or Unified Directory.

And I mentioned that there is in fact an alternative to replicate data from Sun DSEE to ForgeRock OpenDJ - which is using LDAP Synchronization Connector.

I was in ForgeRock Open Identity European Summit in Paris few weeks back and someone shared with me some bad experiences customers had when they used the Replication Gateway. I was told it did not work so beautifully. Strangely, directory schema was replicated!

Err… isn't the directory schema in the new OpenDS (aka OUD) not meant to be the same as the old Sun DSEE? OpenDS is supposed to be a revamp, but an upgrade for Sun DSEE.


.

OpenSSO with embedded OpenDS for high-volume traffic - Part II

I managed to capture 2 graphs taken during the load-testing sessions.

With embedded OpenDS in a 12-nodes OpenSSO environment, there will be consistent spikes happening at 15-20 mins interval and subsequently 25 mins ... (that was when I issued a thread dump and realized there were numerous threads related to OpenDS replication service)

Ideally, we would prefer the following graph. Consistently flat. Spikes at 15-min and 26-mins though, but minor enough to ignore. Still trying to further tune.

PS: Getting the almost flat graph above is not totally contributed by moving the embedded OpenDS to external Sun Directory Server. There were some other tuning made. 

.

OpenSSO with embedded OpenDS for high-volume traffic

I was performing another round of performance load-testing for a customer in Singapore last week. The target was to achieve 10,000 concurrent hit per second (yes, it is a tall order) and the load must be sustained for 1 hour.

The setup was a pretty old version OpenSSO 8 Update 2 Patch 4. The embedded OpenDS was even older. It was still OpenDS 1.0.0. Why no upgrade? Well, better dun ask. Political lah.


We have, in fact, performed many rounds of tuning - OS kernel, TCP stack, Web Servers (reverse proxy), Application Servers, JVM options - prior to last week's activity.

We were almost there, except failing for 1-2 secs at times. Spikes kept coming after 15-20 mins; went away; came back again 10-15 mins later ...


So during one of the spikes, I executed a thread dump. And the following was observed:

At that point of time, we had 12 x OpenSSO servers running, with embedded OpenDS. Me bad, I know. Lazy bum me!

No choice, I made a decision to switch out from 12 x embedded OpenDS to 3 x external Sun Directory Server 7. Political again, dun ask me why you never this.. never that.. :)

Immediately, we could see the effect. The NLWP (Number of Light-Weight Processes) decreased from high-300 to mid-200 upon application servers restart. Pretty good improvement.

Subsequent load-testings also yield better result with less spikes. The NLWP never increased beyond 650 at peak (previously it could easily reached 900+).

At least, the graphs looked flatter.

We are now on our last mile... I think the reverse-proxy servers are over-loaded. Adding more hardware at the moment... Hopefully, we can achieve the expected result in a week or 2!



PS: My friends at ForgeRock also confirm that a deployment of  >4 OpenAM instances with external OpenDJ as the configuration store yields better performance. It is largely due to the meshed replication setup. Also, using the latest version of OpenDJ (2.6.0 at the moment) will help as well. OpenDS is far too old. :)


.

Replication Gateway

I was with a consultant from South Korea this week. So he kept me updated with the technical development in Korea. 

I was told that the latest hype is Oracle Unified Directory (OUD). This used to be Sun OpenDS. 

Surprisingly, he told me that OUD is able to replicate data both ways with Oracle Directory Server Edition Edition (ODSEE). This used to be Sun Directory Server Edition Edition (Sun DSEE).

As I am quite familiar with ForgeRock OpenDJ (which was also from Sun OpenDS), I know that replication both ways is not possible due to the fact that Sun DSEE replication protocol is proprietary. 

Well, but OUD and ODSEE do know each other since they are from the same family. So what if the replication protocol is proprietary? Let's make it even more proprietary! :)

So, here we go ..... let's introduce Replication Gateway! This will bridge the replication issue between ODSEE and OUD.

.

OpenDJ: Replication Setup Error Message

Today, I hit into some issues with OpenOSSO multi-servers setup. I am suspecting there's something not correct with the OpenDS configuration in a multi-servers deployment. Nothing concrete yet, am still investingating.

In order to provide some facts, I think setting 2 instances of OpenDJ in MMR mode would be good. At least, I have something to compare with.

So, the installation of OpenDJ was a breeze. (Do I re-broadcast so many times on this fact? :>)


Setting up replication is not hard if you have the following command line as a template. (No GUI yet to set up replication.)

[opendj@think bin]$ ./dsreplication enable --host1 think.sg.forgerock --port1 8888 --bindDN1 "cn=directory manager" --bindPassword1 password --replicationPort1 8989 --host2 think.sg.forgerock --port2 7888 --bindDN2 "cn=directory manager" --bindPassword2 password --replicationPort2 8989 --adminUID admin --adminPassword password --baseDN "dc=sg,dc=forgerock" -X -n


Establishing connections ..... Done.


You have provided the same replication port (8989) for two servers located on the same machine (think.sg.forgerock).


So, am I done or not done? Initially, I thought I'm done. But ... actually, it's not done.

Wouldn't the following be better?

ERROR: You have provided the same replication port (8989) for two servers located on the same machine (think.sg.forgerock).


.

Sun Software Product Map - Part III

If you take a good look at the list, you'll notice that OpenDS is missing.

FYI, there is a commercial version of OpenDS - Sun OpenDS Standard Edition.

From what I know, Oracle is still defining the strategy for OpenDS. Until then, we do not know what will happen to OpenDS.

A lot of development work has been poured into OpenDS. It's a total rewrite in Java from Sun Java System Directory Server. I'm keeping my fingers crossed.


By the way, Directory Service is another strength of us. So we'll be watching closely on the development of OpenDS.


.

Fastest Virtual Directory - from Symlabs

I came across Symlabs's website today. Symlabs provides Identity Management Solutions.

Symlabs Virtual Directory Server is ranked #1 in the world. Pretty impressive!

Well done!

OpenDS 2.0 Installation on MacBook

OpenDS 2.0 has been released for 2 weeks. I did not have the time until now to install a copy on my laptop for self-discovery purpose.

As long as you have JDK installed on MacBook, installation is a breeze. It took me less than 5 mins to get a decent instance up and running.

cheechongs-macbook:OpenDS-2.0.0 cheechong$ ./setup --cli

OpenDS Directory Server 2.0.0

Please wait while the setup program initializes...

What would you like to use as the initial root user DN for the Directory

Server? [cn=Directory Manager]:     

Please provide the password to use for the initial root user:         

Please re-enter the password for confirmation:         

On which port would you like the Directory Server to accept connections from

LDAP clients? [1389]: 

On which port would you like the Administration Connector to accept

connections? [4444]: 8888 <-- I dun like 4444. I'll use 8888 instead.

What do you wish to use as the base DN for the directory data?

[dc=example,dc=com]: dc=sg,dc=sun

Options for populating the database:

    1)  Only create the base entry

    2)  Leave the database empty

    3)  Import data from an LDIF file

    4)  Load automatically-generated sample data

Enter choice [1]: 4

Please specify the number of user entries to generate: [2000]: 

Do you want to enable SSL? (yes / no) [no]: yes

On which port would you like the Directory Server to accept connections from

LDAPS clients? [1636]: 

Do you want to enable Start TLS? (yes / no) [no]: 

Certificate server options:

    1)  Generate self-signed certificate (recommended for testing purposes

        only)

    2)  Use an existing certificate located on a Java Key Store (JKS)

    3)  Use an existing certificate located on a JCEKS key store

    4)  Use an existing certificate located on a PKCS#12 key store

    5)  Use an existing certificate on a PKCS#11 token

Enter choice [1]: 

Do you want to start the server when the configuration is completed? (yes /

no) [yes]: 

Setup Summary

=============

LDAP Listener Port:            1389

Administration Connector Port: 8888

LDAP Secure Access:            Enable SSL on LDAP Port 1636

Create a new Self-Signed Certificate

Root User DN:                  cn=Directory Manager

Directory Data:                Create New Base DN dc=sg,dc=sun.

Base DN Data: Import Automatically-Generated Data (2000 Entries)

Start Server when the configuration is completed

What would you like to do?

    1)  Setup the server with the parameters above

    2)  Provide the setup parameters again

    3)  Cancel the setup

Enter choice [1]: 

Configuring Directory Server ..... Done.

Configuring Certificates ..... Done.

Importing Automatically-Generated Data (2000 Entries) ....... Done.

Starting Directory Server ......... Done.

See /tmp/opends-setup-17177.log for a detailed log of this operation.

To see basic server configuration status and configuration you can launch /Users/cheechong/Documents/Technical/OpenDS-2.0.0/bin/status

To verify:

cheechongs-macbook:bin cheechong$ ./status 

>>>> Specify OpenDS LDAP connection parameters

Administrator user bind DN [cn=Directory Manager]: 

Password for user 'cn=Directory Manager':         

 

--- Server Status ---

Server Run Status:        Started

Open Connections:         1

--- Server Details ---

Host Name:                       cheechongs-macbook.local

Administrative Users:     cn=Directory Manager

Installation Path:             /Users/cheechong/Documents/Technical/OpenDS-2.0.0

Version:                             OpenDS Directory Server 2.0.0

Java Version:                    1.5.0_16

Administration Connector: Port 8888 (LDAPS)

--- Connection Handlers ---

Address:Port : Protocol : State

-------------:----------:---------

--                   : LDIF        : Disabled

0.0.0.0:161  : SNMP      : Disabled

0.0.0.0:1389 : LDAP     : Enabled

0.0.0.0:1636 : LDAPS   : Enabled

0.0.0.0:1689 : JMX      : Disabled

--- Data Sources ---

Base DN:     dc=sg,dc=sun

Backend ID:  userRoot

Entries:     2002

Replication: Disabled

This installation is far easier and faster than Sun Directory Server Enterprise Edition installation.

How to monitor Sun Directory Server 5.2?

Yes, there are still a lot of Sun Directory Server 5.2 deployment in Asia region. (even though 6.x has been released for quite a while)

I was meeting a customer today. He is interested in migrating from Sun Directory Server 5.2 to Sun OpenDS Standard Edition 2.0 (target release date is Mid July 2009). 

Why Sun OpenDS SE 2.0 and not Sun Directory Server 6.3.1? I was told the price for commercial support differ by quite a lot, at least S$12k for his case. Wow!

Anyway, that's not my objective. I'm providing Professional Service/Consultancy. I'll leave the pricing to the folks.

So, I asked during the meeting when is the peak period and what is the concurrent hit like? I was given figures ranging from 70-80k per hour to 5 millions hit per day. 

I was not convinced. Thus I asked for facts which can be easily extracted with the tools provided by Sun Directory Server 5.2.

You can use the GUI to monitor the suffix status online. (Click on "Continuous refresh") Otherwise, for people like me, command-line is best.

bash-3.00# ldapsearch -D "cn=Directory Manager" -w password -b "cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config" objectclass=*

version: 1

dn: cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config

objectClass: top

objectClass: extensibleObject

cn: monitor

database: ldbm database

readonly: 0

nextid: 10

dbentrycount: 9

ldapentrycount: 9

entrycachehits: 26

entrycachetries: 36

entrycachehitratio: 72

currententrycachesize: 14796

maxentrycachesize: 10485760

currententrycachecount: 9

maxentrycachecount: -1

 

Remember to execute the above command over a period of time in order to capture the overall statistics. A cron job will help.

 

  

How to scale Directory Server?

Again, the following typical question is asked whenever I provide Directory Service consultancy:

How do we scale our Directory Service?

There are actually 2 considerations - READ and WRITE operations.

The answer to READ operation is very simple. 

1. Add more nodes;
2. Configure replication across all nodes;
   
3. Place a load-balancer in-front of all nodes;

Done. Simple!

The answer to WRITE operation is far more complex.

• If the solution is to cater for more WRITE than READ, then I'll ask "Are you actually looking for a database?" (LDAP is built for fast and frequent read ops, with occasionally write ops.)

• You'll be surprised at the answers you get. There are really customers who cannot differentiate between LDAP and database.

• If the customer is really sure they need Directory Service, then I'll suggest that they spilt their users into various organizations/groups. Then each logical group will be stored in different LDAP sources.
  

Quoted from OpenDS site:

Note that you cannot use replication to scale write operations because a write operation to one directory server results in a write operation to every other server in the topology. The only way to scale write operations horizontally is to split the directory data among multiple databases and place those databases on different servers.

Corrupted database exception in Sun Directory Server

I am reading OpenDS System Requirement in preparation for a presentation tomorrow to a local teleco in Singapore.

The section on File Descriptor Requirements (Linux Systems) reminds me of an escalation case I was assigned to in Feburary.

The case involved a global logistics company in Malaysia. It has an issue with Sun Directory Server 5.2 SP4 running on Redhat Enterprise Linux Server AS 4. Long story short - the problem was due to insufficient tuning.

Linux systems limit the number of file descriptors that any one process may open to 1024 per process.

After the directory server has exceeded the file descriptor limit of 1024 per process, any new process and worker threads will be blocked.

For example, if the directory server attempts to open a Oracle Berkeley JE database file when the operating system has exceeded the file descriptor limit, the directory server will no longer be able to open a connection, which results in a corrupted database exception. Likewise, if you have a directory server that exceeds the file descriptor limit set by the operating system, the directory server can become unresponsive as the LDAP connection handler consumes all of the CPU's processing in attempting to open a new connection.

To fix this condition, set the maximum file descriptor limit per process on Linux
machines.