OpenAM 11.0.0 Upgrade - Persistent Search Error

In OpenAM 11, the Identity Repositories (IdRepo) has been revamped. The new implementation is using OpenDJ LDAP SDK to perform LDAP operations. 

(If you want to know more about IdRepo, read from Peter's blog - Identity Repositories. He knows IdRepo inside out. :> )

I observed two errors when I configured an external OpenDJ and Microsoft Active Directory as additional Data Store.

If you are connecting to an external OpenDJ server (e.g. OpenDJ 2.6.0), the following error is frequently displayed in IdRepo:

PersistentSearch:12/11/2013 03:52:06:353 PM SGT: Thread[OpenDJ LDAP SDK Default Scheduler,5,main]ERROR: An error occurred while executing persistent searchorg.forgerock.opendj.ldap.ConnectionException: Server Connection Closed: Heartbeat timed out after 500 ms        at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:163)        at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:125)        at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:76)        at org.forgerock.opendj.ldap.HeartBeatConnectionFactory.newHeartBeatTimeoutError(HeartBeatConnectionFactory.java:1326)        at org.forgerock.opendj.ldap.HeartBeatConnectionFactory.access$000(HeartBeatConnectionFactory.java:107)        at org.forgerock.opendj.ldap.HeartBeatConnectionFactory$ConnectionImpl.checkForHeartBeat(HeartBeatConnectionFactory.java:816)        at org.forgerock.opendj.ldap.HeartBeatConnectionFactory$ConnectionImpl.access$2300(HeartBeatConnectionFactory.java:238)        at org.forgerock.opendj.ldap.HeartBeatConnectionFactory$1.run(HeartBeatConnectionFactory.java:1103)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)        at java.util.concurrent.FutureTask.run(FutureTask.java:138)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:662)

Of course, restarting process immediately kicks-in:

PersistentSearch:12/11/2013 03:52:06:354 PM SGT: Thread[OpenDJ LDAP SDK Default Scheduler,5,main]Restarting persistent search connection against: LoadBalancer(Failover(AuthenticatedConnectionFactory(HeartBeatConnectionFactory(LDAPConnectionFactory(xxx.yyy.sg/192.168.x.x:1389)), SimpleBindRequest(name=cn=Directory Manager, authentication=simple, controls=[]))))PersistentSearch:12/11/2013 03:52:07:378 PM SGT: Thread[SystemTimerPool,5,main]Starting persistent search against baseDN: dc=azlabs,dc=sg, scope: sub, filter: (objectclass=*), attrs: [dn] against LoadBalancer(Failover(AuthenticatedConnectionFactory(HeartBeatConnectionFactory(LDAPConnectionFactory(xxx.yyy.sg/192.168.x.x:1389)), SimpleBindRequest(name=cn=Directory Manager, authentication=simple, controls=[]))))

If you are connecting to an external Microsoft Active Directory server, the following error is frequently displayed in IdRepo:

PersistentSearch:12/11/2013 04:06:04:265 PM SGT: Thread[OpenDJ LDAP SDK Connection Timeout Checker,5,main]ERROR: An error occurred while executing persistent searchorg.forgerock.opendj.ldap.TimeoutResultException: Client-Side Timeout        at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:182)        at com.forgerock.opendj.ldap.AbstractLDAPFutureResultImpl.setResultOrError(AbstractLDAPFutureResultImpl.java:125)        at com.forgerock.opendj.ldap.AbstractLDAPFutureResultImpl.adaptErrorResult(AbstractLDAPFutureResultImpl.java:114)        at com.forgerock.opendj.ldap.LDAPConnection.cancelExpiredRequests(LDAPConnection.java:562)        at com.forgerock.opendj.ldap.TimeoutChecker$2.run(TimeoutChecker.java:92)

Again, restarting process immediately kicks-in:

PersistentSearch:12/11/2013 04:06:37:347 PM SGT: Thread[OpenDJ LDAP SDK Connection Timeout Checker,5,main]Restarting persistent search connection against: LoadBalancer(Failover(AuthenticatedConnectionFactory(HeartBeatConnectionFactory(LDAPConnectionFactory(ad1.XXXX.sg/192.168.xx.yyy:389)), SimpleBindRequest(name=CN=SSO Service Account,CN=Users,DC=XXXX,DC=sg, authentication=simple, controls=[]))))PersistentSearch:12/11/2013 04:06:38:373 PM SGT: Thread[SystemTimerPool,5,main]Starting persistent search against baseDN: DC=XXXX,DC=sg, scope: sub, filter: (objectclass=*), attrs: [dn, isDeleted, whenChanged, whenCreated] against LoadBalancer(Failover(AuthenticatedConnectionFactory(HeartBeatConnectionFactory(LDAPConnectionFactory(ad1.XXXX.sg/192.168.xx.yyy:389)), SimpleBindRequest(name=CN=SSO Service Account,CN=Users,DC=XXXX,DC=sg, authentication=simple, controls=[]))))

Why? I do not know. I hope improvement can be made to this.

.

Active Directory Authentication Error via OpenSSO

We have configured OpenSSO Authentication Service to perform authentication via Microsoft Active
Directory.

We know that Microsoft Active Directory is able to detect first-time-login, password expired, account locked, account disabled etc ...

However, by using the in-built Active Directory Authentication module in OpenSSO, it keeps displaying the same error "Invalid credentials" whenever any of the errors is encountered. This confuses the users a lot. It also gives administrator wrong impression of what exactly is the error.


If a manual search is performed, the following is what we get:

bash-3.00# ldapsearch -h 192.168.131.50 -p 389 -D "cn=cclow,cn=users,dc=central,dc=sg,dc=sun" -b "dc=central,dc=sg,dc=sun" -s sub "objectclass=*"
Enter bind password:
ldap_simple_bind: Invalid credentials
ldap_simple_bind: additional info: 80090308: LdapErr: DSID-0C0903AA, comment: AcceptSecurityContext error, data 773, v1771


bash-3.00# ldapsearch -h 192.168.131.50 -p 389 -D "cn=cclow,cn=users,dc=central,dc=sg,dc=sun" -b "dc=central,dc=sg,dc=sun" -s sub "objectclass=*"
Enter bind password:
ldap_simple_bind: Invalid credentials
ldap_simple_bind: additional info: 80090308: LdapErr: DSID-0C0903AA, comment: AcceptSecurityContext error, data 533, v1771


Microsoft Active Directory has this habit of sending back error messages in 2 lines. The 1st contains what I call it General Error Message. The 2nd will contain the Actual Error Message ("additional info").

In this 2nd line, you need to tokenizes the message to grab the part that contains ", data xxx,". This will give you the Exact Error Message.

ldap_simple_bind: additional info: 80090308: LdapErr: DSID-0C0903AA, comment: AcceptSecurityContext error, data 533, v1771

So we went ahead to develop our own custom Active Directory Authentication module, and we have the following mapping:

public static final String ERROR_FIRSTTIME = "773";
public static final String ERROR_PASSWORDEXPIRED = "532";
public static final String ERROR_ACCOUNTLOCKED = "775";
public static final String ERROR_ACCOUNTDISABLED = "533";
public static final String ERROR_ACCOUNTEXPIRED = "701";


.

Microsoft Exchange business is good

I was reading an article in The Microsoft Exchange Team Blog on the Release Candidate of Exchange 2010. Read here.

One interesting paragraph caught my attention.

According to a 2008 Ferris research report, Notes/Domino share has dwindled to a mere 10% in enterprises, while Exchange has grown to 65% market share across all organizations and continues to grow with more than 4.7M starting the switch to Exchange last year. In fact, Exchange is now is approaching $2B in annual revenues. If Exchange were a standalone business, it would be the 9th largest software company in the world. We expect that momentum to accelerate with Exchange 2010, the most compelling version yet.

Wow!

What do customers look for when they want a Portal solution?

In my conversations with customers, their requirement for a Portal solution is fairly simple.

(OK, in the Asia-South region. I do not know about other region)

Must-have

0. Friendly UI

1. Integrated Content Management Server (CMS)

2. Integrated Document Management Server (DMS)

3. Versioning feature

4. Approval Process

5. Search Engine that can index all types of documents (especially Words and PDF)

6. Document Conversion (see here)

7. LDAP authentication (with their existing infrastructure)

Good-to-have

1. Integrated Email and Calendar Services

2. Instant Messaging

3. Different theme for different department/branch

Notice that they are not very much into Single Sign-On or integration with other backend applications. This, to me, is not appealing at all. There is no stickiness in this solution.

A good Portal solution should integrate with as many useful backend applications as possible. This will make end-users come back again and again.

A Portal should be a one-stop place for everyone.

Mail Journaling Solution - Part II

After my post on Mail Journaling Solution, I continue to search for more solution.


My objective is simple:

1. It has to be simple architecturally;
2. It has to be value-for-money;
3. Investment should be progressive, rather than huge up-front

It's in my character to think in the shoes of my customers when I introduce a product to them. That explains why I do not like to pay too much up-front, even though the money does not belong to me. Yes, revenue-wise it would be good for the company. However, I think I need to take a higher priority on my customers' interest.

When they grow big or even bigger, they'll always come back for us. So why worry about no recurring business?



Today, I found another interesting product from deepinvent Software GmbH - MailStore Server.





This product works very well with Microsoft Exchange. It is ideal for small companies. Actually it should work well for larger setup if the architecture is deployed correctly, my opinion.


So how about Sun Java Messaging Server?

MailStore Server comes with another component called MailStore Proxy. It can be deployed in front of the Sun Messaging Server.




The only change is to simply point your email client to this Proxy. All in-coming and out-going emails will now flow through the Proxy, thus achieving mail journaling objective.

Customer can choose to deploy this in-house if they like.

The pro-and-cons are discussed here.

Edge Deployment

As the workforce becomes more and more mobile these days, an Edge Architecture is highly recommended.

What is an Edge Architecture?

A two-tiered architecture that provides for secure connections over the Internet

I do not know who first came out with this term "Edge", but one will find this term more commonly used in Microsoft Exchange context.

The role of the Exchange Server 2007 Edge Transport Server is to be an SMTP Gateway for incoming and outgoing E-Mail messages

Below is how Sun Microsystems deploy their internal Edge Architecture.

FE-net: Front-End Network (Edge Network)
BE-net: Back-End Network






Basically, there is a firewall in front of FE-net and there is another firewall between FE-net and BE-net. Pretty straight-forward.


PS: I'hv been asked many times what are the recommended hardware.

As a general guideline, for a small setup with less than 5,000 users, T1000 for FE components and T2000 for BE components. (Of course, these 2 models are EOL. Find equivalent will do)

For larger deployment, I do see T2000 for FE components and V490 for BE components.



Sun internally uses the following configuration:

Migrating from Microsoft Exchange to Sun Communications Suite

It has been a challenging week for me so far. There are requests to migrate users from Microsoft Exchange platform to Sun Communications Suite. One from Vietnam for a 10,000 user base; another in Philippines for a 3,000 user base.

The main talking points are always:

1. How compatible is Sun Communications Suite?

2. How to migrate Exchange information to Sun Communications Suite?

Sun Enterprise Messaging Reference Architecture

To promote understanding of the Sun Java System Communications Services architecture, Sun created the Sun Enterprise Messaging Reference Architecture (SEMRA), which simulates a corporate Microsoft Exchange site consisting of 5,000 users. This reference architecture proved the concept that migration from Microsoft Exchange to a new Java Enterprise System target deployment works from a functional point of view.

In this architecture:

Tier 0 contains the load balancer (software or switch).
Tier 1 acts as the front end that provides access to the messaging applications.
Tier 2 forms the layer where the messaging data is handled by the applications.

Compatibility is easy to comply; Migration is more challenging. Luckily, Sun has a tool called SGMT (Sun Groupware Migration Toolkit).

I'll cover more on SGMT later. For now, you can read from here.

Highly Available Microsoft SQL Server 2008

Of late, I must admit Microsoft has improved its suite of products in terms of performance and scalability, not to mention its ease of use during installation. 

With the release of Windows Server 2008, clustering of Microsoft SQL Server has never been easier.

Connecting from the Windows server to the Storage server can be via Fibre Cable (FC) or iSCSI.

What is good is the iSCSI Initiator is already included in the OS in Windows Server 2008. This makes setting up of the Windows Cluster easier.

Exchange System Manager crashes when Help is accessed

My customer (the Singapore ministry one) called again today. It's regarding his newly migrated Exchange Server 2003. (Yes, 2003, not 2007. He's afraid to upgrade. I do not know why)

Today, the complain was that whenever he spawns the Exchange System Manager and accesses the Help, the ESM will crash. I asked him to try the ESM on the old Exchange Server. No problem, he said. 

Hmm.. How can that be? I had ensure both ESM and Exchange are of the same patch level before I left.

After some investigation, we found out that the only difference between the 2 Exchange servers is the IE version. The old server has IE 6, while the new server has IE 7. A quick search in Microsoft Knowledge-Base pointed to 

A conflict between the newer version of the Psapi.dll file that Internet Explorer 7 uses and the older version that Exchange System Manager uses

I'm left wondering why can't there be a hotfix for Exchange Server 2003 such that all cumulative fixes for bugs or conflicting .dll be packaged and released to all Exchange customers. 

This will save us a lot of time debugging.

SMTP Connector Delivery Restrictions on Microsoft Exchange 2003

The company I work for focuses on Portal, Messaging, and Identity. While we implement a lot of solutions based on Sun technology, we do provide consultation and deploy technology from other vendors. 

As long as it's Portal, Messaging and Identity.

For messaging alone, we are pretty comfortable with Sun Java System Messaging Server, Microsoft Exchange, Zimbra, Postfix and Sendmail. 

Personally, I prefer Postfix over Sendmail. Our company used to provide email hosting based on Postfix, until we switched to Sun Messaging Server some time back. 

Interestingly enough, a teleco in Singapore is using Sendmail as their MTA with Exchange as their backend. I configured the Sendmail for them, with integration to TrendMicro Anti-Virus/Anti-Spam engine.

As for Microsoft Exchange, I start to like it with Exchange 2007 onwards. I like the architecture for the new releases. I can't say the same about Exchange 2000/2003 though. :)

Anyway, I received a call from my customer (a ministry) that the Delivery Restrictions on their newly migrated Exchange Server 2003 is not working. As they have not purchased a support contract with us, we are not able to make a trip on-site to quickly resolve the issue.

It took quite a while to pin-point what had actually gone wrong remotely. It turned out that a registry key was not turned on while the migration was carried out. It was a rather rush job then. See here for detailed information.

Connector restriction checking is turned off by default because it can significantly affect performance to expand distribution groups and check the restrictions for each message that passes through the system. If possible, turn on this setting on where it is necessary (for example, on the bridgehead server for the restricted connector).

HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Resvc/Parameters/

On the Edit menu, click Add Value, and then add the following registry value:

Value Name: CheckConnectorRestrictions

Data Type: REG_DWORD

Radix: Hexadecimal

Value: 1

It slipped my mind then. 

But come to think of it again, isn't Microsoft all about user-friendliness?

Why is there no warning message after the "OK" button is clicked? Or at least an instruction to modify the registry right on top of the "OK" button? :)