This afternoon, we encountered a very weird issue with the OpenAM Policy Agent. The background story is the customer has 4 policy rules created. The port number was initially wrong. The customer then changed from port 18081 to the correct 18080 as seen below.
The OpenAM server was restarted a few times, so did the server where the Policy Agent was installed. But whenever a protected page is accessed, the old policies were still been evaluated (rules with port 18081 were still found in agent debug log).
So there were 8 policy rules, instead of the intended 4. And the evaluation result was also consistently incorrect.
It took us quite a while to resolve the issue.
Before I resolved the issue, I interviewed the customer what they were doing while amending the old rules. And I was told the OpenAM Admin Console was "buggy" and sometimes it did not saved what they have keyed in. At times, when they clicked on "Save" more than once, they'll be prompted with something like "policy/rule already exists".
This triggered me to an issue I encountered a year ago on another OpenAM project back in Singapore. So I opened Apache Directory Studio to connect to the embedded OpenDJ.
From my experience with that project, I know that when a "Save" is clicked on the OpenAM Admin Console, OpenAM will write to 2 places in the OpenDJ. (Yes, we turned on audit logging on OpenDJ back then)
The 2 places are:
And that's where I found the issues!
The entries in ou=sunEntitlementIndexes are "corrupted"! It still contained the 4 old policy rules.
So, we went ahead to remove all 8 policies that belong to POL_UUI-1 (for this customer's example). Then we clicked on "Save" again on the OpenAM Admin Console. This will populate the correct rules (rules with port 18080) back to OpenDJ again.
That's it! Hard work.