OpenDJ - preferred platform

In the OpenDJ mailing list, there was a discussion on the preferred platform for OpenDJ deployment for production.

We know OpenDJ is 100% Java. Thus technically, it can run anywhere.

Below is the reply from Ludovic, the Product Manager for OpenDJ:

The truth is that historically OpenDJ (and it's legacy OpenDS) was primarily developed on Unix by Unix engineers (mostly Solaris). Remember it all started at Sun Microsystems. Since we've created ForgeRock, I've seen a big shift of use of OpenDJ, from being mostly deployed on Solaris to now being mostly deployed on Linux and even more frequently on Linux in a VMWare environment. This said, we do have customers that have deployed and are running OpenDJ on Windows servers. Coming soon, we will have a better integration on Linux with RPM and Debian packages. We already have SVR4 package for Solaris and family. We do not have plans nor requirements for Microsoft packaging, for now.

Yes, Linux (especially RHEL) on VMWare environment is what I get these days in customers' sites.

Not convinced with OpenDJ or have not play around with OpenDJ before? Download here. You'll love it!

.

amsfo - /lib/ld-linux.so.2: bad ELF interpreter

I am in customer's site supporting their load-testing activity this week.

Prior to the load-test, I made a round of checking to make sure all instances of OpenAM are functioning properly. The dependent components were checked as well, including Sun GlassFish(tm) Message Queue 4.4.

Not good, one of the nodes are showing the following errors when I stopped and restarted:

[webadm@node4 ~]$ bin/tools/amsfo/sfo/bin/amsfo start


/home/ssoadm/bin/tools/amsfo/jmq/imq/bin/imqbrokerd: /usr/java/latest/bin/java: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
/home/ssoadm/bin/tools/amsfo/jmq/imq/bin/imqbrokerd: /usr/java/latest/bin/java: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

A Google search indicated that this is an issue with the installation of JRE 1.7. Read here.

Seemed to me the Unix administrator had done something without informing me while I was back home in Singapore. :-|

I was lazy to argue though as I wanted OpenAM and related components to be run as independently as possible, so that any system-related patching/upgrading (without informing/discussing with me) will not impact the continuous service of OpenAM to end-users.

There are 2 basis things to do:

I went ahead to install my own JDK in a specific directory /apps/bin/jdk1.6.0_31 and created a symbolic link /apps/bin/jdk.


1. Edit .bash_profile


[webadm@node4 ~]$ vi .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

JAVA_HOME=/apps/bin/jdk
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin

export JAVA_HOME PATH


2. Edit amsfo

[webadm@node4 ~]$ vi bin/tools/amsfo/sfo/bin/amsfo

BROKER_OPTIONS="-silent -javahome /apps/bin/jdk"

That's it!

.

ORA-00942: table or view does not exist

My customer has enabled database logging in OpenAM. The backend is Oracle 11g RAC in production. (yes, pretty rich customer :>)

In production, the schema created for OpenAM logging is slightly complicated. According to customer's security standard, any application account that connects to the Oracle database will have limited privileges. That includes CREATE TABLE privilege. 

Therefore, all the possible tables that will be dynamically created by OpenAM during logging have to be pre-populated by the DBA. That's tedious but doable.

However, do note you'll see the following upon every restart of OpenAM:


amLog:11/27/2012 12:47:54:051 PM ICT: Thread[LoggingThread,5,jboss]
ERROR: amSSO_access:DBHandler:flush:execUpdate:reconnect successful.
amLog:11/27/2012 12:47:54:076 PM ICT: Thread[LoggingThread,5,jboss]
ERROR: DBHandler:createTable: 'alter table amSSO_access add (LoginID varchar(255), ContextID varchar(255), IPAddr varchar(255), LogLevel varchar(255), Domain varchar(255), LoggedBy varchar(255), MessageID varchar(255), ModuleName varchar(255), NameID varchar(255))'; error (942); msg = ORA-00942: table or view does not exist

amLog:11/27/2012 12:47:54:144 PM ICT: Thread[LoggingThread,5,jboss]
ERROR: amAuthentication_access:DBHandler:flush:execUpdate:reconnect successful.
amLog:11/27/2012 12:47:54:304 PM ICT: Thread[LoggingThread,5,jboss]
ERROR: DBHandler:createTable: 'alter table amAuthentication_access add (LoginID varchar(255), ContextID varchar(255), IPAddr varchar(255), LogLevel varchar(255), Domain varchar(255), LoggedBy varchar(255), MessageID varchar(255), ModuleName varchar(255), NameID varchar(255))'; error (942); msg = ORA-00942: table or view does not exist

This error can be safely ignored. Read here.

Sometimes, when a view or a procedure is created that accesses a table, Oracle issues the error ORA-00942: table or view does not exist, although this table (or view) definitely exists. 

.

AM Session DB - Sun GlassFish(tm) Message Queue 4.4

We know Sun GlassFish(tm) Message Queue 4.4 is the underlying message queue for OpenAM Session DB.

I was deploying AM Session DB a week ago when I was at customer's site. It was supposed to be a 3-nodes MQ message cluster.

The 1st day of deployment was not that enjoyable.

I kept hitting the following warning messages:



[15/Nov/2012:18:32:28 ICT] [B1004]: Starting the cluster service using tcp [ amnode1/172.19.176.115:50582 ] with min threads 1 and max threads of 1 
[15/Nov/2012:18:32:28 ICT] [B1228]: Cluster ping interval is 60 seconds
[15/Nov/2012:18:32:28 ICT] [B1039]: Broker "ambroker@192.168.1.115:7676" ready.
[15/Nov/2012:18:32:28 ICT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://192.168.1.116:7676/?instName=???&brokerSessionUID=null failed: No route to host
[15/Nov/2012:18:32:28 ICT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://192.168.1.117:7676/?instName=???&brokerSessionUID=null failed: No route to host
[15/Nov/2012:18:35:48 ICT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://192.168.1.116:7676/?instName=???&brokerSessionUID=null failed: No route to host
[15/Nov/2012:18:35:49 ICT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://192.168.1.117:7676/?instName=???&brokerSessionUID=null failed: No route to host


Note at this point of time, I had not configured OpenAM to use AM Session DB yet. My intention was to make sure that the 3-nodes cluster is working properly.

The log was not showing something I wanted it to show. So I spent a considerable amount of time to debug, and that includes asking the Unix team to loosen the rules in the OS's iptable (see here). Yet, after the firewall rule was loosen, I kept getting the same warning messages.

I called it a day then. Too tired, too hungry.

The next day when I came back, I was getting the same warning messages again and again. That's when I changed strategy. Maybe I should just configure OpenAM for Session Failover to see whether if there's any luck.

Bingo!



[16/Nov/2012:10:06:30 ICT] [B1179]: Activated broker 
Address = mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=6143819218583616768
StartTime = 1353034753349
ProtocolVersion = 410 
[16/Nov/2012:10:06:30 ICT] [B1071]: Established cluster connection to broker mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=6143819218583616768[/192.168.1.117:37897]
[16/Nov/2012:10:06:30 ICT] [B1179]: Activated broker 
Address = mq://192.168.1.116:7676/?instName=ambroker&brokerSessionUID=6894512981523671040
StartTime = 1353034959341
ProtocolVersion = 410 
[16/Nov/2012:10:06:30 ICT] [B1071]: Established cluster connection to broker mq://192.168.1.116:7676/?instName=ambroker&brokerSessionUID=6894512981523671040[/192.168.1.116:57351]
[16/Nov/2012:10:11:07 ICT] [B1173]: Received GOODBYE [mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=6143819218583616768] from mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=6143819218583616768
[16/Nov/2012:10:11:07 ICT] [B1072]: Closed cluster connection to broker mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=6143819218583616768
[16/Nov/2012:10:11:07 ICT] [B1180]: Deactivated broker 
Address = mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=6143819218583616768
StartTime = 1353034753349
ProtocolVersion = 410 
[16/Nov/2012:10:12:32 ICT] [B1179]: Activated broker 
Address = mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=5327541786327008256
StartTime = 1353035551372
ProtocolVersion = 410 
[16/Nov/2012:10:12:32 ICT] [B1071]: Established cluster connection to broker mq://192.168.1.117:7676/?instName=ambroker&brokerSessionUID=5327541786327008256[/192.168.1.117:51002]

So, that warning messages could simply be ignored. I still have no idea what the hell are those messages for. They are super mis-leading! I do not seem to see them in my previous deployments. 

.

AM Session DB - To be improved

In amsessiondb.log, you'll see the following being captured:


Starting...  true
/data/webserver/jdk1.6.0_31/jre/bin/java -Xms128m -Xmx512m -classpath /home/webadm/bin/tools/amsfo/jmq/imq/lib/imq.jar:/home/webadm/bin/tools/amsfo/jmq/imq/lib/jms.jar:/home/webadm/bin/tools/amsfo/ext/je.jar:/home/webadm/bin/tools/amsfo/locale:/home/webadm/bin/tools/amsfo/lib/am_sessiondb.jar:. com.sun.identity.ha.jmqdb.client.FAMHaDB
Initializing and connecting to the Message Queue server ...
Successfully started.
done. \n
Error: Connection to Broker Lost. Broker could be down.
Error: Connection to Broker Lost. Broker could be down.
Starting...  true
/data/webserver/jdk1.6.0_31/jre/bin/java -Xms128m -Xmx512m -classpath /home/webadm/bin/tools/amsfo/jmq/imq/lib/imq.jar:/home/webadm/bin/tools/amsfo/jmq/imq/lib/jms.jar:/home/webadm/bin/tools/amsfo/ext/je.jar:/home/webadm/bin/tools/amsfo/locale:/home/webadm/bin/tools/amsfo/lib/am_sessiondb.jar:. com.sun.identity.ha.jmqdb.client.FAMHaDB
Initializing and connecting to the Message Queue server ...
Successfully started.
done. \n
Error: Connection to Broker Lost. Broker could be down.
Error: Connection to Broker Lost. Broker could be down.
Starting...  true
/data/webserver/jdk1.6.0_31/jre/bin/java -Xms128m -Xmx512m -classpath /home/webadm/bin/tools/amsfo/jmq/imq/lib/imq.jar:/home/webadm/bin/tools/amsfo/jmq/imq/lib/jms.jar:/home/webadm/bin/tools/amsfo/ext/je.jar:/home/webadm/bin/tools/amsfo/locale:/home/webadm/bin/tools/amsfo/lib/am_sessiondb.jar:. com.sun.identity.ha.jmqdb.client.FAMHaDB
Initializing and connecting to the Message Queue server ...
Successfully started.


Useful? Not at all.. The basic element of timestamp is not even there.

.

To Enable Broker Connections Through a Firewall

The network/OS team here in Bangkok is really on the ball. Really!

Besides a strictly configured firewall between server zones, the system administrators even have iptables enabled on each machine. This is to say, even if all my OpenAM servers are in the same zone, they are not able to communicate with each other!!

This is annoying. So I have to submit all ports that I want them to allow and to explain the usage of each port. (yes, more paper work!).

Nevertheless, once the ports are enabled, the 5 OpenAM servers are now able to communicate and work as a site. Cool!

But today, when I configure AM Session Failover, I was not too lucky.

I kept getting the following errors:


2012-11-15 15:46:46,583 WARNING [javax.jms] (main) [C4003]: Error occurred on connection creation [192.168.1.117:41612]. - cause: java.net.NoRouteToHostException: No route to host
2012-11-15 15:46:49,586 WARNING [javax.jms] (main) [C4003]: Error occurred on connection creation [192.168.1.116:52916]. - cause: java.net.NoRouteToHostException: No route to host

Very strange! I am damn sure I have requested port 7676 to be opened. That is the port I have assigned to the Java Message Queue on each node.

Until I saw this ... 

So, lesson learnt:

1. Port 7676 is only the Message Queue PortMapper port

2. The actual port used for communication is the JMS port (in this case, port 10234)

Careless me ...

.

java.lang.OutOfMemoryError when I have 12 cores and 48GB of RAM

These days, virtual machines are very popular in customers' sites. Almost 80-90% of my customers require us to install OpenAM/OpenDJ on their VMs.

But sometimes, it can get really annoying if the virtual machines show errors that are not useful in debugging!!

How is it possible that I can run out of memory when I was given a 12 cores + 48 GB RAM VM?



2012-11-14 10:42:00,184 INFO  [javax.enterprise.resource.webservices.rpc.server.http] (http-172.19.176.116-8280-1) JAXRPCSERVLET14: JAX-RPC servlet initializing
2012-11-14 10:42:25,503 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-3,172.19.176.116:8100) failure passing message down
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.jgroups.protocols.TP.down(TP.java:1175)
        at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2308)
        at org.jgroups.protocols.Discovery.down(Discovery.java:374)
        at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)
        at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:381)
        at org.jgroups.protocols.FD.down(FD.java:315)
        at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95)
        at org.jgroups.protocols.BARRIER.down(BARRIER.java:107)
        at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:803)
        at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:604)
        at org.jgroups.protocols.UNICAST.down(UNICAST.java:455)
        at org.jgroups.protocols.pbcast.STABLE$1.run(STABLE.java:644)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

Obviously, there is some mis-match in how the virtual machine is setup versa that with the physical hardware.

[webadm@tmsappr5 ~]$ jboss-adm/SSOADMIN_node1 stop
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: Resource temporarily unavailable
JBOSS_CMD_START = cd /data/webserver/jboss-eap-5.1/jboss-as/bin; /data/webserver/jboss-eap-5.1/jboss-as/bin/run.sh -c SSOADMIN_node1 -P /data/webdata/SSOADMIN_node1/server.properties
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable

.

Changing the password of cn=Directory Manager in OpenAM

With default installation of OpenAM, the passwords for amadmin and cn=Directory Manager (for embedded OpenDJ) are the same.

Some customers would like to make the 2 passwords distinct. And some even want the passwords to be changed every 90 days based on their corporate security policies.

Assumption


1. OpenAM Configuration Directory : /home/openam10/var/openam
2. Perform Step 1 to 3 while OpenAM server is shutdown

How to change the password of cn=Directory Manager?

Step 1: Create a class to generate the encoded password for bootstrap file

[openam10@am10 crypt]$ java -cp .:../ssoadmin/lib/opensso-sharedlib.jar:../ssoadmin/lib/amserver.jar AZCrypt password

AQIC5wM2LY4SfczLlj6134qMTx0nkE5XiFMg  (old password)

[openam10@am10 crypt]$ java -cp .:../ssoadmin/lib/opensso-sharedlib.jar:../ssoadmin/lib/amserver.jar AZCrypt P@ssw0rd

AQIC5wM2LY4SfcyUmw%2B6K%2FG%2FBYPuMDaX2Tj3 (new password)

[openam10@am10 ~]$ cd /home/openam10/var/openam
[openam10@am10 openam]$ cp bootstrap bootstrap.BAK.20121109
[openam10@am10 openam]$ vi bootstrap

Replace the existing password (AQIC5wM2LY4SfczLlj6134qMTx0nkE5XiFMg) with the new one (AQIC5wM2LY4SfcyUmw%2B6K%2FG%2FBYPuMDaX2Tj3).




Step 2: Change encoded password for OpenDJ Configuration file

[openam10@am10 ~]$ cd /home/openam10/var/openam/opends/bin

[openam10@am10 bin]$ ./encode-password -s SSHA512 -c P@ssw0rd
Encoded Password:  "{SSHA512}rPyik4bSqX68JnmpWolSIn/IL6SpKyuThh1yho3fPSFDR6qYS+RcKKgfqZ5zVdVXt4S34nE7syBQSXQQxrO9+DgdeaUV2Uf5"


[openam10@am10 ~]$ cd /home/openam10/var/openam/opends/config
[openam10@am10 config]$ cp config.ldif config.ldif.BAK.20121108
[openam10@am10 config]$ vi config.ldif

Replace userPassword with the one generated by encode-password as shown above.

Step 3: Change encoded password for Directory Server Configuration in OpenAM

[openam10@am10 ~]$ cd /home/openam10/var/openam/opends/bin
[openam10@am10 bin]$ ./start-ds


Navigate to the following dn with a LDAP tool like Apache Studio:


dn: ou=http://am10.sg.azlabs:9080/openam,ou=com-sun-identity-servers,ou=default,ou=GlobalConfig,ou=1.0,ou=iPlanetAMPlatformService,ou=services,dc=opensso,dc=java,dc=net

Replace DirPassword with the one generated by encode-password as shown above in Step 2.


At this moment, let's do a check-point...

If the above steps are not executed correctly, you'll see the famous OpenAM Configuration Options Page when the OpenAM server is next started.

Otherwise, everything should be good and you'll see the OpenAM Login Page as expected.



Step 4: Change password for Policy Configuration in OpenAM

This last step can be configured when the OpenAM server is up and running with the new password for cn=Directory Manager. Of course, it's definitely possible to change the password via a LDAP tool like Step 3. (But I'm just lazy to do it after hacking for so long. Ha!)


.

Changing the password of amAdmin

Changing the password of amAdmin is very simple in OpenAM.

Navigate to Subjects and click on amAdmin. (Note: Regardless of the type of datastore that has been created, amadmin will always be shown as a subject. In this example, my datastore is an Active Directory.)

Click on "Edit". Then change password.

Done.

.

OpenAM Policy Agent Audit Logging to Database

In my customer's OpenAM environment, database logging is configured instead of file-based logging.

To enable database logging, we need to navigate to Configuration -> System -> Logging. The default Logging Type is File. Switch to DB. My customer back-end database is Oracle 11g RAC.

When this is done, restart OpenAM server and logging will be redirected to the Oracle database. 

Next, my customer likes to track what his users are doing for every Policy Agent that have been deployed. So audit logging has to be turned on at the Policy Agent configuration. In addition, customer would like to centralize the logging. This means logging shall be consolidated at the OpenAM servers. 

Easy, the default Audit Access Types is LOG_NONE. Switch to LOG_ALLOW. Next, select REMOTE for Audit Log Location. Cool, it's done!

Well .... but no... the database shows nothing. There isn't a new table created for this agent. Restarted a few times, nothing. Strange!

So we dig into the CoreSystem debug log and found this:

WARNING: amAgent_esys-uat_abc_com_18081_log:DBHandler:Maximum DB memory buffer size < Buffer Size, setting to buffer size (25)

DBHandler:tableName = amAgent_esys-uat_abc_com_18081_log, LOG_STATUS = ACTIVE

amAgent_esys-uat_abc_com_18081_log:DBHandler: the query string for creating is create table amAgent_esys-uat_abc_com_18081_log (time date,  data CLOB, LoginID varchar2 (255), ContextID varchar2 (255), IPAddr varchar2 (255), LogLevel varchar2 (255), Domain varchar2 (255), LoggedBy varchar2 (255), MessageID varchar2 (255), ModuleName varchar2 (255), NameID varchar2 (255), HostName varchar2 (255)) 

ERROR: amAgent_jboss_truecorp_co_th_18443_log:DBHandler:createTable:Execute:SQLEx (972): ORA-00972: identifier is too long

ERROR: amAgent_jboss_truecorp_co_th_18443_log:DBHandler: sql operation unsuccessful (972): ORA-00972: identifier is too long

Ah! It's a Oracle table name restriction!

The default Remote Log Filename is being used as the Oracle table name, but the length is too long for Oracle to accept.

Changed to a shorter name. Restarted OpenAM server and Policy Agent. It works!


.

Weird OpenAM Policy Agent Evaluation Behavior

I am now in the implementation of a OpenAM project in Bangkok, Thailand.

This afternoon, we encountered a very weird issue with the OpenAM Policy Agent. The background story is the customer has 4 policy rules created. The port number was initially wrong. The customer then changed from port 18081 to the correct 18080 as seen below.

The OpenAM server was restarted a few times, so did the server where the Policy Agent was installed. But whenever a protected page is accessed, the old policies were still been evaluated (rules with port 18081 were still found in agent debug log).

So there were 8 policy rules, instead of the intended 4. And the evaluation result was also consistently incorrect.

It took us quite a while to resolve the issue. 

Before I resolved the issue, I interviewed the customer what they were doing while amending the old rules. And I was told the OpenAM Admin Console was "buggy" and sometimes it did not saved what they have keyed in. At times, when they clicked on "Save" more than once, they'll be prompted with something like "policy/rule already exists".

This triggered me to an issue I encountered a year ago on another OpenAM project back in Singapore. So I  opened Apache Directory Studio to connect to the embedded OpenDJ.

From my experience with that project, I know that when a "Save" is clicked on the OpenAM Admin Console, OpenAM will write to 2 places in the OpenDJ. (Yes, we turned on audit logging on OpenDJ back then)

The 2 places are:
1. ou=Policies,ou=default,ou=OrganizationConfig,ou=1.0,ou=iPlanetAMPolicyService,ou=services,xxx
2. ou=default,ou=default,ou=OrganizationConfig,ou=1.0,ou=sunEntitlementIndexes,ou=services,xxx

And that's where I found the issues!

The entries in ou=sunEntitlementIndexes are "corrupted"! It still contained the 4 old policy rules.

So, we went ahead to remove all 8 policies that belong to POL_UUI-1 (for this customer's example). Then we clicked on "Save" again on the OpenAM Admin Console. This will populate the correct rules (rules with port 18080) back to OpenDJ again.

That's it! Hard work.

.