More and more APIs exposed by Ministries

The trend is pretty clear these days - ministries in Singapore are encouraged to expose more and more APIs for external entities to consume, in particular organizations/companies.

This will save huge amount of human hours if manual tasks like Form Submissions can be automated. Yes, we have a labour crunch here in Singapore. We do not have enough local people in our workforce. Thus automation is key going forward!


My team is currently designing APIs for one of the financial institutions in Singapore. This week, we have a new request to integrate with Inland Revenue Authority of Singapore (IRAS) for Commission Income Information Submission.

The current process of submission is very manual.

It requires a person to download a program (Offline Validation Program - OVP) from IRAS to validate a XML/Excel file. Once validated, one has to log in via SingPass (Singapore Personal Access) and then manually upload the XML/Excel file to IRAS portal.

The new API service allows applications to directly submit Commission Income Information to IRAS , thus eliminating the need to download and validate using OVP and login to IRAS portal for submission.


.

Microgateway

“CA Microgateway is a great example of how more and more of the tech industry is realizing that APIs are not just how we integrate tools and apps, APIs are the new way of doing business and building new revolutionary technologies,” Said Geoff Domoracki, Founder of API World.

CA Microgateway has received a 2017 API Award for the category: Microservices APIs. While the product is still in beta, its ability to deliver aggregation, orchestration, security and management to microservice architectures impressed the judging panel for API World this year.

Kind of strange to me that a beta product can win an award. :)

So what is really a Microgateway? Why do you need one if API Gateway has been recommended over the years?

I could not find a good explanation in CA documentation. Well, since it is still in beta, I would expect there are not too many real use cases to talk about.

Google Apigee has better explanation.

• Reduce latency of traffic for close proximity services 
• Keep API traffic within the enterprise-approved boundaries because of security or compliance purposes 
• Continue processing messages if internet connection is lost

Features of Google Apigee Microgateway:

When to use Microgateway over traditional Enterprise Gateway?

This is the most important slide. 

Summary:

• API has to be light-weight. (otherwise, how do you classify yourself as microservice?)
• Few APIs to a gateway 
• No complex mediation rule (aka no heavy-duty processing)
• Small scale (100+ APIs per sec)
• Internal API for app-to-app integration 

.

In-place Upgrade vs Greenfield Upgrade

There are different school of thoughts when it comes to upgrade - In-place vs Greenfield (or Side-by-side). 

In-place Upgrade could mean 2 things:
U1. Executing an upgrade wizard that comes with the software
U2. Install the latest version on another disk partition, separated from the current version that is Live.

I refer to the former in this context.

When cost is involved, most will go with In-place Upgrade as a greenfield upgrade requires another identical set of hardware and operating system. This, I totally agree.

When customers have cost constraint, I will usually recommend U2 for major version upgrade. Minor version upgrades are usually bug and security fixes. I will usually use approach U1. No brainer.


Now, there are 3 groups of specialists who will give you different opinions when you ask them if it would be wise to perform upgrade using approach U1:
S1. Product Engineers
S2. Product Presales
S3. Product Professional Service Engineers or Service Delivery Partners


Now, if the wizard is written by you (S1), would you say "No, U1 is not the better approach"? This group of specialists also tend to assume that the product is deployed out-of-the-box with no or minimal customization. My foot. Now, even if they agree that the product is highly customized, I come to realize that QA of the wizard is usually not well carried out. There are few too many edge cases that they would not be able to think of in their own labs. (can you blame them? no. product engineers are usually the ones who do not like to meet and interact with customers. thus they do not get to know the business-world requirements and they definitely do not have the full picture of how the products they developed are deployed in real world)

What would S2 say? You know the KPI of a presales? Can you say that the product that you are selling so hard has no seamless upgrade wizard in front of your customers? This group of specialists has the least hands-on. They are really good in sweet talking and reading documentation. You ask them to set up the software from scratch. They look at you like you are an alien. "Dude... there's a VM with everything installed. Come on...". Asking S2 deep technical/operational/migration questions is usually a waste of time, unless you do not mind hearing white lies. (well, to be honest, they do not want to cause harm. just that they do not have the right hands-on experience)

S3 is the group of specialists that are doing the shit job, even to the extend of implementing empty promises from S2 that are totally out-of-scope. S3 usually tells the hard truth, which customers would not like to hear at times. "well, I have paid tons of dollars for this piece of software? I don't care. You get it fixed to what was being promised". However, they have years of hands-on and have seen many types of deployments. They also have many years of migration experience. (ok, I have also seen delivery partners that are incompetent. there was once a big customer in SG asking this delivery partner how to extend their current SSO infrastructure to support multiple domains. to our surprise, a consultant from this delivery partner suggested to set up a totally new environment to support this new domain. omg! most SSO products can support cross-domain single sign-on these days. It's a out of the box feature)


Azimuth has been in the Identity business for a long time and we have performed numerous upgrades for customers. Softwares, especially IDM and SSO, are "fragile" animals. IDM products are usually highly customized, while SSO products are usually deployed as a farm of server nodes to handle huge traffic.

SSO solutions usually have low tolerance for downtime. When in-place upgrade is adopted, there are many questions to ask. 

1. How do you upgrade to a new major release then? Say there are 10 nodes to upgrade.
2. How do you upgrade to a release that is at least 2-3 versions apart? 
3. How do you upgrade the underlying configuration store, if any? Say there are 4 nodes to upgrade.
4. How do you ensure protected applications work as before? Say at least 50 applications are involved which belong to 10 application teams? 
5. How do you ensure customized codes work as before? 
6. How do you ensure deprecated functions that are replaced with new functions in the new release work as before? 
7. How do you ensure new features requested by customers work as promised? 

How do you complete the above all within a narrow downtime window of 5 hours (12am to 5am)? Nearly impossible.

A presales will never take the above into consideration when being asked about which is the better upgrade approach. Stick with product documentation is always the right approach. Crap.

One would also question why would there be challenge and surprise if well-conducted UAT is carried out prior to Production upgrade. From experience, I can conclude that most UAT environment is not identical to Production. Seldom, less than 10%. Why take the risk and bang your balls for that tiny 5 hours window? It's not worth taking the risk. Rolling back, investigating why, perform another round of upgrade, activating the same whole village of application teams to perform end-to-end testings are just not worth the trouble. In fact, reputation is at stake.

Some customers are also one of the kind. They are not technical enough, yet they want to have a say. So they go around asking opinion from various sources. A service delivery partner does not come from the Product organization. The customers will take the advice from the Product Presales over what the Service Delivery Partner recommends. "How can I not believe what XXX has said? XXX is from the same Product organization. He sure knows better than you."

Oh, really? 

Now, who is performing the upgrade? What if something goes wrong during upgrade? Whose responsibility is that? To ask a service delivery partner to perform upgrade again at a fixed cost (we practice fixed cost in Asia) is really not a fair deal.

Approval Anywhere

These days, Approval Anywhere seems to be a must-have feature in product development.

One Identity recently launched Safeguard - a Privileged Access Management (PAM) solution.

The biggest feature is what One Identity is calling “Approval Anywhere,” a cloud-based workflow to securely approve session or password requests from any device.

CA has a similar "Approval Anywhere" feature from the CA Identity Portal.

It comes with an OOTB Mobile Web application. No configuration is required to enable and configure this mobile application. The application can be browsed through any mobile browser. CA Identity Suite will automatically detect that you're browsing from a mobile device and redirect you to the mobile application.


This feature helps a lot during presentation, especially to the C-level. Believe me. :)

.

Exposing APIs in the Financial Services

I met up with a number of financial institutions over the past few months. You know these are established companies with long history and their IT infrastructure is strong and big.

Out of 10, 9 of them will have SOA (Service Oriented Architecture) in place. Then they'll have ESB (Enterprise Service Bus) solution to support their SOA initiatives. Web services are everywhere. Integration with internal and external parties are also common.

Why would they need a API Management then? Below are some of my slides:

Security, Sandboxing and Scalability is my most-used tagline.

Yes, they could have deployed their own DIY solutions, as the concept of API Management is not something new. Honestly, if your team is highly technical and security-conscious, there is really no need to buy a ready-built APIM product out there.

It's a matter of time to market and where you want to spend your time in.

Do you want to spend most of your time in your core business? Or do you want to split your precious time in managing the security risks and scalability issues when your APIs are exposed?

It's a choice - $ vs Time.


Below is a statistics showing the target audience of exposed APIs in the Financial Services:

The goals of deploying a APIM solution in the Financial Services section as illustrated below:

Interesting.

CA API Developer Portal UI Customization - Part II

CA Developer API Portal aims to build and monetize a broad and effective digital ecosystem, enterprises must make it easy for developers to find and use APIs. It enables developer communities and support app marketplaces with essential capabilities for management, security and monitoring.

For the developer portal to be useful, the forums must be a go-to place for developers. The UI has to be intuitive and modern to use.

Our designers continues to wow us. They have converted the original look-n-feel of the forum, which looks like a piece of software developed by engineers, into an UI where information nicely presented and easy to navigate.

Well done.

.

CA API Developer Portal UI Customization

CA API Developer Portal is an add-on to CA API Gateway.

CA API Developer Portal simplifies API discovery for developers and provides them with access to enterprise data so they can get to building apps fast. Relationships with developers, partners and third parties are easily managed and analytics provide valuable operational data for optimizing your business.

Developers can use the portal to:

• Discover what APIs are available and apply for access 
• Receive authorization to use APIs and get API keys 
• Get started on development faster with educational tools including an API catalog, sample applications, mobile device SDKs and code generation

There is a in-built CMS (Content Management Server) that shipped together with CA API Developer Portal.

Most customers use the CMS to customize to their corporates' look and feel. However, almost all look the same in terms of layout.

We have a customer who demands that he does not want his CA API Developer Portal to look like any other CA API Developer Portal. :) 

Can it be done, yet utilizing the same CMS? If you ask your engineers, no way! 

"It can't be done. I have played with the CSS within the CA API Developer Portal already. It's very tough. Not worth the effort."

Customer did not buy it. He was willing to pay. Well, we engaged a professional design firm (owned by a friend of mine). 

Here we go ...

Customer is super happy! The best part is there is no total overhaul. The designers are very well-versed with CSS. They used back the same CMS and tweaked the CSS only.

I am impressed. 

.

Kong API Developer Portal

Kong is a scalable, open source API Gateway. Kong runs in front of any RESTful API and is extended through Plugins, which provide extra functionality and services beyond the core platform.

We explored Kong few months back and liked it for its flexibility and scalability. A customer came along and was willing to give open source a try. Deal on.

During development, we discovered that Kong is overly flexible. :)  

I still remember the days when we were Compuware partner deploying and giving consultation to Dynatrace customers. There were far too many features and it allowed great flexibility. Beginners or customers who were not too technical complained. They got lost. 

With that experience still fresh in my mind, I suggested that we build a API Developer Portal on our own. And this portal should cater for this customer's daily operations. Nothing else. Whatever is not required, we should hide it.


The team has done a great job!

For this particular customer, the business use case is simple.

1. They have their internal application teams ("Clients") that want to consume APIs.
2. Each client will be issued a Client ID and Client Secret to exchange for a OAuth2 token prior to calling APIs.
3. Each client can only be granted access to specific set of APIs ("Service")


So, the UI is simplified to suit this need.

a. Create API

b. Assign API to Service

 c. Create Client and assign Service to Client

Very simple to use to get the job done.


.

API Management Lifecycle

Reset. Recharge. Refocus.


These days, API Management is a hot topic. Every ministry, every insurance company, every bank ... they are trying to publish APIs to the external world.

What's the purpose? It is clearly illustrated in the diagram below:

1. Higher customer satisfaction
2. Higher transaction volumes
3. Higher digital reach

As simple as that.

So, a lot of vendors rush in to deliver API projects. Most think it's as simple as deploying the API Gateway, configure a few security policies and create a few endpoints. Then proceed to sign-off.

So simple. Easy money.

Well, if money is so easy to earn.... :)

From our experience with a few customers who have gone live so far, this has been a long journey - the Full Lifecycle API Management journey.




The whole process is time-consuming. It involves multiple decision makers from different business units. The implementation is tough and requires multiple iterations to get an API published in an optimal manner.

The lifecycle requires a API architect to oversee the project and to get the various stakeholders to head towards a common goal. The architect has to manage internal customers (backend application owners) and external customers (API consumers/developers).

The published APIs have to be easy to consume, yet they have to adhere to the highest security standard.


Not as simple as many vendors would have expect a API project to be.

This is where we like to position ourselves. This challenge is what we are looking forward to.

.

Shibboleth SP - listener failed to initialize

I was trying to set up Shibboleth SP for a customer for POC.

And so I was playing around with configuration and then I tried to restart shibd process. I kept getting "listener failed to initialize".

Newbie to Shibboleth. Took me a while to find out that the process was killed, but the pid and .sock files are not removed.

Once both files are removed from /var/run/shibboleth directory, the shibd process is able to start successfully!


PS: By the way, the proper way to restart shibd process is:

[root@cdemo shibboleth]# /etc/init.d/shibd restart
Stopping shibd:                                            [  OK  ]
Starting shibd:                                            [  OK  ]

.

OpenAM Login Page - Bad request error

I just started a migration for a customer who is still on OpenAM 10.2. Yes, my old-time customer who is still using the old JATO UI. :)

So, the deployment architecture is simply straight-forward. We have OpenAM 13.5 deployed on a Tomcat server and we have Apache HTTPd server acting as a reverse proxy. This is where we will tighten the rules for allowing/disallowing OpenAM endpoints from being exposed to the Internet.

Maybe I'm rusty, as I have been busy with other projects using other products. During setup, I kept hitting into issues - Bad request error and No values provided for the request parameter '_action'.

Anyway, after a long while investigation, then did I realized my reverse proxy setting on my Apache HTTPd server was configured wrongly! :)

Incorrect Setting

Correct Setting

So I forgot to add a slash to the URI.

ProxyPass /sso/ http://localhost:8080/sso/
ProxyPassReverse /sso/ http://localhost:8080/sso/


Silly me! :)


.

Integrate OpenAM 13.5 with Atlassian Jira 7.3 - Part 2

So I dumped OpenAM ClientSDK and decided to go the RESTful way. It turns out to be fairly easy.

References:

1. Single Sign-on Integration with the Atlassian stack
2. HTTP authentication with Seraph
3. Sample code - OpenSsoAuthenticator.java 

Step 1: Extend OpenAMAuthenticator from JiraSeraphAuthenticator

I took reference from OpenSsoAuthenticator.java.

In public Principal getUser(HttpServletRequest request, HttpServletResponse response), there is this line that attempts to retrieve a user name.

String username = obtainUsername(request);


I removed all other methods in the class and replace the method obtainUsername(HttpServletRequest request) with the following:

Step 2: Copy JSON library (json-20090211.jar) to JIRA library directory

(As we are using RESTful calls to OpenAM to validate user's session, the response from OpenAM is in JSON format. Thus the need for the JSON library)

[azlabs@sg-jira ~]$ cp json-20090211.jar /appl/jira/jira/atlassian-jira/WEB-INF/lib/

Step 3: Extend log4j.properties

(If this is not configured, the logging will not be output to JIRA logging system - catalina.out)

[azlabs@sg-jira classes]$ vi log4j.properties
< # AZLABS
< log4j.logger.sg.azlabs.openam.jira.seraph.OpenAMAuthenticator = INFO, console, filelog
< log4j.additivity.sg.azlabs.openam.jira.seraph.OpenAMAuthenticator = false
< # EOF - AZLABS


Step 4: Edit seraph-config.xml to redirect to OpenAM Login/Logout pages

(This is where the iPlanetDirectoryPro cookie will be generated on user's browser after OpenAM Login authentication)

[azlabs@sg-jira ~]$ cd /appl/jira/jira/atlassian-jira/WEB-INF/classes
[azlabs@sg-jira classes]$ cp seraph-config.xml seraph-config.xml.20170308
[azlabs@sg-jira classes]$ vi seraph-config.xml

Step 5: Hide default Login gadget from JIRA login page

(The SSO with OpenAM only happens when a user clicks on the Login hyperlink. Keying in user name and password via the Login gadget should be removed to avoid confusion)

[azlabs@sg-jira ~]$ cd /appl/jira/jira-home
[azlabs@sg-jira jira-home]$ touch jira-config.properties
jira.disable.login.gadget=true

Step 6: Restart JIRA


We are done!


.

Integrate OpenAM 13.5 with Atlassian Jira 7.3 - Part 1

There is this very old wiki from ForgeRock that talks about how to integrate OpenAM with JIRA.

It used to work for us when we were using the older version of OpenAM Client SDK and JIRA. When I follow the same steps now with OpenAM Client SDK 13.5 and JIRA7.3, nothing works.

Some of the errors I encountered are listed below:

1) Loader constraint violation

Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of org/apache/catalina/loader/ParallelWebappClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of java/net/URLClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature

I needed to override the existing jar files in JIRA lib directory (/jira/atlassian-jira-software-7.3.0-standalone/lib) with slf4j-1.7.5.

[azlabs@sg-jira lib]$ cp /home/azlabs/openam/slf4j/slf4j-1.7.5/jul-to-slf4j-1.7.5.jar .

[azlabs@sg-jira lib]$ cp /home/azlabs/openam/slf4j/slf4j-1.7.5/slf4j-api-1.7.5.jar .

[azlabs@sg-jira lib]$ cp /home/azlabs/openam/slf4j/slf4j-1.7.5/slf4j-log4j12-1.7.5.jar .

[azlabs@sg-jira lib]$ cp /home/azlabs/openam/slf4j/slf4j-1.7.5/jcl-over-slf4j-1.7.5.jar

Still not working. In the end, I removed the SLF4J classes in ClientSDK-13.5.0.jar.

[azlabs@sg-jira tmp]$ rm -fr org/slf4j
[azlabs@sg-jira tmp]$ jar cvf ClientSDK-13.5.0.jar *



2) Different JODA versions

017-02-13 21:11:26,243 JIRA-Bootstrap WARN      [o.twdata.pkgscanner.ExportPackageListBuilder] Package Scanner found duplicates for package 'org.joda.time.tz.data.Africa' with different versions. Files: joda-time-2.8.2.jar and ClientSDK-13.5.0.jar.
      '/appl/jira/atlassian-jira-software-7.3.0-standalone/atlassian-jira/WEB-INF/lib/joda-time-2.8.2.jar'
      '/appl/jira/atlassian-jira-software-7.3.0-standalone/atlassian-jira/WEB-INF/lib/ClientSDK-13.5.0.jar'


Same thing: Removed JODA classes from Client SDK and re-jar again.

[azlabs@sg-jira tmp]$ rm -fr org/joda
[azlabs@sg-jira tmp]$ jar cvf ClientSDK-13.5.0.jar *

3) java.lang.ClassNotFoundException: com.iplanet.dpro.session.service.cluster.ClusterMonitor

Read in detail - OPENAM-9800.

Setting com.iplanet.am.serverMode=false did not helped at all. I went into extreme!!

I downloaded the OpenAM Core and copied to JIRA lib directory.

[azlabs@sg-jira tmp]$ cp openam-core-13.5.0.jar /home/azlabs/appl/jira/jira/atlassian-jira/WEB-INF/lib/

4) '/debugconfig.properties' isn't valid

This error message kept popping up. Quite annoying. To fix it is fairly simple.

[azlabs@sg-jira ]$ cd ../jira/jira/atlassian-jira/WEB-INF/classes
[azlabs@sg-jira classes]$ touch debugconfig.properties
[azlabs@sg-jira classes]$ vi debugconfig.properties

org.forgerock.openam.debug.prefix=
org.forgerock.openam.debug.suffix=
org.forgerock.openam.debug.rotation=


Overall, it was a painful experience. The worst part was when every "seen-able" errors were fixed, I was still not able to successfully redirected to JIRA home page after OpenAM Login Page authentication.

This was when I dumped OpenAM Client SDK totally and rewrote the codes using OpenAM REST APIs (See Part 2). The decision was made after reading OPENAM-9800 in detail.

Seems like Client SDK will become legacy pretty soon.. and I personally find it not too convenient to work with.


.