In-place Upgrade vs Greenfield Upgrade

There are different school of thoughts when it comes to upgrade - In-place vs Greenfield (or Side-by-side). 

In-place Upgrade could mean 2 things:
U1. Executing an upgrade wizard that comes with the software
U2. Install the latest version on another disk partition, separated from the current version that is Live.

I refer to the former in this context.

When cost is involved, most will go with In-place Upgrade as a greenfield upgrade requires another identical set of hardware and operating system. This, I totally agree.

When customers have cost constraint, I will usually recommend U2 for major version upgrade. Minor version upgrades are usually bug and security fixes. I will usually use approach U1. No brainer.


Now, there are 3 groups of specialists who will give you different opinions when you ask them if it would be wise to perform upgrade using approach U1:
S1. Product Engineers
S2. Product Presales
S3. Product Professional Service Engineers or Service Delivery Partners


Now, if the wizard is written by you (S1), would you say "No, U1 is not the better approach"? This group of specialists also tend to assume that the product is deployed out-of-the-box with no or minimal customization. My foot. Now, even if they agree that the product is highly customized, I come to realize that QA of the wizard is usually not well carried out. There are few too many edge cases that they would not be able to think of in their own labs. (can you blame them? no. product engineers are usually the ones who do not like to meet and interact with customers. thus they do not get to know the business-world requirements and they definitely do not have the full picture of how the products they developed are deployed in real world)

What would S2 say? You know the KPI of a presales? Can you say that the product that you are selling so hard has no seamless upgrade wizard in front of your customers? This group of specialists has the least hands-on. They are really good in sweet talking and reading documentation. You ask them to set up the software from scratch. They look at you like you are an alien. "Dude... there's a VM with everything installed. Come on...". Asking S2 deep technical/operational/migration questions is usually a waste of time, unless you do not mind hearing white lies. (well, to be honest, they do not want to cause harm. just that they do not have the right hands-on experience)

S3 is the group of specialists that are doing the shit job, even to the extend of implementing empty promises from S2 that are totally out-of-scope. S3 usually tells the hard truth, which customers would not like to hear at times. "well, I have paid tons of dollars for this piece of software? I don't care. You get it fixed to what was being promised". However, they have years of hands-on and have seen many types of deployments. They also have many years of migration experience. (ok, I have also seen delivery partners that are incompetent. there was once a big customer in SG asking this delivery partner how to extend their current SSO infrastructure to support multiple domains. to our surprise, a consultant from this delivery partner suggested to set up a totally new environment to support this new domain. omg! most SSO products can support cross-domain single sign-on these days. It's a out of the box feature)


Azimuth has been in the Identity business for a long time and we have performed numerous upgrades for customers. Softwares, especially IDM and SSO, are "fragile" animals. IDM products are usually highly customized, while SSO products are usually deployed as a farm of server nodes to handle huge traffic.

SSO solutions usually have low tolerance for downtime. When in-place upgrade is adopted, there are many questions to ask. 

1. How do you upgrade to a new major release then? Say there are 10 nodes to upgrade.
2. How do you upgrade to a release that is at least 2-3 versions apart? 
3. How do you upgrade the underlying configuration store, if any? Say there are 4 nodes to upgrade.
4. How do you ensure protected applications work as before? Say at least 50 applications are involved which belong to 10 application teams? 
5. How do you ensure customized codes work as before? 
6. How do you ensure deprecated functions that are replaced with new functions in the new release work as before? 
7. How do you ensure new features requested by customers work as promised? 

How do you complete the above all within a narrow downtime window of 5 hours (12am to 5am)? Nearly impossible.

A presales will never take the above into consideration when being asked about which is the better upgrade approach. Stick with product documentation is always the right approach. Crap.

One would also question why would there be challenge and surprise if well-conducted UAT is carried out prior to Production upgrade. From experience, I can conclude that most UAT environment is not identical to Production. Seldom, less than 10%. Why take the risk and bang your balls for that tiny 5 hours window? It's not worth taking the risk. Rolling back, investigating why, perform another round of upgrade, activating the same whole village of application teams to perform end-to-end testings are just not worth the trouble. In fact, reputation is at stake.

Some customers are also one of the kind. They are not technical enough, yet they want to have a say. So they go around asking opinion from various sources. A service delivery partner does not come from the Product organization. The customers will take the advice from the Product Presales over what the Service Delivery Partner recommends. "How can I not believe what XXX has said? XXX is from the same Product organization. He sure knows better than you."

Oh, really? 

Now, who is performing the upgrade? What if something goes wrong during upgrade? Whose responsibility is that? To ask a service delivery partner to perform upgrade again at a fixed cost (we practice fixed cost in Asia) is really not a fair deal.