Skip to content

05.-Hot-Fix-Deployment.md

Latest commit

 

History

History
301 lines (181 loc) · 11 KB

05.-Hot-Fix-Deployment.md

File metadata and controls

301 lines (181 loc) · 11 KB

Hot Fix is a type of update when Changes are made in production and need to flow in the opposite direction from production back to DEV and Test

  • In a case of ANY hotfix, even the smallest and most obvious, you MUST perform steps 1-6 and 8 below to properly record the changes in Liquibase catalog and DB controller.

1. Make Required Changes in Prod

In case of hotfix, make and test ( as much as possible) all required changes in Prod.

2. Check out the New Branch From Main

Start a new branch, exactly the same way as you would for Ongoing development.

3. Export from Prod

Connect to Prod and export all changed objects using project export -o command the same way as you would for Ongoing development.

For example, I have a hotfix for four APEX apps I need to backport now.

SQL> proj export -o APEX.104,APEX.116,APEX.202,APEX.666
The current connection //10.194.2.167:1521/pdb01.privsubnet2reg.vcnXXXXX01.oraclevcn.com CLA_DEPLOYER will be used for all operations
*** APEX_APPLICATIONS ***
Exporting Workspace CLA_INTERNAL - application 104:MCL Manager
Exporting Workspace CLA_INTERNAL - application 666:Applicant Tracking
Exporting Workspace CLA_INTERNAL - application 202:In Flight Services
Exporting Workspace CLA_INTERNAL - application 116:Financial Planning and Analysis
-------------------------------
APEX_APPLICATION              4
-------------------------------
Exported 4 objects
Elapsed 29 sec

4. Stage Changes

  • Commit changes to git to allow staging

  • Connect to Prod and stage all exported changes using project stage -o command the same way as you would for Ongoing development.

  • Commit changes to git, ideally push them to the remote, and get them approved ( if possible)

  • As usual, don’t forget to discard fake ORDS changes

  • !git restore --source=main --worktree --staged -- dist/releases/ords

  • If necessary, do other required stage-time changes, for example add-custom

5. Run Against VM

As usual, before committing changes and making them permanent, run them against VM to make sure the changeset runs

conn -name proj_vm
cd dist.
@install

Run against VM until you get a clean deployment.

6. Sync Changesets in Prod

  • Commit the staged changes
git add .
git commit -m 'hotfix-automations ready to merge'
  • Mark new changesets as run in prod:
conn -name proj_prod
cd dist --- if needed, run `pwd` to check
lb changelog-sync -changelog-file releases/main.changelog.xml
  • If possible, verify that nothing is waiting for prod deployment:
lb status  -changelog-file releases/main.changelog.xml

**Critical Note: **If your hotfix includes APEX applications, that are designed as ` runOnChange="true"` changesets, and there is no way to exclude them from rerunning. i.e. if you change app 116 as a part of your hotfix, it still will rerun during next normal deployment. This can only be prevented my manually excluding apps from a controller dist/releases/apex/apex.changelog.xml .

7. Baseline DEV

Note: For small changes, you can replace this step by manually running scripts with changes in DEV and TEST. However, it may introduce consistency problems later. So, please use it with a lot of caution, and only for the smallest changes and when you are sure there would be no merge conflicts!

For every hotfix, we need to "baseline" LB control tables in DEV to current main branch, to make sure we ONLY deploy prod changes and don't kill any unexpected work-in-progress in DEV…

  • Checkout the main branch in git
git switch main

  • connect to dev in SQLcl

  • drop all LB objects

lb drop-all
--Starting Liquibase at 2025-11-24T14:00:53.818350 using Java 17.0.13 (version 4.30.0 #0 built at 2025-04-01 10:24+0000)
INFO: The drop-all command may result in unrecoverable destructive changes to objects at 'jdbc:oracle:thin:@//10.194.2.158:1521/dev.privsubnet2reg.vcnXXXXX01.oraclevcn.com'.
To protect against unwanted drops, set --requireForce=true, which will require a --force=true flag on the command.
Learn more at https://docs.liquibase.com/dropall.

All objects dropped from CLA_DEPLOYER@jdbc:oracle:thin:@//10.194.2.158:1521/dev.privsubnet2reg.vcnXXXXX01.oraclevcn.com


Operation completed successfully.
  • Sync up the Dev environment and the main branch
lb changelog-sync -changelog-file releases/main.changelog.xml
  • At this point from LB perspective your DEV environment looks exactly as Prod before this hotfix.

DO NOT Deploy to DEV at this point! Not yet; too early!

8. Merge into Main

  • Make sure you are on the main branch by running git status

  • Merge hotfix branch into main.

  • The right way to do it would be submitting, reviewing, and aproving PR

  • It can be done locally via the following command, but it’s a risk of lost updates:

git merge hotfix_automations

Critical Notes:

  • If you have merge conflicts here, it would mean that your hotfixes conflict with approved changes deployed in TEST (this is HEAD on main branch). Because hotfixes are usually done very fast, it's very unlikely that another developer would manage to merge into main while you are working on the hotfix, however, if it happens the merge should be aborted by git merge --abort and/or if needed do git rebase
  • Another very likely threat is losing the latest changes. For example, if you hotfix a package in Prod, you can easily lose five new procedures added to the package. This is the reason why every hotfix should ideally go via PR process and peer review!

9. Deploy to DEV

Note: For small changes, you can replace this step by manually running scripts with changes in DEV and TEST. However, it may introduce consistency problems later. So, please use it with a lot of caution, and only for the smallest changes and when you are sure there would be no merge conflicts!

After a successful merge into main, we can run a deployment to DEV.

  • As a precaution, check/verify what changsets would be deployed by running
  !git switch main
  conn -name proj_dev
  lb status  -changelog-file releases/main.changelog.xml
  • If results are as expected, proceed with the deployment
 cd dist --- if needed
 @install

10. Deploy to Test

Note: For small changes, you can replace this step by manually running scripts with changes in DEV and TEST. However, it may introduce consistency problems later. So, please use it with a lot of caution, and only for the smallest changes and when you are sure there would be no merge conflicts!

The final step is deploying to Test. It’s critical to closely watch that there are no (new ) compilation errors.

conn -name proj_test
cd dist
lb status  -changelog-file releases/main.changelog.xml
@install