[If you missed Part 2 Blog: How to Prepare]
Once you have all of the prerequisite tasks complete including the project plan and team, regression testing suite, testing strategy, and the delivery of a new test environment or refresh of the existing test environment, if applicable, you are ready to begin the real work of actually receiving the new, refreshed, or upgraded environments, performing the pre and post upgrade tasks, and performing the regression testing to determine if issues exist in the environment that must be resolved before you are ready to upgrade in Production.
Upgrade the Test Environment-
Emtec recommends refreshing your test environment before upgrading so you have recent Production data to test with after the upgrade. You should plan for some testing after the refresh just to validate the environment and make sure it is working correctly. We recommend focusing on testing workflows, reporting, and RICE items you have implemented such as integrations with external systems. If you have provisioned a new test environment, we recommend full regression testing be performed to ensure all functionality is working correctly.
Once you have checked out the new or refreshed test environment, identified any issues and resolved as many issues as possible, you are ready to proceed with the upgrade of the environment. The next step is to perform the pre-upgrade steps recommended by Oracle prior to Oracle performing the upgrade. At this point, preserve your custom reports by moving them to the Shared>Custom folder, publish any sandboxes that you want to keep and delete the others, and perform the accounting period close process on any open past periods.
You do not have to close the current open period. It is a good idea to create accounting in the sub-ledgers to ensure all transactions are accurately accounted for in the general ledger. Also run a few key financial reports, maybe one from each sub-ledger application and the GL, just prior to the upgrade and save them to compare to the same reports after the upgrade as an audit control.
Oracle will take the test environment on a Tuesday morning and has a 72 hour window to perform the upgrade so you can expect to have the upgraded test environment back on Friday morning to start your post upgrade steps as outlined in your plan.
Test, Test, Test-
After the post upgrade steps have been completed, it is time to start the system regression testing. The regression testing suite plays a valuable role as your guide to efficiently performing all of the test cases needed to validate the functionality in the Cloud applications, record the results of the testing, and identify any issues caused by the upgrade.
Promptly report any issue to Oracle Support by submitting an Oracle SR (Support Request). Oracle will help resolve the issue using their knowledge base, troubleshooting tools, and expert resources. The SR should have REL10UPG at the beginning of the Problem Summary so Oracle Support can use this tag to prioritize the upgrade issues appropriately.
Also use the Oracle Success Manager assigned to your company to escalate the high priority issues for faster resolution:
- We recommend that the core project team conduct the first round of testing to identify any technical issues and work to resolve as many as possible before the business resources start their user acceptance testing.
- Remember, business testers may have limited time available (because they are still doing their “regular” jobs) so it will likely take them longer to complete their testing than it took the core team.
Please note that Oracle will continue to perform monthly patch updates on your test and Production environments, even when you are in the middle of your upgrade. Pay attention to when these patches will be applied so you can complete your focused testing per your regular monthly patch routine regardless of the upgrade in progress.
Use the Regression Test Suite tracking spreadsheet to track the status of all test scenarios. You might want to include a second set of tester name and summary status columns on your tracking spreadsheet for the Business testers and their results for each test case. The business tester should enter their actual results in the detailed test script, of course. We recommend that the test lead should keep track of completed tests and enter the summary status from each test script and update the tracking spreadsheet.
The summary status page can also be used as a communication tool to keep the project team up to date on which test cases have been completed and whether the test cases passed or failed. All failed test cases should be reviewed by a core project team member to determine if the issue is an Oracle problem for which an SR should be submitted. In some cases, the test script may not be clear or accurate and could have caused the issue. In other cases, it may be a setup or security issue that can be resolved in-house before sending the test script sent back to the business tester with a request to retest the test case.
In any event, an Issue Log should be created and all issues should be documented, progress to resolution tracked, and the actual solution should be documented. The issue log and test scenarios status should be reviewed often with the team to help everyone understand current status, outstanding tasks and issues, and how the issues were resolved so they understand root cause and actual solution. We also recommend tracking Action Items, along with the Issues, so they can also be assigned to a resource, tracked, and completed.
One other item to note, Oracle may respond to your SR’s and let you know that the issue will be fixed in a future patch release. If that is the case, make sure there is a suitable work around (fully documented, please) that can be used until the patch is applied. If no suitable work-around is available, go back to Oracle and request an expedited solution as the issue will require a delay of your upgrade schedule. Include your Oracle Success Manager in these communications so they can help expedite a patch or find a suitable work around so the upgrade can stay on track.
Plan the Production Cutover-
The Production Cutover plan will mirror the pre-upgrade and post-upgrade steps documented in the Test environment upgrade plan. Pull the project team together to review those steps, add in any others that were not a part of the original list, and determine any others tasks that are required for production that were not a part of the Test upgrade process. Be sure to include any post upgrade steps surrounding issues and resolutions uncovered in the Test upgrade process. You will also need a plan to distribute new upload spreadsheet templates that are needed with the new release. Some spreadsheet templates will not change while others will be updated – make sure you are using the latest version on your new release upgrade.
Additional steps needed for the Production upgrade that were not needed for the Test upgrade include but are not limited to:
- Follow whatever change control process you have in place. Add those steps at the top of the plan to ensure that the required documentation is created and submitted for approval early enough to obtain approval prior to starting the upgrade in production.
- Identify and place any scheduled processes on hold (several hours) before the upgrade is scheduled to start. You don’t want automatic processes to kick off during the upgrade process.
- You will need a post upgrade task to release those scheduled processes after the upgrade is complete. Have someone on the team monitor those processes to ensure they are complete successfully.
- Identify any external processes that are sending data files to Oracle. Place those on hold before the upgrade process begins and be sure to re-start them after the upgrade.
- Include the need to communicate with the business and end users of Oracle applications advising them of the Production upgrade schedule and any changes that they will notice when they login after the upgrade. You should also communicate the procedure they should follow for support if they run into issues using Oracle Cloud after the upgrade.
Based on your experiences, there may be some refinements or extensions to test scenarios that were identified along the way. Upgrade your plan and test scenarios to include these refinements so that they are included in the next upgrade process.
You will also want to identify critical business processes that should be tested as soon as possible after the upgrade and/or monitored during the first week to be sure issues don’t recur after initial testing. Some of these, like those for reporting and security, can be completed by the core project team. Other test cases will require end user participation as they use the upgraded Production environment. Identify the business people that will be asked to perform those tests. Review the cutover plan and test cases with involved individuals so they know what they are assigned to do and when to do it.
You may want to include such items as:
- All Banking integrations including Payments, Statements, Positive Pay, and Credit Card transactions. If you have more than one bank, make sure you test what is appropriate for each bank.
- Journals, Invoices, and expenses both online, using spreadsheet uploads, and from mobile devices for expenses.
- Verify that transactions are being routed correctly for approval and approval notifications are being received.
- Check that invoices sent to scanning and OCR services are being received for processing.
Our experience with Production upgrades has taught us that you cannot just test it once and forget about it. There may be issues that don’t reveal themselves until regular processes are resumed.
Execute the Production Cutover Plan and Go-Live-
At this point, you have done everything you can do to plan and prepare for the Production upgrade. The weeks prior to and after the Production upgrade should be focused on executing the Cutover Plan, ensuring all the tasks are completed, responding to any issues encountered, and working with Oracle Support to ensure your issues are understood and timely resolution is achieved.
The expectation should be set with everyone that there may be some issues encountered after the upgrade but they will be worked to resolution as quickly as possible. Tell the users that they may encounter issues during the first few days after the upgrade and let them know how to report the issues they encounter. Also let them know that things should get back to normal by the end of the first week on the new release.
As noted previously, we recommend that key business processes and system integrations be monitored by the core project team during the first week after the Production upgrade so any issues are immediately recognized and investigated.
Stay tuned for our Part 4 of our ERP Cloud Upgrade blog series. Have questions? Comment below or send us an email to firstname.lastname@example.org.