Database copy error crash

[jchu-sandag]

Proposed changes

Edits the copy tool such that the script uses a Windows robocopy in the event that the EMME database cannot be read by the EMME script.

Impact

Recent runs of ABM have all resulted in an error in which the EMME databases could not be read by the EMME script on the remote drive. Thus, the databases could not be copied over from the local drive to the remote drive, forcing users to manually go into the system and close the programs before copying over the rest of the EMME databases. This caused downtime in metrics that required these databases to be within the remote drive. This update will instead initiate a Windows robocopy in the event that the EMME script fails to copy over all the databases to the remove drive.

Types of changes

What types of changes does your code introduce to ABM?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

How has this been tested?

• Full 2022 scenario run
• Checks for byte size between both remote and local drive
  • This check has not been checked within a scenario run and has only been checked by manual copy locally

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings

Further comments

Although not the explicit usage of the following method, the _copy_dir method is a good general copy method for instances where other copying methods fail or run into errors. In the future, an additional update to separate the file masks into file_masks and dir_masks could be beneficial in case any specific directories and files would want to be excluded in a single step. However, this update is not within the scope of this PR and would require further testing.

[cundo92]

@jchu-sandag thanks for working on and applying this change. I was able to confirm successful copy of a full model run, including the emme_project directory.

[aber-sandag]

I am somewhat concerned about the effects of overwriting Emme files that are currently in use by Emme. Could you test the following:

• The file copy output shows that all files were copied successfully when copying emme_project with robocopy? Are there any warnings?
• After running, when the project is opened from T drive on another server, can Emme matrices and other Emme data be accessed in Emme?

[cundo92]

I am somewhat concerned about the effects of overwriting Emme files that are currently in use by Emme. Could you test the following:

• The file copy output shows that all files were copied successfully when copying emme_project with robocopy? Are there any warnings?
• After running, when the project is opened from T drive on another server, can Emme matrices and other Emme data be accessed in Emme?

@jchu-sandag feel free to use my test run at "T:\STORAGE-63T\car_tests\log_test\2022_copyTest" for these follow up comments

[JiaXu1024]

I am somewhat concerned about the effects of overwriting Emme files that are currently in use by Emme. Could you test the following:

• The file copy output shows that all files were copied successfully when copying emme_project with robocopy? Are there any warnings?
• After running, when the project is opened from T drive on another server, can Emme matrices and other Emme data be accessed in Emme?

@jchu-sandag feel free to use my test run at "T:\STORAGE-63T\car_tests\log_test\2022_copyTest" for these follow up comments

Hi @jchu-sandag , have you done the test mentioned in Alexander's comments? Any updates?

[jchu-sandag]

While performing some test scenarios, some runs crashed with errors including a Fatal Error 5 (regular scenario run with no update to the script) and connection error to EMME (scenario run with updates from the current PR). These runs were done on two separate servers but both occurred over the weekend run. All TOD databases and transit databases are within the local drive, so it is possible that the scenario run update completed a run but was corrupted during the copy step. @aber-sandag proposed a possible reason for why this might have happened may be the logbook\project.mlbk was copied over the file was still being written into.

More tests will be run to see if this is just a onetime error copying over from local to remote drive or if it may be a common error that will occur. (It's likely that this may be due to a corrupted file during a copy step, since neither the logbook nor the databases can be accessed using the EMME software. However, previous runs done by @cundo92 and I have successfully copied and deleted the files from the local drive and can also be accessed from different servers). Alexander also proposed to retry attempts to close the EMME modeler using methods _m.Modeller().desktop.project.save() and _m.Modeller().desktop.close() to mimic what is done during a manual copy. This should be done to ensure that nothing is being written into or corrupted while the emme_project files are being copied over from local to remote drive.