feat(logging): Bug 2039779 Copy chain_of_trust.log to live_backing.log on CoT verification failure

[hneiva]

When verify_chain_of_trust fails, run_task never runs and live_backing.log is never created. Treeherder parses live_backing.log as the task's failure log, so users could not see why CoT verification failed.

Copy chain_of_trust.log to live_backing.log on verification failure so the verification output is visible from TH.

[jcristau]

Instead of copying the file we can create a link or redirect artifact?

[hneiva]

What does redirecting the artifact mean? Just moving instead of copying? Do we know for a fact that there's nothing externally pointing to chain_of_trust.log?

What is the benefit of either vs just copying the file?

[jcristau]

https://docs.taskcluster.net/docs/reference/platform/queue/api#createArtifact describes link and redirect artifacts. The benefit is to make it obvious that they're the same artifact, just like on generic-worker live.log redirects to live_backing.log.

[jcristau]

just catch CoTError

[hneiva]

That would make it skip unforeseeable exceptions, which we don't want.

scriptworker/src/scriptworker/cot/verify.py

Lines 2053 to 2074 in 0ae448c

	try:
	# build LinkOfTrust objects
	if check_task:
	await add_link(chain, chain.name, chain.task_id)
	await build_task_dependencies(chain, chain.task, chain.name, chain.task_id)
	# download the signed chain of trust artifacts
	await download_cot(chain)
	# verify the signatures and populate the ``link.cot``s
	verify_cot_signatures(chain)
	# download all other artifacts needed to verify chain of trust
	await download_cot_artifacts(chain)
	# verify the task types, e.g. decision
	await verify_task_types(chain)
	# verify the worker_impls, e.g. docker-worker
	await verify_worker_impls(chain)
	await trace_back_to_tree(chain)
	except (BaseDownloadError, KeyError, TypeError, AttributeError) as exc:
	log.critical("Chain of Trust verification error!", exc_info=True)
	if isinstance(exc, CoTError):
	raise
	else:
	raise CoTError(str(exc))

This block in verify_chain_of_trust() only captures a set of expected exceptions, but doesn't capture json, OS, Value or aiohttp errors.