Cache Shell Script of `spack load` and `spack env activate` by RikkiButler20 · Pull Request #47755 · spack/spack

RikkiButler20 · 2024-11-23T02:39:33Z

This will fix the problem of spack load and spack env activate taking too long and currently iterating over all the package's dependency's python package files. This will also prevents the drifting of the installed spec from the current, updated version on Spack

adamjstewart · 2024-11-23T11:14:05Z

Very excited for this and happy to test on fish!

aweits · 2024-11-24T13:47:59Z

What if we cached the environment changes at install-time instead? That would also solve issues related to allowing older installs to still function/load... (willing to help with this, would allow me to jettison some local code that is getting harder and harder to maintain)

tgamblin · 2024-11-24T14:06:12Z

What if we cached the environment changes at install-time instead?

This is the plan. There are some details in the slack discussion linked from #47603, and the bit about “package drift” there.

i think @RikkiButler20 is doing this in steps (see checklist) — probably need one or two more to move the logic to package install time and env install time.

@aweits I think if you want to help, iterating on reviews would be great so we can get this in for 1.0!

RikkiButler20 · 2024-11-27T01:07:41Z

@spackbot Fix style

spackbot-app · 2024-11-27T01:07:44Z

Let me see if I can fix that for you!

spackbot-app · 2024-11-27T01:09:19Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/load.py
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
2 files reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2024-12-05T01:15:56Z

@spackbot Fix style

spackbot-app · 2024-12-05T01:15:59Z

Let me see if I can fix that for you!

spackbot-app · 2024-12-05T01:17:28Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/test/cmd/load.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/cmd/load.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/variant.py:131: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
Found 4 errors in 2 files (checked 624 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

adamjstewart · 2025-01-15T11:01:48Z

Going to unsubscribe from this PR while it's still a WIP (too many notifications) but please ping me to review fish support once it is complete.

RikkiButler20 · 2025-01-16T20:27:11Z

@spackbot Fix style

spackbot-app · 2025-01-16T20:27:14Z

Let me see if I can fix that for you!

spackbot-app · 2025-01-16T20:28:19Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/load.py
  lib/spack/spack/hooks/__init__.py
  lib/spack/spack/hooks/cache_shell_script.py
  lib/spack/spack/test/environment_modifications.py
  lib/spack/spack/test/hooks/cache_shell_script.py
  lib/spack/spack/test/util/environment.py
  lib/spack/spack/util/environment.py
==> Running import checks
import check requires Python 3.9 or later
  import checks were clean
==> Running isort checks
Fixing /tmp/tmpu87evsa3/spack/lib/spack/spack/test/hooks/cache_shell_script.py
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/hooks/cache_shell_script.py
reformatted lib/spack/spack/util/environment.py
All done! ✨ 🍰 ✨
2 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/cmd/load.py:12: [F401] 'spack.user_environment as uenv' imported but unused
lib/spack/spack/cmd/load.py:13: [F401] 'spack.util.environment' imported but unused
  flake8 found errors
==> Running mypy checks
lib/spack/spack/util/environment.py:662: error: Item "NameModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/util/environment.py:662: error: Item "NameValueModifier" of "Union[NameModifier, NameValueModifier]" has no attribute "_cache_str"  [union-attr]
lib/spack/spack/version/version_types.py:135: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Tuple[str]")  [assignment]
lib/spack/spack/variant.py:130: error: Unsupported right operand type for in ("Union[Collection[Any], Callable[..., Any]]")  [operator]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [misc]
lib/spack/spack/build_environment.py:171: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [misc]
Found 6 errors in 4 files (checked 636 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

Signed-off-by: Kayla Butler <[email protected]>

RikkiButler20 · 2026-06-09T03:20:05Z

@spackbot fix style

spackbot-app · 2026-06-09T03:20:08Z

Let me see if I can fix that for you!

spackbot-app · 2026-06-09T03:21:04Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, ruff-format, ruff-check, mypy
==> Running import checks
  import checks were clean
==> Running ruff-format checks
  ruff-format checks were clean
==> Running ruff-check checks
E501 Line too long (100 > 99)
   --> lib/spack/spack/test/hooks/generate_spec_scripts.py:104:100
    |
102 |         )
103 |         assert re.search(
104 |             f"_spack_env_remove_value {uenv.spack_loaded_hashes_var} {pkg.dag_hash()} {os.pathsep}",
    |                                                                                                    ^
105 |             unload_script,
106 |         )
    |
  ruff-check found errors
==> Running mypy checks
Success: no issues found in 639 source files
  mypy checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

RikkiButler20 · 2026-06-09T03:32:10Z

@spackbot fix style

spackbot-app · 2026-06-09T03:32:13Z

Let me see if I can fix that for you!

spackbot-app · 2026-06-09T03:33:08Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: import, ruff-format, ruff-check, mypy
==> Running import checks
  import checks were clean
==> Running ruff-format checks
  ruff-format checks were clean
==> Running ruff-check checks
E501 Line too long (100 > 99)
   --> lib/spack/spack/test/hooks/generate_spec_scripts.py:104:100
    |
102 |         )
103 |         assert re.search(
104 |             f"_spack_env_remove_value {uenv.spack_loaded_hashes_var} {pkg.dag_hash()} {os.pathsep}",
    |                                                                                                    ^
105 |             unload_script,
106 |         )
    |
  ruff-check found errors
==> Running mypy checks
Success: no issues found in 639 source files
  mypy checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I wasn't able to make any further changes, but please see the message above for remaining issues you can fix locally!

Signed-off-by: Kayla Butler <[email protected]>

…pack-load-and-activate

…/cache-spack-load-and-activate

Signed-off-by: Richarda Butler <[email protected]>

…/cache-spack-load-and-activate

…kkiButler20/spack into feature/cache-spack-load-and-activate

RikkiButler20 added feature A feature is missing in Spack commands shell-support labels Nov 23, 2024

RikkiButler20 self-assigned this Nov 23, 2024

spackbot-app Bot added core PR affects Spack core functionality tests General test capability(ies) labels Nov 23, 2024

adamjstewart added this to the v0.24 milestone Nov 23, 2024

becker33 modified the milestones: v1.0.0, v1.0.0 stretch goals Nov 25, 2024

tgamblin modified the milestones: v1.0.0 stretch goals, v1.0.0 Nov 26, 2024

aweits reviewed Dec 2, 2024

View reviewed changes

Comment thread lib/spack/spack/cmd/load.py Outdated

spackbot-app Bot added the utilities label Dec 18, 2024

spackbot-app Bot added the documentation Improvements or additions to documentation label Jan 3, 2025

tldahlgren reviewed Jan 14, 2025

View reviewed changes

Comment thread lib/spack/spack/hooks/cache_shell_script.py Outdated

tldahlgren reviewed Jan 14, 2025

View reviewed changes

Comment thread lib/spack/spack/test/hooks/cache_shell_script.py Outdated

RikkiButler20 marked this pull request as draft January 16, 2025 20:26

Us correct separator

5730c29

Signed-off-by: Kayla Butler <[email protected]>

[@spackbot] updating style on behalf of RikkiButler20

ab496c2

RikkiButler20 added 3 commits June 8, 2026 21:01

Fix long line

256172b

Signed-off-by: Kayla Butler <[email protected]>

commands update-completion

9e64d57

Signed-off-by: Kayla Butler <[email protected]>

Simplify asserts

61b45f4

Signed-off-by: Kayla Butler <[email protected]>

RikkiButler20 force-pushed the feature/cache-spack-load-and-activate branch from c1c1829 to 61b45f4 Compare June 10, 2026 19:13

RikkiButler20 added 3 commits June 10, 2026 15:44

stdout load & unload cmds

d68d37e

Signed-off-by: Kayla Butler <[email protected]>

Use the right syntax to source scripts

eede5ae

Signed-off-by: Kayla Butler <[email protected]>

Fix shell check when load/unload

c665310

Signed-off-by: Kayla Butler <[email protected]>

RikkiButler20 force-pushed the feature/cache-spack-load-and-activate branch 2 times, most recently from e692553 to 94e1bd7 Compare June 11, 2026 21:04

Add quotes to fish environ mods

33c8148

Signed-off-by: Kayla Butler <[email protected]>

RikkiButler20 force-pushed the feature/cache-spack-load-and-activate branch from d8020be to 33c8148 Compare June 15, 2026 22:37

becker33 and others added 2 commits June 16, 2026 15:09

try refactoring fish setup-env script

877fba6

Merge branch 'develop' of github.com:spack/spack into feature/cache-s…

178a112

…pack-load-and-activate

RikkiButler20 removed binary-packages utilities ci Issues related to Continuous Integration labels Jun 17, 2026

Merge branch 'develop' of github.com:RikkiButler20/spack into feature…

4da57a7

…/cache-spack-load-and-activate

tgamblin modified the milestones: v1.2.0, v1.3.0 Jun 21, 2026

RikkiButler20 and others added 3 commits June 22, 2026 11:48

Merge branch 'develop' into feature/cache-spack-load-and-activate

2c4ddcb

Signed-off-by: Richarda Butler <[email protected]>

Merge branch 'develop' of github.com:RikkiButler20/spack into feature…

f301de6

…/cache-spack-load-and-activate

Merge branch 'feature/cache-spack-load-and-activate' of github.com:Ri…

3b570ca

…kkiButler20/spack into feature/cache-spack-load-and-activate

Conversation

RikkiButler20 commented Nov 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Nov 23, 2024

Uh oh!

aweits commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RikkiButler20 commented Nov 27, 2024

Uh oh!

spackbot-app Bot commented Nov 27, 2024

Uh oh!

spackbot-app Bot commented Nov 27, 2024

Uh oh!

Uh oh!

RikkiButler20 commented Dec 5, 2024

Uh oh!

spackbot-app Bot commented Dec 5, 2024

Uh oh!

spackbot-app Bot commented Dec 5, 2024

Uh oh!

Uh oh!

Uh oh!

adamjstewart commented Jan 15, 2025

Uh oh!

RikkiButler20 commented Jan 16, 2025

Uh oh!

spackbot-app Bot commented Jan 16, 2025

Uh oh!

spackbot-app Bot commented Jan 16, 2025

Uh oh!

RikkiButler20 commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

RikkiButler20 commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

spackbot-app Bot commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

RikkiButler20 commented Nov 23, 2024 •

edited

Loading

aweits commented Nov 24, 2024 •

edited

Loading

tgamblin commented Nov 24, 2024 •

edited

Loading