diff --git a/deactivate-reactivate/.vscode/settings.json b/deactivate-reactivate/.vscode/settings.json new file mode 100644 index 0000000..d41ebe4 --- /dev/null +++ b/deactivate-reactivate/.vscode/settings.json @@ -0,0 +1,5 @@ +{ + "python.languageServer": "Pylance", + "snyk.advanced.organization": "70158a6b-3a6d-4bff-aace-9699582f5950", + "snyk.advanced.autoSelectOrganization": true +} \ No newline at end of file diff --git a/deactivate-reactivate/README.md b/deactivate-reactivate/README.md index dab80aa..0f10773 100644 --- a/deactivate-reactivate/README.md +++ b/deactivate-reactivate/README.md @@ -1,64 +1,98 @@ # deactivate-reactivate-all-projects -The purpose of this script is to run through Snyk organizations, deactivate every selected project, then activate it again. That cycle helps rebuild webhooks for SCM integrations. +Cycle Snyk projects **deactivate → activate** so SCM **webhooks** can be rebuilt. You pick one or more organizations; the script walks the projects you care about and calls the Snyk API for each. -The script uses the Snyk API with **retry behavior for rate limits**: HTTP **429** responses are retried using the **`Retry-After`** header (seconds or HTTP-date), and transient **5xx** responses use exponential backoff via the `pysnyk` client. +**Optional:** **`--activate-inactive-only`** skips deactivate and only **activates** projects that are already inactive (not monitored)—useful when you only want to turn monitoring back on. -# Requirements +**Reliability:** HTTP **429** (rate limit) responses are retried using **`Retry-After`**; transient **5xx** errors use backoff from the underlying `pysnyk` client. -- Python 3.9+ recommended -- Dependencies in `requirements.txt` (`pysnyk`, `yaspin`, and their transitive packages) +--- -# Usage +## Quick start (most customers) -1. Clone this repository locally. +1. **Clone** this repo and install: -2. **Authenticate.** The script picks credentials in this order (first match wins): + ```bash + pip install -r requirements.txt + ``` - | Priority | Variables | Notes | - | --- | --- | --- | - | 1 | `SNYK_OAUTH_CLIENT_ID` and `SNYK_OAUTH_CLIENT_SECRET` | OAuth 2.0 **client credentials** (recommended for automation). Short-lived access tokens are obtained from the OAuth token endpoint and refreshed automatically before expiry. | - | 2 | `SNYK_OAUTH_TOKEN` | A short-lived OAuth **access token** (same usage as the Snyk CLI). You must refresh it yourself when it expires; use client credentials instead if you need long runs. | - | 3 | `SNYK_TOKEN` | Classic API token (`Authorization: token …`). **Not available in Snyk for Government (FedRAMP)**; see below. | +2. **Set a token** (same idea as the Snyk CLI—use a token that can manage the target orgs): - If none of these are set and you are **not** targeting Gov, the script prompts for `SNYK_TOKEN` interactively. + ```bash + export SNYK_TOKEN="your-snyk-api-token" + ``` -3. Install dependencies: +3. **Preview**, then run: ```bash - pip install -r requirements.txt + # See what would change (no API writes) + python3 main.py --orgs your-org-slug --dry-run + + # Then run for real + python3 main.py --orgs your-org-slug ``` -4. Run `main.py` with the options below. +If your Snyk data is **not** on the default US instance, set region first (see **Region** below)—wrong region usually means “wrong orgs or empty project list.” -## Snyk for Government (FedRAMP / `SNYK-GOV-01`) +**Large orgs (many thousands of projects):** add **`--workers 12`** (or 8–16) so deactivate/activate calls run in parallel. Requires `SNYK_TOKEN` (or OAuth env vars) in the environment—not an interactive prompt. -[Snyk for Government (US)](https://docs.snyk.io/snyk-data-and-governance/snyk-for-government-us) does **not** allow static API tokens. You must use **OAuth 2.0**—typically a service account with the **client credentials** grant. Access tokens are used like API keys but with `Authorization: Bearer` semantics and a short TTL; this script refreshes them when you use `SNYK_OAUTH_CLIENT_ID` / `SNYK_OAUTH_CLIENT_SECRET`. +--- -1. Create an OAuth 2.0 service account (UI or API) and store **client ID** and **client secret** securely. -2. Point the script at the Gov API endpoints using **`--environment SNYK-GOV-01`** (or `export SNYK_ENVIRONMENT=SNYK-GOV-01`). That selects `https://api.snykgov.io` for API v1, REST, and `/oauth2/token`, consistent with [regional URL documentation](https://docs.snyk.io/snyk-data-and-governance/regional-hosting-and-data-residency). -3. Export credentials and run: +## What to pass (cheat sheet) -```bash -export SNYK_ENVIRONMENT=SNYK-GOV-01 -export SNYK_OAUTH_CLIENT_ID="your-client-id" -export SNYK_OAUTH_CLIENT_SECRET="your-client-secret" -pip install -r requirements.txt -python3 main.py --orgs your-org-slug -``` +| You want to… | Use | +| --- | --- | +| Pick org(s) | **`--orgs slug-or-uuid`** (repeat or space-separate multiple). Omit to be prompted. | +| Preview only | **`--dry-run`** | +| Speed up huge runs | **`--workers N`** (try **8–16**; max **64**) | +| Limit to GitHub / GitLab / etc. | **`--origin github`** (repeat) or **`--origins github gitlab`** | +| Only turn on inactive projects | **`--activate-inactive-only`** | +| EU / AU / second US shard | **`--environment SNYK-EU-01`** (etc.) or **`export SNYK_ENVIRONMENT=...`** | +| More 429 retries per request | **`--rate-limit-attempts 12`** (default **8**) | +| Debug API / org visibility | **`-v`** / **`--verbose`** | + +Use **`--dry-run`** until the org names, region, and optional origin filter look right. -Optional: override URLs with `--api-url`, `--rest-api-url`, or `--oauth-token-url`, or the `SNYK_API_URL`, `SNYK_REST_API_URL`, and `SNYK_OAUTH_TOKEN_URL` environment variables. For security, URLs must use **HTTPS** and hostnames under **`*.snyk.io`** or **`*.snykgov.io`**. If your tenant uses another hostname, set `SNYK_ALLOW_UNVERIFIED_API_URL=1` (HTTPS is still required). +--- -## Region / environment (`--environment`) +## Authentication -Use the same region names as `snyk config environment` (for example `SNYK-US-02`, `SNYK-EU-01`, `SNYK-GOV-01`). This sets default API and OAuth token URLs. You can override individual URLs as needed. +The script uses the **first** of these that is set: -## Selecting organizations (`--orgs`) +| Order | Environment variables | When to use | +| --- | --- | --- | +| 1 | **`SNYK_OAUTH_CLIENT_ID`** + **`SNYK_OAUTH_CLIENT_SECRET`** | Automation / long runs; tokens refresh automatically. | +| 2 | **`SNYK_OAUTH_TOKEN`** | Short-lived access token (you refresh it yourself). | +| 3 | **`SNYK_TOKEN`** | Classic Snyk API token. | -- Pass one or more **organization slugs** and/or **organization IDs** (UUID from the Snyk UI or API). Slug matching is **case-insensitive**; the ID must match exactly. -- Omit `--orgs` to be prompted for orgs at runtime. +If nothing is set and you are **not** on Snyk for Government, the script may **prompt** for `SNYK_TOKEN`. **Parallel mode** (`--workers` > 1) **always** needs credentials in the environment (no prompt). -Examples: +**Snyk for Government (FedRAMP):** static **`SNYK_TOKEN` is not supported**—use OAuth client credentials and **`--environment SNYK-GOV-01`**. Short instructions: **[Snyk for Government](#optional-snyk-for-government-fedramp)** at the end of this file. + +--- + +## Region / environment + +Snyk is hosted in [regions](https://docs.snyk.io/snyk-data-and-governance/regional-hosting-and-data-residency). The script must match **your** org’s region or listing will look wrong. + +- **Default:** **`SNYK-US-01`** if you set nothing (same idea as `snyk config environment`). +- **Set once:** `export SNYK_ENVIRONMENT=SNYK-EU-01` **or** `python3 main.py --environment SNYK-EU-01 --orgs my-org`. + +| `SNYK_ENVIRONMENT` / `--environment` | Typical use | +| --- | --- | +| `SNYK-US-01` | Default US | +| `SNYK-US-02` | US (`api.us.snyk.io`) | +| `SNYK-EU-01` | Europe | +| `SNYK-AU-01` | Australia | +| `SNYK-GOV-01` | US Government / FedRAMP (OAuth only—see appendix) | + +**Advanced:** Override URLs with **`--api-url`**, **`--rest-api-url`**, **`--oauth-token-url`** or **`SNYK_API_URL`**, **`SNYK_REST_API_URL`**, **`SNYK_OAUTH_TOKEN_URL`**. URLs must be **HTTPS**; by default hostnames must be under **`*.snyk.io`** or **`*.snykgov.io`**. Other hosts: set **`SNYK_ALLOW_UNVERIFIED_API_URL=1`** (HTTPS still required). + +--- + +## Organizations (`--orgs`) + +Pass **slug** and/or **org id (UUID)**. Slug match is case-insensitive; UUID must match exactly. ```bash python3 main.py --orgs my-org-slug @@ -66,69 +100,111 @@ python3 main.py --orgs org-one org-two python3 main.py --orgs 70158a6b-3a6d-4bff-aace-9699582f5950 ``` -## Filtering by project origin (`--origin` / `--origins`) +--- -**Either form works**—pick one style, or mix them. Both apply the same filter (and are merged if you use both). +## Filtering by SCM origin (`--origin` / `--origins`) -Snyk stores an **`origin`** on each project (SCM / import source, for example `github`, `gitlab`). By default the script processes **all** projects in the chosen orgs. +By default **all** project origins in the org are processed. To limit to certain import sources, use either style (you can mix them): -To limit work to specific origins: +- **`--origin github --origin gitlab`** +- **`--origins github gitlab`** -- **`--origin ORIGIN`** — pass once per value (repeat the flag): `--origin github --origin gitlab`. -- **`--origins ORIGIN [ORIGIN ...]`** — pass several values in one go: `--origins github gitlab` (same effect as two `--origin` flags). +Values are **case-insensitive** and must match what the API returns. Many GitHub App imports show as **`github-cloud-app`**, not `github`—run **`--dry-run`** once and copy the **`Origin:`** line if you are unsure. -Matching is **case-insensitive**. If you do not pass `--origin` or `--origins`, every project origin is included. +```bash +python3 main.py --orgs my-org --origin github-cloud-app +python3 main.py --orgs my-org --origins github gitlab +``` -Examples: +--- + +## Activate only inactive projects (`--activate-inactive-only`) + +Does **not** deactivate. Only **activates** projects that are already inactive (`isMonitored` false). Same **`--origin` / `--origins`** filters apply. ```bash -# Only GitHub.com projects -python3 main.py --orgs my-org --origin github +python3 main.py --orgs my-org --activate-inactive-only --dry-run +python3 main.py --orgs my-org --activate-inactive-only --origin github +``` -# GitHub.com and GitHub Enterprise -python3 main.py --orgs my-org --origin github --origin github-enterprise +--- -# Equivalent using --origins -python3 main.py --orgs my-org --origins github github-enterprise +## Parallel workers (`--workers`) -# Only GitLab -python3 main.py --orgs my-org --origin gitlab -``` +Each project needs **two** API calls for a full cycle (deactivate, then activate). On large tenants that is mostly **waiting on the network**, so **`--workers N`** runs up to **N** of those calls at the same time (default **1** = same as always). + +- Try **8–16** first; raise if 429s stay rare, lower if runs stall on rate limits. +- Max **64**. +- **`--workers` > 1:** set **`SNYK_TOKEN`** or OAuth variables in the environment (no interactive token). +- With **N > 1**, spinners are replaced by a one-line note per org. -Common `origin` values include `github`, `github-enterprise`, `gitlab`, `bitbucket-cloud`, and `azure-repos` (exact strings depend on how projects were imported in Snyk). +```bash +export SNYK_TOKEN="your-api-token" +python3 main.py --orgs my-org --workers 12 +``` -## Rate limit retries (`--rate-limit-attempts`) +--- -- **`--rate-limit-attempts N`** — maximum number of **consecutive HTTP 429** responses to retry **per API request** before failing (default: **8**). Waits honor `Retry-After` when present. +## Rate limits (`--rate-limit-attempts`) -Example: +Max **consecutive HTTP 429** retries **per request** before failing (default **8**). Waits follow **`Retry-After`** when the API sends it. ```bash python3 main.py --orgs my-org --rate-limit-attempts 12 ``` -# Full examples +--- + +## Dry run and verbose + +- **`--dry-run`** — List what would happen; **no** deactivate/activate. +- **`-v` / `--verbose`** — Print API bases, orgs visible to the token, and per-project ids/results. + +**UI looks unchanged after a full cycle:** projects end **active** again; the goal is often **webhooks / integration**, not a lasting “off” state. If nothing processed, check **region**, **org slug/UUID**, and try without **`--origin`** once. -**Commercial (API token):** +**`RequestsDependencyWarning`:** Usually a **`chardet`** version mismatch in your environment. Use **`pip install -U -r requirements.txt`** in a venv; see `requirements.txt` for the pinned range. + +--- + +## Requirements + +- Python **3.9+** recommended +- **`pip install -r requirements.txt`** + +--- + +## Copy-paste examples + +**API token, commercial (typical):** ```bash export SNYK_TOKEN="your-api-token" pip install -r requirements.txt -python3 main.py \ - --orgs my-org-slug \ - --origin github --origin github-enterprise \ - --rate-limit-attempts 8 +python3 main.py --orgs my-org-slug --dry-run +python3 main.py --orgs my-org-slug --workers 12 ``` -Same run using the other flag: `python3 main.py --orgs my-org-slug --origins github github-enterprise --rate-limit-attempts 8`. - -**OAuth client credentials (Enterprise / Gov-compatible):** +**OAuth client credentials (good for automation):** ```bash export SNYK_OAUTH_CLIENT_ID="…" export SNYK_OAUTH_CLIENT_SECRET="…" -# For Gov: -# export SNYK_ENVIRONMENT=SNYK-GOV-01 pip install -r requirements.txt -python3 main.py --orgs my-org-slug -``` \ No newline at end of file +python3 main.py --orgs my-org-slug --workers 12 +``` + +--- + +## Optional: Snyk for Government (FedRAMP) + +[Snyk for Government](https://docs.snyk.io/snyk-data-and-governance/snyk-for-government-us) does **not** allow static API tokens. Use **OAuth 2.0** (service account **client credentials**). This script refreshes access tokens when **`SNYK_OAUTH_CLIENT_ID`** and **`SNYK_OAUTH_CLIENT_SECRET`** are set. + +```bash +export SNYK_ENVIRONMENT=SNYK-GOV-01 +export SNYK_OAUTH_CLIENT_ID="your-client-id" +export SNYK_OAUTH_CLIENT_SECRET="your-client-secret" +pip install -r requirements.txt +python3 main.py --orgs your-org-slug +``` + +Same **HTTPS / hostname** rules and optional **`SNYK_ALLOW_UNVERIFIED_API_URL=1`** as in **Region / environment** above. diff --git a/deactivate-reactivate/main.py b/deactivate-reactivate/main.py index ad7a01d..023b831 100644 --- a/deactivate-reactivate/main.py +++ b/deactivate-reactivate/main.py @@ -1,11 +1,13 @@ import argparse +import concurrent.futures import email.utils import os import sys +import threading import time import urllib.parse from datetime import datetime, timezone -from typing import Any, Dict, Optional, Set +from typing import Any, Dict, List, Optional, Set, Tuple import requests import snyk @@ -341,6 +343,21 @@ def _should_process_project(origin: str, allowed: Optional[Set[str]]) -> bool: return origin.casefold() in allowed +def inactive_projects_in_org( + org: Any, + allowed_origins: Optional[Set[str]], +) -> List[Any]: + """ + Projects in the organization that match the origin filter and are inactive + (Snyk isMonitored is False: not monitored / deactivated in the UI). + """ + return [ + p + for p in org.projects.all() + if _should_process_project(p.origin, allowed_origins) and not p.isMonitored + ] + + def _org_matches_token(org: Any, token: str) -> bool: """True if token is this org's id (UUID) or slug (case-insensitive).""" t = token.strip() @@ -349,6 +366,155 @@ def _org_matches_token(org: Any, token: str) -> bool: return org.slug.casefold() == t.casefold() +def _create_snyk_client( + args: argparse.Namespace, + urls: Dict[str, str], + *, + interactive_token: Optional[str] = None, +) -> Any: + """ + Build a Snyk client from CLI args, resolved URLs, and environment. + ``interactive_token`` is only used when no OAuth/API env credentials are set. + """ + env_name = urls["environment"] + client_kw = { + "url": urls["api_url"], + "rest_api_url": urls["rest_api_url"], + "rate_limit_max_attempts": args.rate_limit_attempts, + } + oauth_client_id = os.environ.get("SNYK_OAUTH_CLIENT_ID") + oauth_client_secret = os.environ.get("SNYK_OAUTH_CLIENT_SECRET") + oauth_access = os.environ.get("SNYK_OAUTH_TOKEN") + api_key = os.environ.get("SNYK_TOKEN") + + if oauth_client_id and oauth_client_secret: + return OAuthRateLimitAwareSnykClient( + "", + use_bearer=True, + oauth_client_id=oauth_client_id, + oauth_client_secret=oauth_client_secret, + oauth_token_url=urls["oauth_token_url"], + **client_kw, + ) + if oauth_access: + return OAuthRateLimitAwareSnykClient( + oauth_access, + use_bearer=True, + **client_kw, + ) + if api_key: + if env_name == "SNYK-GOV-01": + print( + "Snyk for Government (FedRAMP) does not support static API tokens; " + "use OAuth2 service account credentials (SNYK_OAUTH_CLIENT_ID and " + "SNYK_OAUTH_CLIENT_SECRET) or a short-lived SNYK_OAUTH_TOKEN.", + file=sys.stderr, + ) + raise SystemExit(1) + return RateLimitAwareSnykClient(api_key, **client_kw) + if env_name == "SNYK-GOV-01": + raise SystemExit( + "FedRAMP / Snyk for Government requires OAuth2. Set " + "SNYK_OAUTH_CLIENT_ID and SNYK_OAUTH_CLIENT_SECRET (recommended), or " + "SNYK_OAUTH_TOKEN for a short-lived access token. " + "Static SNYK_TOKEN is not allowed." + ) + if interactive_token is None: + raise SystemExit( + "No Snyk credentials in environment (SNYK_TOKEN or OAuth variables)." + ) + return RateLimitAwareSnykClient(interactive_token.strip(), **client_kw) + + +def _post_project_state_change( + client: Any, org_id: str, project_id: str, verb: str +) -> bool: + """POST deactivate or activate (matches pysnyk Project.activate/deactivate paths).""" + path = f"org/{org_id}/project/{project_id}/{verb}" + return bool(client.post(path, {})) + + +class _ThreadLocalClientHolder: + """One Snyk client per worker thread (OAuth refresh and requests are not shared).""" + + def __init__(self, args: argparse.Namespace, urls: Dict[str, str]) -> None: + self._args = args + self._urls = urls + self._local = threading.local() + + def get(self) -> Any: + client = getattr(self._local, "client", None) + if client is None: + self._local.client = _create_snyk_client(self._args, self._urls) + return self._local.client + + +def _project_work_row(p: Any) -> Tuple[str, str, str, str, str, bool]: + return ( + p.organization.id, + p.id, + p.name, + p.origin, + p.type, + p.isMonitored, + ) + + +def _run_parallel_project_posts( + holder: _ThreadLocalClientHolder, + rows: List[Tuple[str, str, str, str, str, bool]], + verb: str, + *, + workers: int, + verbose: bool, + log_lock: threading.Lock, + count_truthy_ok: bool, +) -> Tuple[int, List[str]]: + """ + Run deactivate or activate for many projects in parallel. + Returns (success_count, error_messages). + + When ``count_truthy_ok`` is True (activate), increments success for truthy POST results, + matching ``if ok: total_projects_cycled += 1`` in the sequential path. + When False (deactivate), any completed POST without an exception counts as success + (the sequential path does not treat a falsy body as failure). + """ + successes = 0 + errors: List[str] = [] + + def _one(row: Tuple[str, str, str, str, str, bool]) -> Tuple[bool, Optional[str]]: + org_id, project_id, name, origin, typ, is_monitored = row + try: + c = holder.get() + if verbose: + with log_lock: + print( + f" [verbose] POST {verb} project id={project_id} " + f"isMonitored={is_monitored!r} origin={origin!r}" + ) + ok = _post_project_state_change(c, org_id, project_id, verb) + if verbose: + with log_lock: + print(f" [verbose] {verb} response ok={ok!r}") + if count_truthy_ok: + return ok, None + return True, None + except Exception as err: + return False, f"{name} ({verb}): {err}" + + with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool: + futures = [pool.submit(_one, row) for row in rows] + for fut in concurrent.futures.as_completed(futures): + ok, err = fut.result() + if err: + with log_lock: + print(f"\u001b[31m {err}\u001b[0m") + errors.append(err) + elif ok: + successes += 1 + return successes, errors + + def _resolve_urls(args: argparse.Namespace) -> Dict[str, str]: """API v1 base, REST base, and OAuth2 token URL for this run.""" env_name = ( @@ -390,9 +556,10 @@ def _resolve_urls(args: argparse.Namespace) -> Dict[str, str]: def main() -> None: parser = argparse.ArgumentParser( description=( - "Deactivate and reactivate Snyk projects to rebuild SCM webhooks. " + "Deactivate and reactivate Snyk projects to rebuild SCM webhooks, " + "or with --activate-inactive-only only activate projects that are already inactive. " "Optionally limit to projects with specific Snyk project origins " - "(SCM / import source, e.g. github, gitlab)." + "(SCM / import source, e.g. github, github-cloud-app, gitlab)." ) ) parser.add_argument( @@ -411,8 +578,10 @@ def main() -> None: metavar="ORIGIN", help=( "Only process projects with this Snyk project origin (repeat for " - "several). Examples: github, github-enterprise, gitlab, " - "bitbucket-cloud, azure-repos. Omit to include all origins." + "several). Examples: github, github-cloud-app (GitHub App imports), " + "github-enterprise, gitlab, bitbucket-cloud, azure-repos. " + "Use the exact string the API returns (see dry-run). " + "Omit to include all origins." ), ) parser.add_argument( @@ -462,61 +631,77 @@ def main() -> None: "Also set via SNYK_OAUTH_TOKEN_URL." ), ) + parser.add_argument( + "--activate-inactive-only", + action="store_true", + help=( + "Only activate projects that are currently inactive (not monitored). " + "Skips deactivate entirely; does not touch already-active projects. " + "Same origin filters as without this flag." + ), + ) + parser.add_argument( + "--dry-run", + action="store_true", + help=( + "List organizations and projects that would be cycled; do not call " + "deactivate or activate. Still requires valid credentials to list data." + ), + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help=( + "Print environment/API endpoints, which organizations the token can see, " + "and per-project API details (id, monitored flag, result). Use when results " + "in the Snyk UI are unclear or something seems skipped." + ), + ) + parser.add_argument( + "--workers", + type=int, + default=1, + metavar="N", + help=( + "Parallel HTTP workers for deactivate/activate (default: 1, fully sequential). " + "Try 8–16 for large orgs; higher values increase throughput but may trigger " + "more HTTP 429 responses (retries still apply). When N>1, credentials must come " + "from the environment (SNYK_TOKEN or OAuth variables), not an interactive prompt." + ), + ) args = parser.parse_args() + if args.workers < 1: + raise SystemExit("--workers must be >= 1") + if args.workers > 64: + raise SystemExit("--workers cannot exceed 64") + allowed_origins = _parse_allowed_origins(args) input_orgs = list(args.orgs) if args.orgs else [] urls = _resolve_urls(args) env_name = urls["environment"] - client_kw = { - "url": urls["api_url"], - "rest_api_url": urls["rest_api_url"], - "rate_limit_max_attempts": args.rate_limit_attempts, - } - oauth_client_id = os.environ.get("SNYK_OAUTH_CLIENT_ID") oauth_client_secret = os.environ.get("SNYK_OAUTH_CLIENT_SECRET") oauth_access = os.environ.get("SNYK_OAUTH_TOKEN") api_key = os.environ.get("SNYK_TOKEN") - - if oauth_client_id and oauth_client_secret: - client = OAuthRateLimitAwareSnykClient( - "", - use_bearer=True, - oauth_client_id=oauth_client_id, - oauth_client_secret=oauth_client_secret, - oauth_token_url=urls["oauth_token_url"], - **client_kw, - ) - elif oauth_access: - client = OAuthRateLimitAwareSnykClient( - oauth_access, - use_bearer=True, - **client_kw, + has_env_creds = bool( + (oauth_client_id and oauth_client_secret) or oauth_access or api_key + ) + if args.workers > 1 and not has_env_creds: + raise SystemExit( + "--workers > 1 requires SNYK_TOKEN or OAuth credentials in the environment; " + "interactive API token entry is not supported with parallel workers." ) - elif api_key: - if env_name == "SNYK-GOV-01": - print( - "Snyk for Government (FedRAMP) does not support static API tokens; " - "use OAuth2 service account credentials (SNYK_OAUTH_CLIENT_ID and " - "SNYK_OAUTH_CLIENT_SECRET) or a short-lived SNYK_OAUTH_TOKEN.", - file=sys.stderr, - ) - raise SystemExit(1) - client = RateLimitAwareSnykClient(api_key, **client_kw) - else: - if env_name == "SNYK-GOV-01": - raise SystemExit( - "FedRAMP / Snyk for Government requires OAuth2. Set " - "SNYK_OAUTH_CLIENT_ID and SNYK_OAUTH_CLIENT_SECRET (recommended), or " - "SNYK_OAUTH_TOKEN for a short-lived access token. " - "Static SNYK_TOKEN is not allowed." - ) + + interactive_token: Optional[str] = None + if not has_env_creds: print("Enter your Snyk API Token") - snyk_token = input() - client = RateLimitAwareSnykClient(snyk_token, **client_kw) + interactive_token = input() + + client = _create_snyk_client(args, urls, interactive_token=interactive_token) user_orgs = [] try: @@ -525,68 +710,228 @@ def main() -> None: print(f"💥 {TOKEN_ERROR_HINT}") raise SystemExit(1) - if not input_orgs: + if args.verbose: print( - "Input the org slug(s) or org id(s) (UUID) for which you would like " - "to reactivate projects to generate webhooks." + f"\n[verbose] environment={env_name!r} api_v1={urls['api_url']!r} " + f"rest={urls['rest_api_url']!r} oauth={urls['oauth_token_url']!r}\n" + f"[verbose] this token can see {len(user_orgs)} organization(s):" ) + for o in user_orgs: + print(f" - id={o.id} slug={o.slug!r} name={o.name!r}") + print() + + if not input_orgs: + if args.activate_inactive_only: + prompt = ( + "to preview which inactive projects would be activated" + if args.dry_run + else "to activate inactive (not monitored) projects only" + ) + else: + prompt = ( + "to preview which projects would be cycled (deactivate + reactivate)" + if args.dry_run + else "to reactivate projects to generate webhooks" + ) + print(f"Input the org slug(s) or org id(s) (UUID) for which you would like {prompt}.") input_orgs = input().split() + total_projects_cycled = 0 for curr_org in user_orgs: matched = [t for t in input_orgs if _org_matches_token(curr_org, t)] if not matched: continue for t in matched: input_orgs.remove(t) - print( - "Processing" - + """ \033[1;32m"{}" """.format(curr_org.name) - + "\u001b[0morganization" - ) - - projects = [ - p - for p in curr_org.projects.all() - if _should_process_project(p.origin, allowed_origins) - ] + if args.dry_run: + if args.activate_inactive_only: + dry_intro = ( + "projects listed below are inactive (not monitored) and would be " + "activated only (no deactivate)." + ) + else: + dry_intro = ( + "projects listed below are what would be deactivated, then reactivated." + ) + print( + "\n\u001b[1;33m[DRY-RUN]\u001b[0m No API changes will be made for organization " + f'\033[1;32m"{curr_org.name}"\u001b[0m ({dry_intro})\n' + ) + else: + print( + "Processing" + + """ \033[1;32m"{}" """.format(curr_org.name) + + "\u001b[0morganization" + ) - for curr_project in projects: - curr_project_details = ( - f"Origin: {curr_project.origin}, Type: {curr_project.type}" + if args.activate_inactive_only: + projects = inactive_projects_in_org(curr_org, allowed_origins) + else: + projects = [ + p + for p in curr_org.projects.all() + if _should_process_project(p.origin, allowed_origins) + ] + + if args.verbose: + filt = ( + "inactive + origin filters" + if args.activate_inactive_only + else "origin filter" ) - action = "Deactivating" - spinner = yaspin( - text=f"{action}\033[1;33m {curr_project.name}", color="yellow" + print( + f"[verbose] matched org: id={curr_org.id} slug={curr_org.slug!r} " + f'name="{curr_org.name}" — {len(projects)} project(s) after {filt}\n' ) - spinner.write( - f"\u001b[0m Processing project: \u001b[34m{curr_project_details}\u001b[0m, Status Below👇" + + if not args.dry_run and not projects: + if args.activate_inactive_only: + warn_detail = ( + "No inactive (not monitored) projects matched the origin filter (if any). " + "No activate was called" + ) + else: + warn_detail = ( + "No projects to process in this org after applying the origin filter (if any). " + "No deactivate/activate was called" + ) + print( + f"\n\u001b[1;33mWARNING\u001b[0m: {warn_detail} for " + f"organization \"{curr_org.name}\" (id {curr_org.id}). " + "If that is unexpected, check \u001b[1m--origins\u001b[0m, " + "\u001b[1m--environment\u001b[0m (region), and that the org has imported projects.\n" ) - spinner.start() - try: - curr_project.deactivate() - spinner.ok("🆗 ") - except Exception as err: - spinner.fail("💥 ") - spinner.write(f"\u001b[31m {err}\u001b[0m") - - for curr_project in projects: - curr_project_details = ( - f"Origin: {curr_project.origin}, Type: {curr_project.type}" + if args.dry_run and projects: + if args.activate_inactive_only: + summary = ( + f"{len(projects)} inactive project(s) would be activated in this org." + ) + else: + summary = f"{len(projects)} project(s) would be cycled in this org." + print(f" \u001b[90m{summary}\u001b[0m") + elif args.dry_run: + if args.activate_inactive_only: + msg = "No inactive projects match the origin filter (if any) in this org." + else: + msg = "No projects match the origin filter (if any) in this org." + print(f" \u001b[90m{msg}\u001b[0m") + + log_lock = threading.Lock() + parallel_holder: Optional[_ThreadLocalClientHolder] = None + if not args.dry_run and args.workers > 1 and projects: + parallel_holder = _ThreadLocalClientHolder(args, urls) + mode_hint = ( + "activate only" + if args.activate_inactive_only + else "deactivate and activate" ) - action = "Activating" - spinner = yaspin( - text=f"{action}\033[1;32m {curr_project.name}", color="yellow" + print( + f" Using \033[1;33m{args.workers}\u001b[0m concurrent worker(s) for " + f"{mode_hint} ({len(projects)} project(s))." ) - spinner.write( - f"\u001b[0m Processing project: \u001b[34m{curr_project_details}\u001b[0m, Status Below👇" + + if not args.activate_inactive_only: + if args.dry_run: + for curr_project in projects: + curr_project_details = ( + f"Origin: {curr_project.origin}, Type: {curr_project.type}" + ) + print( + f" [DRY-RUN] would deactivate: \u001b[1;33m{curr_project.name}\u001b[0m " + f"(\u001b[34m{curr_project_details}\u001b[0m)" + ) + elif args.workers > 1 and parallel_holder is not None: + rows = [_project_work_row(p) for p in projects] + _run_parallel_project_posts( + parallel_holder, + rows, + "deactivate", + workers=args.workers, + verbose=args.verbose, + log_lock=log_lock, + count_truthy_ok=False, + ) + else: + for curr_project in projects: + curr_project_details = ( + f"Origin: {curr_project.origin}, Type: {curr_project.type}" + ) + action = "Deactivating" + spinner = yaspin( + text=f"{action}\033[1;33m {curr_project.name}", color="yellow" + ) + spinner.write( + f"\u001b[0m Processing project: \u001b[34m{curr_project_details}\u001b[0m, Status Below👇" + ) + spinner.start() + try: + if args.verbose: + print( + f" [verbose] POST deactivate project id={curr_project.id} " + f"isMonitored={curr_project.isMonitored!r} origin={curr_project.origin!r}" + ) + ok = curr_project.deactivate() + if args.verbose: + print(f" [verbose] deactivate response ok={ok!r}") + spinner.ok("🆗 ") + except Exception as err: + spinner.fail("💥 ") + spinner.write(f"\u001b[31m {err}\u001b[0m") + + if args.dry_run: + for curr_project in projects: + curr_project_details = ( + f"Origin: {curr_project.origin}, Type: {curr_project.type}" + ) + label = ( + "would activate (inactive)" + if args.activate_inactive_only + else "would reactivate" + ) + print( + f" [DRY-RUN] {label}: \u001b[1;32m{curr_project.name}\u001b[0m " + f"(\u001b[34m{curr_project_details}\u001b[0m)" + ) + elif args.workers > 1 and parallel_holder is not None: + rows = [_project_work_row(p) for p in projects] + n_ok, _ = _run_parallel_project_posts( + parallel_holder, + rows, + "activate", + workers=args.workers, + verbose=args.verbose, + log_lock=log_lock, + count_truthy_ok=True, ) - spinner.start() - try: - curr_project.activate() - spinner.ok("🆗 ") - except Exception as err: - spinner.fail("💥 ") - spinner.write(f"\u001b[31m {err}\u001b[0m") + total_projects_cycled += n_ok + else: + for curr_project in projects: + curr_project_details = ( + f"Origin: {curr_project.origin}, Type: {curr_project.type}" + ) + action = "Activating" + spinner = yaspin( + text=f"{action}\033[1;32m {curr_project.name}", color="yellow" + ) + spinner.write( + f"\u001b[0m Processing project: \u001b[34m{curr_project_details}\u001b[0m, Status Below👇" + ) + spinner.start() + try: + if args.verbose: + print( + f" [verbose] POST activate project id={curr_project.id} " + f"isMonitored={curr_project.isMonitored!r} origin={curr_project.origin!r}" + ) + ok = curr_project.activate() + if args.verbose: + print(f" [verbose] activate response ok={ok!r}") + if ok: + total_projects_cycled += 1 + spinner.ok("🆗 ") + except Exception as err: + spinner.fail("💥 ") + spinner.write(f"\u001b[31m {err}\u001b[0m") if input_orgs: print( @@ -597,6 +942,13 @@ def main() -> None: ) ) + if not args.dry_run and not input_orgs and total_projects_cycled == 0: + # All requested org names were matched, but no activate returned success (0 projects, or errors). + print( + "\n\u001b[1;33mNote\u001b[0m: no successful \u001b[1mactivate\u001b[0m in this run. If you " + "expected projects, re-check region (\u001b[1m--environment\u001b[0m), org slug/UUID, " + "and \u001b[1m--origins\u001b[0m; use \u001b[1m--verbose\u001b[0m to show API details.\n" + ) if __name__ == "__main__": main() diff --git a/deactivate-reactivate/requirements.txt b/deactivate-reactivate/requirements.txt index 6a530c6..bd2ebcb 100644 --- a/deactivate-reactivate/requirements.txt +++ b/deactivate-reactivate/requirements.txt @@ -1,3 +1,7 @@ pysnyk==0.9.19 yaspin==3.0.2 urllib3==2.6.3 +requests>=2.28.0 +# requests imports chardet first if present; chardet 6.x fails requests' version check +chardet>=3.0.2,<6.0.0 +charset-normalizer>=2.0.0,<4.0.0