Updates for runZero 5.1

.github/pull_request_template.md

@@ -31,8 +31,8 @@ Please provide a brief description of your changes and the context for this inte
 
 - [ ] **New Integration Folder:** A new folder has been created for the integration.  
 - [ ] **Updated README:** The README has been updated based on the boilerplate to reflect the new integration details.  
-- [ ] **custom-integration.star File:** The `custom-integration-<name>.star` file has been created/updated as required.  
-- [ ] **config.json File:** The `config.json` is updated with the `name` (product name) and `type` (inbound or outbound) of integration.
+- [ ] **Integration script:** The `<name>.star` file has been created/updated as required.  
+- [ ] **Embedded CONFIG:** The script's `CONFIG` block declares the `name` (product name), `type` (inbound, outbound, or internal), parameters, and any shared option-set includes.
 
 ---
 

.github/skills/create-custom-integration/SKILL.md

@@ -0,0 +1,191 @@
+---
+name: create-custom-integration
+description: 'Create a new runZero custom integration from vendor API documentation and a brief description of the data to import. Use when asked to build, scaffold, generate, or add an inbound Starlark integration. Requires an explicit foreign-ID uniqueness, stability, scope, and cardinality decision before writing code so repeated ingest does not create duplicate assets.'
+argument-hint: '<vendor documentation URL> -- import <assets and fields>'
+user-invocable: true
+disable-model-invocation: false
+---
+
+# Create a Custom Integration
+
+Build a production-ready inbound runZero Starlark integration from vendor
+documentation and a short import brief. The output is a new repository folder
+containing `<integration-name>.star` and `README.md`, plus regenerated catalog
+metadata when available.
+
+## Required Inputs
+
+Collect these before implementation:
+
+1. A public or accessible link to the vendor API documentation.
+2. A brief statement of which vendor objects and fields should become runZero
+   assets, software, services, vulnerabilities, tags, or custom attributes.
+3. Authentication and deployment constraints if they are not documented.
+
+If the documentation link or import brief is missing, ask for it. If the docs
+need authentication and cannot be read, ask for exported docs, an OpenAPI file,
+or representative redacted request/response samples.
+
+## Source of Truth
+
+Read these repository files before coding:
+
+- [Authoring rules](../../../AGENTS.md)
+- [Starlark helper reference](../../../docs/starlark-helpers.md)
+- [Boilerplate script](../../../boilerplate/boilerplate.star)
+- [v1 to v2 migration guide](../../../docs/migration-v1-to-v2.md)
+- [Identity decision reference](./references/identity-decision.md)
+
+Inspect one or two existing integrations with similar authentication,
+pagination, and data shape. Reuse repository patterns and helper modules rather
+than copying generic Python or inventing APIs.
+
+## Workflow
+
+### 1. Research the Vendor API
+
+Read the supplied documentation and identify:
+
+- Base URL and required API version.
+- Authentication flow, token lifetime, and required scopes.
+- Inventory endpoints and object relationships.
+- Pagination mechanism, maximum page size, and termination condition.
+- Rate limits, retry guidance, and relevant response headers.
+- Fields for identity, hostname, IP, MAC, OS, hardware, timestamps, tags,
+  software, services, vulnerabilities, and custom attributes.
+- Whether one response row represents an asset, an event, an interface, an
+  agent installation, a user/device association, or another child record.
+
+Prefer the documented API contract over examples or field names that merely
+look plausible. Record the documentation URLs used for authentication,
+pagination, object schema, and identity in the integration README.
+
+### 2. Complete the Asset Identity Gate
+
+Do not write the integration until the foreign-ID decision is explicit.
+Complete the decision record from
+[identity-decision.md](./references/identity-decision.md) and include it in the
+new integration README under `## Asset identity`.
+
+The key question is not only "is this field unique?" The field must be:
+
+- Stable across repeated polls and normal asset lifecycle changes.
+- Unique within a known scope.
+- One-to-one with the asset type described in the import brief.
+- Present on every record that will be imported.
+- Not an event, session, interface, finding, temporary, or installation ID when
+  multiple such records can describe one asset.
+
+Choose exactly one verdict:
+
+1. **Authoritative foreign ID.** Use a namespaced ID such as
+   `vendor:<account-or-tenant>:<source-id>`. Keep default ID matching. If MAC,
+   IP, or hostname are expected to churn while the vendor ID remains
+   authoritative, use `no-mac-break no-ip-break no-name-break`.
+2. **Scoped foreign ID.** The source ID is unique only inside an account,
+   organization, site, or region. Include that stable scope in the namespaced
+   ID. Never use the raw local ID by itself.
+3. **No authoritative foreign ID.** Build a deterministic, documented composite
+   only from stable source fields. If that composite is not guaranteed unique,
+   set `matchBehavior="no-id-match no-id-break"` and require at least one usable
+   MAC, IP, or hostname for correlation.
+4. **Identity unresolved.** Stop and ask for documentation or representative
+   samples. Do not generate an integration that guesses.
+
+Never use `new_uuid()` or another random value as an imported asset ID or as a
+fallback. A random fallback creates a new asset on every poll. If a record lacks
+the required identity and no safe deterministic fallback exists, skip it with a
+concise message that does not contain credentials or the full vendor object.
+
+### 3. Design the Mapping
+
+Write a short mapping plan before implementation:
+
+| Vendor object/field | runZero field | Conversion | Required |
+| --- | --- | --- | --- |
+| documented identity | `ImportAsset.id` | scoped/namespaced | yes |
+| documented names | `hostnames` | trim and deduplicate | no |
+| IP and MAC fields | `networkInterfaces` | `network_interface(...)` | no |
+| remaining useful fields | `customAttributes` | `to_custom_attributes(...)` | no |
+
+Decide whether child records should be attached as software, services, or
+vulnerabilities rather than emitted as separate assets. This cardinality check
+is part of duplicate prevention.
+
+### 4. Implement the Integration
+
+Create `<slug>/<slug>.star` and `<slug>/README.md`.
+
+The script must:
+
+- Put a literal `CONFIG = {...}` first, with `minVersion: "5.1.0"` or later.
+- Declare every kwarg and mark credential fields as `secret` without defaults.
+- Use `OPTIONS_HTTP` and `OPTIONS_TLS` plus `get_http_options` for HTTP APIs.
+- Use the runZero `http`, `kwargs`, `net`, parsing, and type helpers.
+- Use `validationMode: "compile"` only for direct-protocol integrations.
+- Handle all pages and stop on the documented pagination condition.
+- Stream each page with `report_assets(page_assets)` and return `None` for
+  inventory APIs that can return large datasets.
+- Skip malformed records that lack the approved identity instead of inventing
+  IDs.
+- Keep credentials out of logs, errors, tags, and custom attributes.
+- Respect documented rate limits and use bounded retry behavior.
+
+The README must document setup, required permissions/scopes, parameters,
+imported data, pagination, known limits, validation steps, documentation links,
+and the completed asset identity decision.
+
+### 5. Validate with Representative Data
+
+Validation must exercise more than compilation.
+
+1. Run `runzero script --filename <script> --validate` when the generic local
+   HTTP fixture matches the script response shape.
+2. If generic validation cannot model the vendor response, start a small local
+   HTTP fixture with representative redacted responses and run the real
+   `runzero script` command against it.
+3. Confirm at least one asset serializes successfully when the integration is
+   inbound.
+4. Run the same fixture twice and verify the emitted `ImportAsset.id` is
+   identical across runs.
+5. Include two source rows that represent the same asset when the vendor API can
+   return duplicates or child records; verify the mapping emits one intended
+   asset identity or documented enrichment records.
+6. Include two distinct assets with similar names or addresses; verify their IDs
+   remain distinct.
+7. Test missing identity, empty optional values, pagination, authentication
+   failure, rate limiting, malformed records, and duplicate source records.
+
+If a sibling platform checkout exists, run its compatibility test with this
+repository as `RZ_CUSTOM_INTEGRATION_SCRIPTS_DIR`. Do not modify the platform
+repository unless the user asked for platform changes.
+
+### 6. Regenerate and Review
+
+Run the repository catalog generator. Run it twice and confirm the second run
+does not change generated files. Review the complete diff for:
+
+- Random, temporary, event, session, or child-record IDs.
+- Missing account/tenant namespace on locally scoped IDs.
+- Secrets in logs or examples.
+- Incomplete pagination.
+- Unbounded response accumulation instead of `report_assets`.
+- Hand-written helpers that duplicate the provided Starlark modules.
+- Undocumented assumptions from unavailable vendor docs.
+
+Do not commit or push unless explicitly requested.
+
+## Completion Report
+
+Report:
+
+1. Documentation sources used.
+2. Objects and fields imported.
+3. The foreign-ID verdict, evidence, namespace, fallback behavior, and chosen
+   `matchBehavior`.
+4. Files created or updated.
+5. Validation commands and results.
+6. Any assumptions or vendor behavior that still requires confirmation.
+
+An integration is not complete if the asset identity verdict is absent or
+uncertain.

.github/skills/create-custom-integration/references/identity-decision.md

@@ -0,0 +1,113 @@
+# Asset Identity Decision
+
+Complete this decision before writing `ImportAsset` construction code. Add the
+final decision to the integration README.
+
+## Decision Record
+
+```markdown
+## Asset identity
+
+- Target entity: <physical device, VM, cloud resource, application, etc.>
+- Source ID field: `<documented.field.path>`
+- Documentation evidence: <URL and quoted/paraphrased contract>
+- Uniqueness scope: <global, tenant, account, organization, site, region>
+- Cardinality: <one source row per target asset, or explain many-to-one>
+- Stability: <what lifecycle events preserve or replace the ID>
+- Reuse behavior: <whether deleted IDs can be assigned to another asset>
+- Presence: <required, nullable, or missing for specific object types>
+- Final runZero ID: `<vendor>:<stable-scope>:<source-id>`
+- Missing-ID behavior: <skip, or documented deterministic composite>
+- Match behavior: <default or exact matchBehavior tokens>
+- Verdict: <authoritative, scoped authoritative, derived/non-authoritative,
+  unresolved>
+```
+
+## Required Evidence
+
+Use vendor documentation, an OpenAPI schema, or representative response samples
+to answer all of the following:
+
+1. Does the field identify the desired asset, or a child such as an event,
+   interface, agent installation, session, user association, or finding?
+2. Can one desired asset appear in multiple rows with different values?
+3. Can two desired assets share the value in different accounts, sites, or
+   regions?
+4. Does the value survive rename, reboot, IP/MAC change, agent upgrade, and
+   ordinary inventory refresh?
+5. Does reinstall, reprovision, clone, or restore generate a new value?
+6. Can the vendor recycle a deleted object's value?
+7. Is the field always present in list and detail responses?
+
+An API field named `id`, `uuid`, `device_id`, or `temporary_id` is not evidence
+by itself.
+
+## Decision Rules
+
+### Authoritative ID
+
+Use default ID matching when the source documents a stable, one-to-one ID. Add a
+vendor namespace and every scope required for uniqueness:
+
+```python
+asset_id = "vendor:{}:{}".format(account_id, source_id)
+```
+
+If network identifiers are expected to change independently of that ID:
+
+```python
+matchBehavior = "no-mac-break no-ip-break no-name-break"
+```
+
+This keeps the authoritative ID as the matching signal while preventing normal
+network churn from fragmenting one asset.
+
+### Derived, Non-Authoritative ID
+
+Use a deterministic composite only when its inputs are stable and documented.
+Namespace and delimit each component. Do not hash ambiguous concatenated values.
+
+If uniqueness is not guaranteed, the ID must not drive or block matching:
+
+```python
+matchBehavior = "no-id-match no-id-break"
+```
+
+In that case each record must still contain a usable MAC, IP, or hostname. If it
+does not, skip the record because runZero has no reliable correlation signal.
+
+### Missing Identity
+
+Never write:
+
+```python
+asset_id = item.get("id") or new_uuid()
+```
+
+Use the approved deterministic fallback or skip the malformed record:
+
+```python
+source_id = item.get("id")
+if not source_id:
+    print("vendor: skipping record with no documented asset id")
+    continue
+```
+
+Do not log the full source object because it may contain credentials, personal
+data, or large nested content.
+
+## Duplicate-Oriented Tests
+
+At minimum, test these cases with a local fixture:
+
+| Case | Expected result |
+| --- | --- |
+| Same source object in two polls | Same `ImportAsset.id` |
+| Same local ID in two tenants | Different namespaced IDs |
+| Two child rows for one asset | One asset identity or explicit enrichment |
+| Missing authoritative ID | Record skipped or approved deterministic fallback |
+| Same hostname, distinct source IDs | Distinct assets unless policy says otherwise |
+| Stable ID with changed IP/MAC/name | Same ID; match behavior permits intended merge |
+
+If any expected result cannot be justified, the verdict is unresolved and the
+integration should not be generated yet.