Skip to content

Bad SQL Alchemy Row conversion breaks Snowflake's system metric profiling #27641

@edg956

Description

@edg956

Affected module
Ingestion Framework

Describe the bug
When running the profiler for a Snowflake service and trying to compute system metrics, you get the following error: ValueError: dictionary update sequence element #0 has length 36; 2 is required

The culprit is in metadata.ingestion.source.database.snowflake.models.SnowflakeQueryLogEntry.get_for_table, where we try to pass a SQL Alchemy Row directly to a dict, which breaks the interface expected to build one.

class SnowflakeQueryLogEntry(BaseModel):
    ...

    @staticmethod
    def get_for_table(
        session: Session, tablename: str, service_connection_config: SnowflakeConnection
    ):
        ...
        return TypeAdapter(List[SnowflakeQueryLogEntry]).validate_python(
            [ExtendedDict(r).lower_case_keys() for r in rows]    # ValueError raised by ExtendedDict(r)
        )

The solution would be to turn the row to a dict beforehand, e.g: ExtendedDict(r._asdict()).

To Reproduce

  • Create a Snowflake Database service and let the metadata agent run
  • Create a or edit the profiler agent to include system metric
    Image
  • Deploy, run the agent and check logs

Expected behavior
It runs successfully without failing to obtain system metrics

Version:

  • OS: Reproduced in MacOS
  • Python version: 3.11
  • OpenMetadata version: 1.12.0 (when we upgraded SQLAlchemy 2.0)
  • OpenMetadata Ingestion package version: 1.12.0.0

Additional context

This was probably introduced in #26031 and missed in the CI because we've been skipping Snowflake's Python CLI e2e since #26411 because of credential issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    Status

    Done ✅

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions