Several fixes and additions - #153
Conversation
Fix arrays
Fixing decimal
skip activate_version if metadata not added
fix decimals
# Conflicts: # target_clickhouse/connectors.py # tests/test_core.py
Merge origin
|
Hello! Have you been abel to take a look at the changes or want a bit more details on them? :) |
|
Hi Jesus Manuel Gallego (@gallejesus), apologies I've been on leave for the past few weeks and I miss some of these notifications. Can review this now. |
Ben Theunissen (BTheunissen)
left a comment
Ben Theunissen (BTheunissen)
left a comment
There was a problem hiding this comment.
The changes look great, thanks for adding the comprehensive type tests. Once the date type is mapped to Date32 this should be good to merge.
| return nullabilizer(jsonschema_type, t.cast(sqlalchemy.types.TypeEngine, sqlalchemy.types.DATETIME())) | ||
| if datelike_type in "time": | ||
| return nullabilizer(jsonschema_type, t.cast(sqlalchemy.types.TypeEngine, sqlalchemy.types.TIME())) | ||
| if datelike_type == "date": |
There was a problem hiding this comment.
Can you change this to use the newly added Date32 Clickhouse SQLAlchemy type I added, as the Clickhouse Date type doesn't support date values before epoch I switched to using that type to be able to ingest dates in the past.
There was a problem hiding this comment.
Hi, great! Let me take a look at it and change it :)
There was a problem hiding this comment.
Hi Ben Theunissen (@BTheunissen) , I had time now to check it and I think the last commit fixes it, using the Date32 added in clickhouse-sqlalchemy v0.3.2. Can you check it? Thanks!
| @@ -107,13 +153,6 @@ def to_sql_type(self, jsonschema_type: dict) -> sqlalchemy.types.TypeEngine: | |||
| sqlalchemy.types.TypeEngine, | |||
There was a problem hiding this comment.
I have kept the Date32 here, as I believe is the place where the datatype to ch is created, did you mean this? (I have removed the nulls over the date and time types as the nulls are managed in the nullabilizer function)
3 participants
We have created this PR to fix some issues we have found when workig with ClickHouse target.
We can discuss them more in depth but main changes include the addition of tests covering more data types, explicit override of sql data type conversion and manage of nulls and keeping original name from source.
Let me know what you think of those :)
Thank you!