Configuration Options

This page will attempt to provide more context for managing and detailing configuration options.

CSV Parser

These are the options I found passed all data into the test environments without failure.

  parser:
    charset: UTF-8
    newline: LF
    type: csv
    delimiter: "\t"
    quote: null
    quotes_in_quoted_fields: ACCEPT_STRAY_QUOTES_ASSUMING_NO_DELIMITERS_IN_FIELDS
    escape: '"'
    null_string: \N
    trim_if_not_quoted: false
    skip_header_lines: 0
    allow_extra_columns: false
    allow_optional_columns: false

Deprecated fields

Canvas Data releases new versions with fields marked as deprecated.

[Deprecated] No longer used in Canvas, will always be NULL.

Since this data will always be null, the config files can completely remove them.

This may cause breaking issues for users who currently import these fields, but the config files remove, and deprecate these fields using the column filter plugin.

Deprecating fields is easy with filter and comments

filters:
  - type: column
    drop_columns:
      - {name: process_attempts}
      - {name: has_admin_comment}

column_options:
    ...
    # deprecated # process_attempts: {type: 'INT NULL'}
    ...
    # deprecated # has_admin_comment: {type: 'NVARCHAR(256) NULL'}
    ...