Filters & Examples

Filters make managing Canvas Data easier by fixing or adjusting various issues and adding preferences to your workflow.

More plugins here https://plugins.embulk.org/#filter

embulk-filter-column

Add Columns that don't exist

filters:
  - type: column
    add_columns:
      - {name: pk, type: long, default: 1}

reference tables: assignment_override_user_rollup_fact

Drop Columns

filters:
  - type: column
    drop_columns:
      - {name: type}

reference tables: course_dim, course_section_dim, enrollment_dim, user_dim

embulk-filter-row

Allows you to ignore and filter rows with SQL like syntax. Great for ignoring rows from last school year

filters:
  - type: row
    where: workflow_state != 'unsubmitted'

LIKE and NOT LIKE

where: course_name REGEXP 'Homeroom'
where: course_name REGEXP '^((?!Homeroom).)*$'

embulk-filter-unique

Useful anytime import has issues with duplicate rows. Adding this to each config will slow down import.

filters:
  - type: unique
    columns:
      - id