|
| 1 | +# badata |
| 2 | + |
| 3 | +Data package: BA (Bundesagentur für Arbeit) regional job market statistics by |
| 4 | +district (Kreis) and occupational group (KldB 2010). Pure data package, no |
| 5 | +exported functions (`NAMESPACE` only lists the roxygen boilerplate) — everything |
| 6 | +happens once in `data-raw/*.R` and ships as `data/*.rda`. |
| 7 | + |
| 8 | +## Where the raw data comes from |
| 9 | + |
| 10 | +Not an open download. Each delivery is a paid, bespoke "Sonderauswertung" from |
| 11 | +BA-Statistik-Service-West ( [email protected]), tied to a |
| 12 | +specific order number (e.g. `317-408036`) referenced in both the delivered |
| 13 | +filenames and the email subject line. Requesting an update means emailing that |
| 14 | +address, referencing the previous order number so they replicate the same |
| 15 | +Kreis × KldB-2010 × year breakdown, not finding a public portal download. |
| 16 | + |
| 17 | +Two BA products per delivery: |
| 18 | +- `*_ALO_OST_B_*` / `*-ALO-OST-B-*` → unemployed (`Arbeitslose`) + jobs |
| 19 | + (`Arbeitsstellen`) → feeds `unemployed_total`, `unemployed_foreigners`, `jobs` |
| 20 | +- `*_SvB_GeB*` / `*-SvB-GeB-*` → employees subject to social insurance (SvB) + |
| 21 | + marginally employed (GeB) → feeds `employees_by_workplace` (AO/Arbeitsort), |
| 22 | + `employees_by_residence` (WO/Wohnort) |
| 23 | + |
| 24 | +Source `.xlsx` files are gitignored (`data-raw/*.xlsx`) and never committed — |
| 25 | +only the parsed `.rda` output is. |
| 26 | + |
| 27 | +## Standing gotcha: BA changes the export format between deliveries without notice |
| 28 | + |
| 29 | +**The trap (2026-07-30 update, order 334605 → 408036).** Rerunning the existing |
| 30 | +`data-raw/*.R` scripts unmodified against a new BA delivery produced silently |
| 31 | +wrong data in three different ways, not an error. BA's export format is not a |
| 32 | +stable contract across delivery cycles. Before trusting a rerun against new |
| 33 | +files, diff the raw header structure (row offsets, exact label text, sheet |
| 34 | +count/names) against what the current scripts assume — do not assume "same BA |
| 35 | +product name" means "same layout." |
| 36 | + |
| 37 | +Concretely hit this round: |
| 38 | +- **Row offset shift.** The `unemployed`/`jobs` file's header block moved down |
| 39 | + by exactly one row between deliveries (an extra line inserted somewhere in |
| 40 | + the preamble). Silent until you check exact `skip=`/row content — no error, |
| 41 | + just wrong columns read as headers. |
| 42 | +- **Abbreviated → full label text.** `"Sv-pflichtig Beschäftigte"` → |
| 43 | + `"Sozialversicherungspflichtig Beschäftigte"`, `"Tätigkeit nach KldB 2010"` → |
| 44 | + `"Ausgeübte Tätigkeit nach KldB 2010"`. Any `case_when(x == "old label", ...)` |
| 45 | + match silently falls through to `TRUE ~ x` instead of erroring, so this |
| 46 | + corrupts data quietly rather than crashing. |
| 47 | +- **Header cells that used to be blank now contain literal filler text.** The |
| 48 | + new employees file writes the literal word `"darunter"` into every |
| 49 | + foreigners/women sub-column instead of leaving it blank, which broke |
| 50 | + `fill()`-based label propagation and collided both employment types' |
| 51 | + foreigners/women columns into identical names. The new unemployed/jobs file |
| 52 | + does the opposite: the "total" sub-column's header is blank where the old |
| 53 | + file wrote `"Insgesamt"` literally, so `x == "Insgesamt"` stopped matching |
| 54 | + and the group was dropped instead of set to `"total"`. |
| 55 | +- **File bundling changed.** Employees data moved from one file with two |
| 56 | + sheets (workplace/residence) to two separate files, one sheet each. |
| 57 | + |
| 58 | +**Standing instruction.** When new BA files land in `data-raw/`, before |
| 59 | +re-running anything: `readxl::excel_sheets()` each file, dump the raw header |
| 60 | +rows (`col_names = FALSE`, no skip) for each relevant sheet, and diff that |
| 61 | +against the current script's assumed `skip=`/label strings. Treat every |
| 62 | +`case_when`/exact-string match in the parsing code as a thing that can silently |
| 63 | +break, not just the row offsets. |
| 64 | + |
| 65 | +## Known pre-existing data quirks (not bugs, don't "fix" without a deliberate decision) |
| 66 | + |
| 67 | +- `occupational_group_codes` / `region_codes` won't perfectly 1:1 match every |
| 68 | + BA product's occupational_group/region values — some categories are |
| 69 | + catch-alls with no numeric KldB code (e.g. `"ohne Angabe zum Zielberuf"`). |
| 70 | + As of 0.2.0 these get `code = NA` with the real name preserved (so |
| 71 | + `left_join()` on `code`/`occupational_group` works via NA-matching), rather |
| 72 | + than the code accidentally duplicating into the name column. |
| 73 | +- Historical years get **retroactively restated** to current administrative |
| 74 | + boundaries whenever BA/BKG changes Kreis boundaries (their "Gebietsstand" |
| 75 | + convention). A region code's historical value can *change* between package |
| 76 | + versions with no error — e.g. Hanau split off from Main-Kinzig-Kreis as its |
| 77 | + own `Kreisfreie Stadt` (06415) in the 0.2.0 delivery, retroactively applied |
| 78 | + back to 2016. This is BA doing the right thing (comparable boundaries across |
| 79 | + years), not a defect, but it does mean `06435`'s 2016 value differs between |
| 80 | + badata 0.1.x and 0.2.0. Check `NEWS.md` before assuming a rerun should |
| 81 | + reproduce old values exactly. |
| 82 | +- BA's own disclaimer ("Datenrevisionen können zu Abweichungen ... führen") |
| 83 | + means a small fraction (seen: ~0.3-1.5%) of overlapping-year values will |
| 84 | + legitimately differ release to release. Don't chase 100% reproduction of |
| 85 | + old values as a correctness bar; anti-join on keys (not a naive full outer |
| 86 | + join on value columns) to separate "row genuinely absent" from "value |
| 87 | + merely changed." |
| 88 | + |
| 89 | +## Mapping / geodata |
| 90 | + |
| 91 | +No shapefile ships with the package (it's Kreis-code tabular data only). To |
| 92 | +map, join `region_codes$code` (or any table's `region`/`occupational_group` |
| 93 | +column) against BKG's official VG250 Kreis boundaries |
| 94 | +(`daten.gdz.bkg.bund.de/produkte/vg/vg250_ebenen_0101/aktuell/...`, `VG250_KRS` |
| 95 | +layer, join key `AGS`) — same AGS/Kreis-code system BA uses, so it joins |
| 96 | +cleanly without name-matching. GADM's German boundaries (used elsewhere in |
| 97 | +this workspace, e.g. `RegioPress/data/geo/gadm/`) use `NAME_2`, not AGS codes, |
| 98 | +and are not guaranteed current on recent Kreis boundary changes (Hanau) — |
| 99 | +prefer VG250 for this package specifically. |
| 100 | + |
| 101 | +## CI: "Update CITATION.cff" workflow is broken (org policy, not the workflow itself) |
| 102 | + |
| 103 | +`.github/workflows/update-citation-cff.yaml` runs `cff_write()` correctly but |
| 104 | +its `git push` step fails with a 403 (`Permission ... denied to |
| 105 | +github-actions[bot]`), silently swallowed by the workflow's own |
| 106 | +`git push || echo "No changes to commit"`, so the job still reports success. |
| 107 | +Root cause: RegioHub org-level Actions policy has "Workflow permissions" set |
| 108 | +to read-only, overriding any repo-level setting. Fixing it needs an org owner |
| 109 | +to change it at `github.com/organizations/RegioHub/settings/actions` (or via |
| 110 | +`gh api -X PUT orgs/RegioHub/actions/permissions/workflow`, which needs the |
| 111 | +`admin:org` OAuth scope — `gh auth refresh -h github.com -s admin:org` and |
| 112 | +complete the device-code browser flow). Until that's fixed, regenerate |
| 113 | +`CITATION.cff` manually after any `DESCRIPTION`/`inst/CITATION` change: |
| 114 | +`conda run -n rstats Rscript -e 'library(cffr); cff_write(keys = list())'`, |
| 115 | +then commit it as a normal commit (that push works fine, it's only the bot's |
| 116 | +default token that's restricted). |
| 117 | + |
| 118 | +## Environment |
| 119 | + |
| 120 | +R env: conda env `rstats` (`/home/researcher/miniconda3/envs/rstats`), not |
| 121 | +`base`. Needs `pandoc` on `PATH` for `devtools::build_readme()` — invoke via |
| 122 | +`conda run -n rstats Rscript ...`, not `rstats/bin/Rscript` directly by full |
| 123 | +path (that skips env activation, so `PATH` won't include the env's `pandoc`). |
0 commit comments