fix the conversion

[xiaotong743]

Pandas refuses to assign a Categorical with categories different from the target's

[selmanozleyen]

can you give me an example? ideally the datasets are preprocessed so that even if a categorical is not present in that dataset is should have under x_col.cats all the categories across datasets. See the unify.py file. This was we have common coding for each categories even if the files are separate. This gives us robustness because we would be tracking all the category space in all files.

[selmanozleyen]

Do you agree on this? Maybe I can make it clearer that I expect here in cateogoricals to have the same labeling metadata ie unified categoricals.

[xiaotong743]

To me this is an easy fix to make it more robust. It would be nice to make it explicit that one should unify the dataset in such a way. Maybe either have unify.py logic in also scfit instead of just cf-train and delete this logic here, or make it working now. Either way, I think we should fix this simply bc now it's not doing what it's intended to do.

import pandas as pd
from pandas.api.types import union_categoricals

fa = pd.DataFrame({"dataset": pd.Categorical(["ds_a"] * 3)})   # cats: ['ds_a']
fb = pd.DataFrame({"dataset": pd.Categorical(["ds_b"] * 3)})   # cats: ['ds_b']
union = union_categoricals([fa["dataset"], fb["dataset"]], ignore_order=True).categories
# union == ['ds_a', 'ds_b']

for f in (fa, fb):
    f.loc[:, "dataset"] = f["dataset"].cat.set_categories(union)
  # f["dataset"]        = f["dataset"].cat.set_categories(union) # fix

Output for pandas version >3
TypeError: Cannot set a Categorical with another, without identical categories