Dataset Module

[kshen3778]

1. Splitting data (train/val/test, k-fold): could be done via column in csv
   What would be the best way for the WorkbenchDataset class to switch between different splits. We could have a set_split(splitname) function that can be called:

dataset = WorkbenchDataset(...)
dataset.set_split("train")
# Train Model here (get_item will only return from train portion)
dataset.set_split("test")
# Test model here (get_item will only return from test portion)

Or alternatively, define the splits as an individual Dataset class itself that can be returned

dataset = WorkbenchDataset(...)
train_loader = Dataloader(dataset.get_split("train"),......)
# Train Model here
test_loader = Dataloader(dataset.get_split("test"),......)
# Test model here

Note: This means WorktableDataset won't be a subclass of Pytorch's Dataset since get_item will no longer be used. dataset.get_split will return an instance of a Pytorch Dataset class that will be iterable.

Other
2. getting/querying subset: could be part of create_new_version (not sure if it makes sense to make a separate get_subset function)
3. EDA tools & visualizations (?)

[jmarsil]

What do you think is a better option?

[skim2257]

def __init__(self, split):
    self.split_idxs = {
        'train': [1, 2, 3, 4, 5, 6, 7],
        'val': [8],
        'test': [9, 0]
    }
    self.split = Train

def set_split(self, 
              split: str = "train"):
    self.split = split

def __getitem__(self, idx):
    df_idx = split_idx[self.split][idx]
    subject_id = self.clinical_data.iloc[df_idx].name
        
    path = os.path.join(self.root_directory, "{}", "{}_{}.nrrd")
    image = sitk.ReadImage(path.format('images', subject_id, 'image'))
    mask = sitk.ReadImage(path.format('masks', subject_id, 'mask'))

[kshen3778]

def __init__(self, split):
    self.split_idxs = {
        'train': [1, 2, 3, 4, 5, 6, 7],
        'val': [8],
        'test': [9, 0]
    }
    self.split = Train

def set_split(self, 
              split: str = "train"):
    self.split = split

def __getitem__(self, idx):
    df_idx = split_idx[self.split][idx]
    subject_id = self.clinical_data.iloc[df_idx].name
        
    path = os.path.join(self.root_directory, "{}", "{}_{}.nrrd")
    image = sitk.ReadImage(path.format('images', subject_id, 'image'))
    mask = sitk.ReadImage(path.format('masks', subject_id, 'mask'))

What happens if you do this:

worktable_dataset.set_split("train")
train_loader = DataLoader(worktable_dataset)
worktable_dataset.set_split("val")
val_loader = DataLoader(worktable_dataset)
# would train_loader still be using the train split ?

[skim2257]

train_loader = worktable_dataset.dataloader(version='original', split='train')

or

train_loader = worktable_dataset.get_dataloader(version='original', split='train')

or

train_loader = DataLoader(worktable_dataset.get(version='original', split='train'))

not sure how this last one would work as i dunno what happens in Dataset.call

[skim2257]

side note: for splitting train/val/test
return a split as:
torch.utils.data.Subset(dataset, indices)