coreai-opt v0.2.1

Added

• Support palettization of ConvTranspose1d/ConvTranspose2d/ConvTranspose3d layers via KMeansPalettizer
• Support for EAGER execution mode in model inspection utility

Fixed

• Fixed pruning mask dtype to match that of the weight being pruned
• Fixes to allow better support for bfloat16 dtype in palettization and quantization

coreai-opt v0.2.0

• coreai-opt is a library for applying model optimizations such as quantization, palettization (code book compression), pruning to PyTorch models.

• Key APIs / features include:

  • compressor classes and their corresponding config classes:
    • coreai_opt.quantization.Quantizer : class for applying quantization to a PyTorch model. Supports weight-only quantization and activation quantization, via calibration and training modes, for Integer and FP8/FP4 dtypes. coreai_opt.quantization.QuantizerConfig : class to control the algorithm hyperparameters and which parts of the model get quantized.
    • coreai_opt.palettization.KMeansPalettizer : class for applying palettization to model weights (i.e. look up table or codebook based compression ). Supports kmeans + sensitive kmeans based Palettization. Configuration done via coreai_opt.palettization.KMeansPalettizerConfig
    • coreai_opt.pruning.MagnitudePruner : for pruning weights of the model via fine tuning. Configuration done by coreai_opt.pruning.MagnitudePrunerConfig
  • After compressing PyTorch models, finalize API updates the model to make it ready to conversion via coreai-torch to deploy using the Core AI framework
  • coreai_opt.casting.cast_fp32_to_fp16() : autocast a torch exported program from FP32 precision to FP16 (weight and activations) for faster inference performance
  • coreai_opt.inspection.* : a few utilities to inspect a PyTorch model, to gather info about op types and names, modules types and names, which can be then utilized to create targeted compression configs to be used with the various available compressor classes
  • coreai_opt.coreai_utils.* : a few methods to apply a graph pass to a given AIProgram to compress weights. While compressing a PyTorch model is the recommended path, this maybe useful for testing and debugging.

• Link to documentation : https://apple.github.io/corai-optimization

• Known issues:

  • Tying model weights (e.g. layer1.weight = layer2.weight) after quantizer finalize in eager execution mode will fail
  • For models with shared weights, in Eager mode, MODULE_NAME > MODULE_TYPE precedence is only honored if you express the override via module_state_spec. With op_state_spec alone, it depends on forward-pass execution order.
  • Model Inspector utility only supports graph mode and compressor={None, Quantizer}
  • For models with shared weights using different local names (last part of the name after the rightmost “.”), in graph mode quantization, only one particular local name is matched. To know which name, users must examine the torch exported graph or view ModelInspector summary for modules sharing the weights. Alternatively, users can configure the same weight spec for each distinct local name to be safe.