Skip to content

Add MIOpen (AMDGPU) fast path for batchnorm #752

Description

@CarloLucibello

Following the addition of the functional normalization operators (batchnorm, instancenorm, groupnorm, layernorm) in NNlib v0.9.41, the cuDNN-accelerated batchnorm lives in NNlibCUDACUDNNExt and is selected automatically for CuArrays. There is no equivalent MIOpen fast path for AMDGPU ROCArrays — they fall back to the generic implementation.

Flux is migrating its normalization layers to wrap NNlib.batchnorm (FluxML/Flux.jl#2700) and, as part of that, is removing its own AMDGPU MIOpen BatchNorm specialization (previously in ext/FluxAMDGPUExt/batchnorm.jl). To keep the accelerated AMD path available, it would be good to move that logic into NNlib as an AMDGPU/MIOpen extension, mirroring NNlibCUDACUDNNExt.

The code removed from Flux, as a starting point:

function _amdgpu_batchnorm(x, γ, β; μ, σ², ϵ, within_grad::Bool)
    if within_grad
        return AMDGPU.MIOpen.batchnorm_training(x, γ, β, μ, σ²; ϵ=Float64(ϵ), iteration=0) # TODO iteration
    else
        return AMDGPU.MIOpen.batchnorm_inference(x, γ, β, μ, σ²; ϵ=Float64(ϵ))
    end
end

function ChainRulesCore.rrule(::typeof(_amdgpu_batchnorm), x, γ, β; μ, σ², ϵ, within_grad::Bool)
    y, μ_saved, ν_saved = _amdgpu_batchnorm(x, γ, β; μ, σ², ϵ, within_grad)
    function _batchnorm_pullback(Δ)
        dx, dγ, dβ = AMDGPU.MIOpen.∇batchnorm(unthunk(Δ), x, γ, β, μ_saved, ν_saved)
        (NoTangent(), dx, dγ, dβ)
    end
    y, _batchnorm_pullback
end

Ideally the NNlib method would implement the full batchnorm(g, b, x, running_mean, running_var, momentum; eps, training, track_stats) signature (matching the generic and cuDNN methods) and its ∇batchnorm, honoring training / track_stats / running-statistic updates rather than relying solely on within_gradient as the old Flux code did. It should also cover the 2D/4D/5D input shapes and address the iteration TODO.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions