net/mlx5: Don't zero user_count when destroying FDB tables

esw->user_count tracks how many TC rules are added on an esw via
mlx5e_configure_flower -> mlx5_esw_get -> atomic64_inc(&esw->user_count)

esw.user_count was unconditionally set to 0 in
esw_destroy_legacy_fdb_table and esw_destroy_offloads_fdb_tables.

These two together can lead to the following sequence of events:
1. echo 1 > /sys/class/net/eth2/device/sriov_numvfs
  - mlx5_core_sriov_configure -...-> esw_create_legacy_table ->
    atomic64_set(&esw->user_count, 0)
2. tc qdisc add dev eth2 ingress && \
   tc filter replace dev eth2 pref 1 protocol ip chain 0 ingress \
       handle 1 flower action ct nat zone 64000 pipe
  - mlx5e_configure_flower -> mlx5_esw_get ->
    atomic64_inc(&esw->user_count)
3. echo 0 > /sys/class/net/eth2/device/sriov_numvfs
  - mlx5_core_sriov_configure -..-> esw_destroy_legacy_fdb_table
    -> atomic64_set(&esw->user_count, 0)
4. devlink dev eswitch set pci/0000:08:00.0 mode switchdev
  - mlx5_devlink_eswitch_mode_set -> mlx5_esw_try_lock ->
    atomic64_read(&esw->user_count) == 0
  - then proceed to a WARN_ON in:
  esw_offloads_start -> mlx5_eswitch_enable_locke -> esw_offloads_enable
  -> mlx5_esw_offloads_rep_load -> mlx5e_vport_rep_load ->
  mlx5e_netdev_change_profile -> mlx5e_detach_netdev ->
  mlx5e_cleanup_nic_rx -> mlx5e_tc_nic_cleanup ->
  mlx5e_mod_hdr_tbl_destroy

Fix this by not clearing out the user_count when destroying FDB tables,
so that the check in mlx5_esw_try_lock can prevent the mode change when
there are TC rules configured, as originally intended.

Fixes: 2318b8b ("net/mlx5: E-switch, Destroy legacy fdb table when needed")
Signed-off-by: Cosmin Ratiu <[email protected]>
Reviewed-by: Dragos Tatulea <[email protected]>
Signed-off-by: Tariq Toukan <[email protected]>
Reviewed-by: Simon Horman <[email protected]>
Link: https://patch.msgid.link/[email protected]
Signed-off-by: Jakub Kicinski <[email protected]>

Author: Cosmin Ratiu

drivers/net/ethernet/mellanox/mlx5/core/esw/legacy.c

@@ -66,7 +66,6 @@ static void esw_destroy_legacy_fdb_table(struct mlx5_eswitch *esw)
 	esw->fdb_table.legacy.addr_grp = NULL;
 	esw->fdb_table.legacy.allmulti_grp = NULL;
 	esw->fdb_table.legacy.promisc_grp = NULL;
-	atomic64_set(&esw->user_count, 0);
 }
 
 static int esw_create_legacy_fdb_table(struct mlx5_eswitch *esw)

drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c

@@ -1978,7 +1978,6 @@ static void esw_destroy_offloads_fdb_tables(struct mlx5_eswitch *esw)
 	/* Holds true only as long as DMFS is the default */
 	mlx5_flow_namespace_set_mode(esw->fdb_table.offloads.ns,
 				     MLX5_FLOW_STEERING_MODE_DMFS);
-	atomic64_set(&esw->user_count, 0);
 }
 
 static int esw_get_nr_ft_offloads_steering_src_ports(struct mlx5_eswitch *esw)