Hard-load of Casbin policy on permission validation

[BryanttV]

Description

We are currently performing a hard load of the policy when validating the permissions a user has in the permissions/validate/me endpoint. This is done to ensure that the most up-to-date policy is used at the time of validation.

openedx-authz/openedx_authz/rest_api/v1/views.py

Lines 96 to 131 in e374fbf

	@apidocs.schema(
	body=PermissionValidationSerializer(help_text="The permissions to validate", many=True),
	responses={
	status.HTTP_200_OK: PermissionValidationResponseSerializer,
	status.HTTP_400_BAD_REQUEST: "The request data is invalid",
	status.HTTP_401_UNAUTHORIZED: "The user is not authenticated",
	},
	)
	def post(self, request: HttpRequest) -> Response:
	"""Validate one or more permissions for the authenticated user."""
	AuthzEnforcer.get_enforcer().load_policy()
	serializer = PermissionValidationSerializer(data=request.data, many=True)
	serializer.is_valid(raise_exception=True)
	data = serializer.validated_data
	username = request.user.username
	response_data = []
	for permission in data:
	try:
	action = permission["action"]
	scope = permission["scope"]
	allowed = api.is_user_allowed(username, action, scope)
	response_data.append({"action": action, "scope": scope, "allowed": allowed})
	except ValueError as e:
	logger.error(f"Error validating permission for user {username}: {e}")
	return Response(data={"message": "Invalid scope format"}, status=status.HTTP_400_BAD_REQUEST)
	except Exception as e: # pylint: disable=broad-exception-caught
	logger.error(f"Error validating permission for user {username}: {e}")
	return Response(
	data={"message": "An error occurred while validating permissions"},
	status=status.HTTP_500_INTERNAL_SERVER_ERROR,
	)
	serializer = PermissionValidationResponseSerializer(response_data, many=True)
	return Response(serializer.data, status=status.HTTP_200_OK)

This change was made because, when relying only on the SyncEnforcer, which reloads the policy every 5 seconds, an odd behavior has been observed where the policy seems to be completely cleared from memory periodically. As a result, when permissions are validated, the system incorrectly returns that the user does not have them, even though they do exist in the database.

The main issue with performing a hard load is that the entire policy is reloaded every time permission validation is needed. This is not a sustainable solution due to the large number of policies that may exist in the database, which would significantly affect the performance of an endpoint that is widely used across the platform.

Impact

• Performance degradation: Continuously performing a hard load of the entire policy on every permission validation causes high database and memory usage, increasing response times for the permissions/validate/me endpoint.
• Scalability issues: As the number of policies grows, the cost of each validation request increases linearly, making this approach unsustainable for production environments with large datasets.
• Inconsistent authorization results: The unexpected behavior of the SyncEnforcer (clearing policies from memory) may cause users to temporarily lose permissions that actually exist in the database.

Possible Approaches

1. Investigate the SyncEnforcer behavior: Review how the SyncEnforcer instance is managed in memory. Validate whether it is being reinitialized or losing its internal policy cache due to concurrency or lifecycle issues.
2. Implement selective or partial policy reloads: Instead of a full hard load, reload only the policies related to the specific user, role, or scope being validated. This approach would significantly reduce the I/O and memory overhead of loading the full policy on every request.

[mariajgrimaldi]

Thanks for opening this. I’ve been thinking about this too, especially the hard reload of policies.

Some time ago, I opened #86 that added a decorator to load and unload policies on each API call to avoid using an outdated enforcer. Later, we moved to the SyncEnforcer, which reloads every few seconds. That solved some cases but also introduced new ones.

The main issue is a timing gap: if I create a library and immediately go to Manage Access, I sometimes get blocked because the policy has not synced yet. Waiting a few seconds fixes it, but it is confusing from a user’s perspective. We should evaluate how impactful this is from a UX point of view.

This is why we now force a hard reload before validation. It fixes the inconsistency but is not scalable or performant in the long run.

I have been considering revisiting the subset-loading approach I worked on before, loading only the policies relevant to the current request during the Django request lifecycle. That would reduce memory usage and keep data fresher without a full reload.

The risk is ending up with two behaviors:

• API endpoints using the per-request subset loader.
• The Python API still relying on the SyncEnforcer.

That could cause inconsistencies and be harder to maintain.

I also noticed an edge case that might be related to these behaviors: Manage Access sometimes loads an empty user list right after creation but works if you wait. This is likely because validate/me forces a reload while GET roles does not, so one uses fresh data and the other stale data.

So the current setup works but feels inconsistent. The hard reload ensures accuracy in some places but is too heavy. The sync enforcer is scalable but sometimes stale. Maybe a hybrid approach would work, using subset loading selectively (for example, right after create or update actions) while keeping the sync enforcer for normal checks, but both paths (REST and Python API) need to stay aligned.

Before diving into fixes, we should probably prioritize which of these issues has the biggest impact on the MVP user experience.

[mariajgrimaldi]

Any thoughts on this, @bmtcril @rodmgwgu @MaferMazu? I’ll keep testing and thinking about it to see if I can come up with a more concrete next step.

[bmtcril]

I've been thinking about this as well. SyncEnforcer was probably just the best of the few bad options we had, and I wanted to revisit it around now as well. My rough thoughts on requirements for this are:

• Should cache recently used policies
• Should not need to keep every policy in memory (it will grow too big on large instances)
• Should invalidate the cache on relevant changes
• Role management / auditing API actions only use the cache when looking at specific users, otherwise pull only the relevant data directly from the database (acceptable since they are comparitively very rare?)

To me this sounds like a pull-through cache of all policies related to a user, which can be invalidated when a user's roles change. I know it's a lot more complicated when taking into account things like changes to role definitions that happen during an upgrade (assuming we don't let people change role definitions on the fly?) but we could either have a mechanism to clear the whole cache during upgrades or a short enough timeout that it won't matter much.

I'm not sure that covers all of the cases, curious to hear what everyone else thinks.

[rodmgwgu]

• Should invalidate the cache on relevant changes

I agree with your points, but for the short term, I think we could focus on the cache invalidation, even if we are still loading the whole thing into memory.

Maybe we could keep the SyncEnforcer, but add a way for signaling a cache invalidation, so next check forces a reload. (haven't looked into code details yet so not sure if there are any blockers for this).

[rodmgwgu]

I've been doing some research on this, here are my findings so far:

There are 2 different levels where "Caching" may apply with this Casbin implementation:

1. Loading and re-loading the policies (As done by SyncedEnforcer periodically)
2. Caching the enforcer decisions, as CachedEnforcer does.

I think we could take advantage of (2), but we would need to implement something a little different for (1).

I still don't fully understand how the memory model of Casbin works when combined with Django, but taking into account that we will have more than one process running (cms, lms, multiples of those), then we will be running one instance of Casbin on each process.

I'm thinking this may be related to the behavior we are seeing where the policy seems to be "lost".

I'll keep looking into this, my high level idea would be:

• We'll avoid reloading the policies on normal use, but need to have a mechanism to make sure the policies were already loaded at least once
• We'll need to have a way of signaling cache invalidation across processes, perhaps via celery?
• On invalidation, we would reload all policies, and if we enable the CachedEnforcer, we would also call the invalidation function on that one.

[mariajgrimaldi]

Thanks for all the comments and ideas so far. I think where we stand is that we have three main problems to solve, and caching could help with part of them, but not all:

Repeated DB hits on reads.

Every time we load the enforcer, we pull the full policy from the database into memory. This ensures fresh reads but results in frequent and unnecessary DB hits, especially with the SyncEnforcer reloading periodically.

I think this can be addressed with the caching approach we’ve been discussing. We could cache responses like validate/me or even policy slices, and invalidate them when something changes. That would immediately reduce database pressure while keeping reads reasonably up to date.

Stale reads right after writes.

When we assign a new role (or perform a similar write), the change is stored in the casbin_rule table, but the in-memory enforcer doesn’t reflect it until the next reload. Since the enforcer is just a memory structure with all role definitions, permissions, and assignments as strings, operations right after a write may still use stale data.

I don't think caching alone will fix this, but we can reuse the same invalidation mechanism: whenever something changes for a specific scope, we clear the cache and reload the scoped enforcer. I’ve already explored this idea a bit before, using a decorator that loads filtered policies by scope only when needed. This could give us fresher reads after writes without forcing a global reload each time.

Heavy full-policy loads.

Loading the entire policy into memory doesn’t scale well, even if we cache results, because we still end up loading everything at once. Casbin recommends filtered policy loading for large setups, and I think that’s what we need here. It would let us load only the relevant subset of policies into memory and combine nicely with cache invalidation. I’d like to keep exploring this direction, since it could address both memory and performance issues.

Overall, I think combining cache + invalidation + filtered policy loading (possibly via a decorator) would help reduce DB hits, avoid unnecessary full loads, and improve consistency after writes without needing a hard reload each time.

[mariajgrimaldi]

One more point on the role of the SyncEnforcer with the approach we are discussing.

If we move forward with cache + invalidation + filtered policy loading (via a decorator on strict freshness paths), I am not sure how useful the periodic reload from the SyncEnforcer remains. We might not need timed reloads if we depend on scoped reloads only where needed.

That said, I am still concerned about concurrency. My understanding is that the SyncEnforcer already manages concurrent access to the enforcer state. If we drop the “sync” behavior, we may lose that safety and need to handle it ourselves. Another option is to keep the SyncEnforcer instance for concurrency control, but disable or bypass its periodic reloads, and rely on:

• cache invalidation on writes,
• scoped filtered loads for read-after-write paths,
• and normal reads using the shared enforcer without a global reload.

I do not know the performance impact of keeping a single shared enforcer while doing frequent scoped loads, so I think this is something we should test. We could compare:

1. SyncEnforcer with periodic reloads,
2. SyncEnforcer without periodic reloads plus decorator-based filtered loads,
3. Per-request scoped enforcer for strict paths only.