Treat webhook 422 responses as proper failures (#6426)

HookRelay returns 422 when it rejects a webhook delivery request, for
example when its SSRF protection blocks the target URL. Previously, the
`discard_on` handler only called `increment! :failure_count`, which
doesn't contribute to the disable threshold. This meant webhooks with
blocked URLs would accumulate failures indefinitely without ever being
disabled.

Call `failure!` instead of `increment! :failure_count` in the
`discard_on` handler. `failure!` tracks `failures_since_last_success`,
which will automatically disable the webhook after exceeding the
failure threshold.

422s were previously given special treatment under the assumption they
might not represent a real delivery failure. But there's no scenario
where a 422 from HookRelay should not count as a failure -- the request
was understood and rejected.

Author: flavorjones

app/jobs/notify_web_hook_job.rb

@@ -33,7 +33,7 @@ class NotifyWebHookJob < ApplicationJob
   # has to come after the retry on
   discard_on(Faraday::UnprocessableEntityError) do |j, e|
     logger.info({ webhook_id: j.webhook.id, url: j.webhook.url, response: JSON.parse(e.response_body) })
-    j.webhook.increment! :failure_count
+    j.webhook.failure!(completed_at: Time.current)
   end
 
   def perform(*)

test/jobs/notify_web_hook_job_test.rb

@@ -85,5 +85,34 @@ class NotifyWebHookJobTest < ActiveJob::TestCase
       assert_performed_jobs 1, only: NotifyWebHookJob
       assert_enqueued_jobs 0, only: NotifyWebHookJob
     end
+
+    should "record a failure on a 422" do
+      stub_request(:post, "https://api.hookrelay.dev/hooks///webhook_id-#{@hook.id}")
+        .to_return_json(status: 422, body: { error: "Invalid url" })
+
+      perform_enqueued_jobs do
+        @job.enqueue
+      end
+
+      @hook.reload
+
+      assert_equal 1, @hook.failures_since_last_success
+    end
+
+    should "disable the webhook after repeated 422s exceed the failure threshold" do
+      @hook.update!(
+        failures_since_last_success: WebHook::FAILURE_DISABLE_THRESHOLD - 1,
+        created_at: (WebHook::FAILURE_DISABLE_DURATION + 1.minute).ago
+      )
+
+      stub_request(:post, "https://api.hookrelay.dev/hooks///webhook_id-#{@hook.id}")
+        .to_return_json(status: 422, body: { error: "Invalid url" })
+
+      perform_enqueued_jobs do
+        @job.enqueue
+      end
+
+      refute_predicate @hook.reload, :enabled?
+    end
   end
 end