Skip to content

Jitter throttling client retries so clients do not retry in lockstep #4207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
1991santhu wants to merge 1 commit into
base: master
Choose a base branch
from
+17 −1
Open

Jitter throttling client retries so clients do not retry in lockstep #4207

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion ...t/src/main/java/org/apache/gobblin/util/limiter/RedirectAwareRestClientRequestSender.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import java.net.URISyntaxException;
import java.util.List;
import java.util.Random;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
Expand Down Expand Up @@ -63,6 +64,10 @@ public class RedirectAwareRestClientRequestSender extends RestClientRequestSende

private static final int MIN_RETRIES = 3;

private static final long RETRY_MAX_DELAY_MILLIS = 10000L;
private static final long RETRY_MIN_INITIAL_DELAY_MILLIS = 500L;
private static final long RETRY_INITIAL_DELAY_RANGE_MILLIS = 500L;

/**
* A {@link SharedResourceFactory} that creates {@link RedirectAwareRestClientRequestSender}s.
* @param <S>
Expand Down Expand Up @@ -195,7 +200,18 @@ void updateRestClient(String uri, String reason, Throwable errorCause) throws No
private class CallbackDecorator implements Callback<Response<PermitAllocation>> {
private final PermitRequest originalRequest;
private final Callback<Response<PermitAllocation>> underlying;
private final ExponentialBackoff exponentialBackoff = ExponentialBackoff.builder().maxDelay(10000L).initialDelay(500L).build();
// The initial delay is randomised so that clients which lose the throttling server do not all
// retry at the same instants. Every subsequent delay is a multiple of this one, so spreading the
// initial value decorrelates the whole sequence rather than just the first retry.
//
// The random amount is added to the 500ms floor rather than centred on it, so a client never
// retries the throttling service sooner than the fixed delay already allowed. This mirrors
// MysqlMultiActiveLeaseArbiter, which randomises its initial delay the same way.
private final ExponentialBackoff exponentialBackoff = ExponentialBackoff.builder()
.maxDelay(RETRY_MAX_DELAY_MILLIS)
.initialDelay(RETRY_MIN_INITIAL_DELAY_MILLIS
+ ThreadLocalRandom.current().nextLong(RETRY_INITIAL_DELAY_RANGE_MILLIS))
.build();
private int redirects = 0;
private int retries = 0;

Expand Down