Skip to content

Cluster mode does not handle load shedding properly if timeouts are exceeded. #677

Description

@jdmarshall

While looking at #628, I noticed a few other problems with the cluster mode.

The request is not deleted from the requests map until all responses are returned, which may not happen if a child is restarted.

It is also processing messages from the workers even if the request is done

	if (message.error) {
		request.done(new Error(message.error));
		return;
	}

	message.metrics.forEach(registry => request.responses.push(registry));
	request.pending--;

	if (request.pending === 0) {
		// finalize
		requests.delete(message.requestId);
		clearTimeout(request.errorTimeout);

Which means all responses from workers are processed regardless of whether another request has already sent.

The clients also aren't given a deadline, which means they have no way to abort the work if it is not processed in time.

I don't recall if there's a way to avoid the JSON.parse() call. Given the structure of the response that is unlikely.

Perhaps more importantly: every time a worker fails to deliver a response at all, the responses from its siblings are left in the lookup table in perpetuity, representing a memory leak, and a rather substantial one.

This also has implications for #645

Additionally, cluster mode does not handle benchmarking well. Multiple calls to new AggregatorRegistry() leads to addListeners() calls if more than one instance of the module is loaded - which is a frequent if unfortunate occurrence in benchmarking code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions