[Bug] Edge 4.1 cloud uplink does not recover after gRPC keepalive failure when EpollEventLoopGroup creation fails

[bcblr1993]

Describe the bug

On a ThingsBoard Edge 4.1.0-based deployment, the Edge-to-Cloud uplink stopped permanently although the host-to-cloud network was reachable and local data ingestion continued.

The first transport error was a gRPC keepalive failure. Every subsequent reconnect attempt (once per minute) failed locally with Cannot create Epoll EventLoopGroup. The Edge stayed up and continued storing cloud events locally, but no data was sent to the Cloud until the Edge process was restarted. After restart, the uplink and PostgreSQL history backfill resumed without any Cloud restart.

This report is based on the incident logs, a thread dump, a heap histogram, an Eclipse MAT leak-suspect report, process/cgroup snapshots, and a source review. Identifiers, addresses, and payloads have been removed.

Your Server Environment

• Own setup, Docker on Linux x86_64
• ThingsBoard Edge: custom build based on upstream v4.1 (4.1.0EDGE); the two relevant upstream classes below are byte-for-byte identical to tag v4.1
  • common/edge-api/.../EdgeGrpcClient.java
  • common/script/script-api/.../NashornJsInvokeService.java
• Java: OpenJDK 17.0.15
• Cloud RPC: port 7070 only
• TELEMETRY_SEPARATION_ENABLED=false
• Queue: in-memory (no Kafka)
• Local cloud-event queue: PostgreSQL, with realtime-priority/backfill enabled
• History storage: Cassandra
• Latest-value storage: Cassandra
• Host memory: approximately 32 GiB

Impact and timeline

• Process started on July 7.
• Last successful uplink was on July 9 at 19:08:49.
• Persistent Cannot create Epoll EventLoopGroup started on July 9 at 19:10:08.
• From July 10 through July 16, both errors occurred 1,440 times per day (approximately one reconnect attempt per minute), with no successful uplink.
• Across the retained logs there were approximately 11,936 keepalive failures and 10,370 Epoll group creation failures.
• The Edge was restarted on July 17 at 18:31; uplink and PostgreSQL history backfill resumed at 18:33.

Representative sanitized log sequence:

... Trying to reconnect due to error:
io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone!

... Exception during connect: Cannot create Epoll EventLoopGroup

The same pair then repeated every minute. The current reconnect log records only ex.getMessage(), so the nested cause of the Epoll construction failure is missing.

Resource and heap evidence captured while affected

• Container RSS: approximately 15.65 GiB
• Java process swap: approximately 3.08 GiB
• No Docker memory limit and no explicit -Xmx
• Java threads: approximately 619
• Open file descriptors: approximately 504 of 65,536
• Sockets: 139; epoll FDs: 175
• No Java deadlock in the thread dump
• cgroup PID limit was unlimited
• PostgreSQL cloud-event writes continued successfully and local events accumulated for uplink

The FD, PID/thread-limit, and local database snapshots therefore do not indicate an exhausted FD/PID limit or a local storage outage.

Eclipse MAT found two dominant org.openjdk.nashorn.internal.runtime.PropertyMap retainers:

• 1,292,323,656 bytes (44.52%)
• 1,282,165,656 bytes (44.17%)
• Combined: approximately 2.574 GB / 88.69% of the reachable heap

The retained path is centered on:

PropertyMap.history -> WeakHashMap -> PropertyHashMap

The heap histogram included:

PropertyMap                         3,918,530
PropertyHashMap$Element             9,018,082
PropertyHashMap$Element[]           2,562,535

One MAT path reaches the retained Nashorn state through NashornSandboxImpl -> NashornJsInvokeService -> rule-engine consumer manager.

In upstream tag v4.1, NashornJsInvokeService.doRelease() releases a generated script function by assigning undefined:

sandbox.eval(scriptInfo.getFunctionName() + " = undefined;");

The observed heap suggests that repeated script load/release leaves Global-object shape history/property metadata retained even after the function value is cleared.

The runtime also emitted two sampled Netty leak-detector reports:

LEAK: ByteBuf.release() was not called before it's garbage-collected
MessageFramer.writeCompressed
  -> EdgeGrpcClient.sendUplinkMsg
  -> BaseCloudManagerService.sendUplinkMsgPack

This is additional evidence of resource pressure, but it is not claimed as the sole cause of the outage.

To Reproduce / validation performed

The long-running production trigger has not yet been reduced to a short deterministic upstream-only test.

1. Run Edge 4.1 with local Nashorn JS evaluation/sandbox and a sustained stream of telemetry.
2. Repeatedly load, replace, and release uniquely named JS rule scripts while watching PropertyMap, PropertyHashMap$Element, heap, RSS, and swap.
3. After prolonged runtime/resource pressure, interrupt the Cloud RPC connection on port 7070 so the gRPC client must reconnect.
4. Observe whether the original channel is shut down and a new EpollEventLoopGroup can be created.
5. Restore network connectivity and verify whether the uplink self-recovers without restarting Edge.

Two controlled tests help narrow the behavior:

• Network churn alone (43 disconnect/reconnect cycles) produced gRPC UNAVAILABLE errors and one sampled ByteBuf leak, but did not reproduce the persistent Epoll creation failure. This indicates that an ordinary network flap alone is insufficient.
• Artificially forcing native-thread creation failure while closing the 7070 socket reproduced the same externally visible state: Edge HTTP stayed healthy, local PostgreSQL events continued accumulating, and the RPC uplink did not self-recover. This is only a symptom-level validation; the field snapshot did not have an exhausted PID limit.

Expected behavior

• A keepalive failure should cause the old channel/event-loop resources to be fully released and the Edge should reconnect automatically after network recovery.
• Repeated JS script load/release should not cause unbounded Nashorn PropertyMap retention.
• If channel creation fails, the full nested exception should be logged and later reconnect attempts should recover when resources are available.
• No Edge restart should be required to resume uplink/backfill.

Actual behavior

• All reconnect attempts failed with Cannot create Epoll EventLoopGroup for several days.
• Local PostgreSQL/Cassandra activity continued, but the 7070 Cloud uplink remained down.
• Only restarting the Edge process released the condition and resumed history backfill.

Relevant source paths / investigation areas

1. common/edge-api/src/main/java/org/thingsboard/edge/rpc/EdgeGrpcClient.java
   • A new NettyChannelBuilder/channel is built on every connect.
   • Please verify channel, executor, and event-loop termination across repeated disconnect/reconnect and send-in-progress races.
2. common/script/script-api/src/main/java/org/thingsboard/script/api/js/NashornJsInvokeService.java
   • doRelease() assigns undefined rather than deleting the generated Global property or periodically rebuilding the sandbox.
   • Please add a repeated eval/release regression test that verifies Nashorn property metadata is bounded.
3. Edge reconnect error logging
   • Please log the complete exception (log.error(..., ex)) rather than only ex.getMessage() so native-memory, direct-memory, thread-creation, and underlying epoll IOException causes can be distinguished.
4. Uplink buffer lifecycle
   • Please test disconnect during compressed batch send with Netty leak detection enabled.

Root-cause assessment and limits

The direct failure is well supported: after the gRPC keepalive failure, the Edge could no longer create the local Netty Epoll event-loop resources required for a new Cloud channel.

The strongest preceding resource-pressure evidence is the Nashorn heap retention (about 88.69% of the reachable heap), with sampled Netty ByteBuf leaks/reconnect churn as a possible additional contributor. However, because the reconnect code logs only the outer message, the existing evidence cannot prove whether the final Epoll constructor failed specifically because of native memory, direct memory, thread creation, or another underlying IOException.

A sanitized MAT report, histogram, thread dump, and selected log excerpts can be provided if maintainers specify a safe upload method. The original heap dump and logs contain customer data and cannot be posted publicly.

[volodymyr-babak]

@bcblr1993

Thank you for the exceptionally detailed report — the logs, heap histogram, MAT leak-suspect analysis, and source-level investigation made this very easy to confirm.

We reviewed the code at v4.1 and verified your root-cause chain:

• Nashorn leak (root cause): NashornJsInvokeService.doRelease() clears the script function with = undefined but never removes the property from the Nashorn Global object, so PropertyMap grows unbounded — the source of the heap retention you captured.
• Failure surface: under that memory/native pressure, the shared EpollEventLoopGroup can't be re-created on reconnect, which is why the uplink never recovers without a restart.
• Diagnosability gap: the reconnect path logs only ex.getMessage(), which is why the nested cause was hidden.

We've created internal tracking issues for all of these (the Nashorn leak, full-exception logging on reconnect, reconnect resilience, and the sampled ByteBuf leak) and will address them in an upcoming release. We'll follow up here once the fixes are merged.

Thanks again for the great write-up!

[bcblr1993]

Thank you for confirming the root cause and for creating the internal tracking issues. We really appreciate the detailed follow-up.

We look forward to seeing the fixes for the Nashorn leak, reconnect resilience, full exception logging, and the sampled ByteBuf leak included in an upcoming release as soon as possible.

Please let us know if any additional sanitized logs, MAT reports, or reproduction results would be helpful. Thanks again!

[bcblr1993]

Follow-up: direct-memory retention and terminated-stream retry evidence

A deeper pass over the same sanitized field artifacts provides a much stronger, independent explanation for the native/direct-memory pressure. This supplements the confirmed Nashorn heap-retention issue; it does not invalidate that finding.

Timeline correction

The captured Docker metadata reports:

Created=2026-07-07T10:52:40Z
Started=2026-07-07T10:52:41Z
Restarts=0

Therefore, the recovery seen on July 17 at 18:32 should be described as a Cloud Edge service/RPC reinitialization, not a confirmed container restart. The same container later experienced another transport failure at 19:28.

Exact failure sequence

The July 17 log contains the following sequence while a 2,000-event telemetry/backfill batch was being sent:

19:28:16 Stream was terminated due to error
Caused by: writevAddresses(..) failed: Connection timed out

19:29:10 Failed to deliver Timeseries batch (size: 2000) on attempt 1

19:29:11 RejectedExecutionException: event executor terminated
  at WriteQueue.scheduleFlush
  at WriteQueue.enqueue
  at NettyClientStream$Sink.writeFrame
  at MessageFramer.flush
  at ClientCallImpl.sendMessage
  at EdgeGrpcClient.sendUplinkMsg(EdgeGrpcClient.java:241)
  at BaseCloudManagerService.sendUplinkMsg(BaseCloudManagerService.java:839)
  at BaseCloudManagerService.sendUplinkMsgPack(BaseCloudManagerService.java:822)
  at BaseCloudManagerService.processMsgPack(BaseCloudManagerService.java:781)

19:29:11 Failed to deliver Timeseries batch (size: 2000) on attempt 2
19:29:26 Failed to deliver Timeseries batch (size: 2000) on attempt 3

Shortly afterwards, Netty's leak detector reported the exact allocation path:

LEAK: ByteBuf.release() was not called before it was garbage-collected

PooledByteBufAllocator.newDirectBuffer
  -> NettyWritableBufferAllocator.allocate
  -> MessageFramer.writeCompressed / writePayload
  -> ClientCallImpl.sendMessage
  -> CallToStreamObserverAdapter.onNext
  -> EdgeGrpcClient.sendUplinkMsg
  -> BaseCloudManagerService.sendUplinkMsgPack
  -> BaseCloudManagerService.processMsgPack

The same leak path was already sampled on July 7, minutes after the first network timeout.

Direct-memory object evidence

The heap histogram records the Java-side metadata for the shaded gRPC Netty allocator:

Class	Count
io.grpc.netty.shaded.io.netty.buffer.PoolChunk	3,976
PoolSubpage	1,011,539
java.nio.DirectByteBuffer	5,229
DirectByteBuffer$Deallocator	4,276
PooledUnsafeDirectByteBuf	3,545
SendGrpcFrameCommand	3,488
NettyClientStream$Sink$2	3,487
UplinkMsg	1,539

With the gRPC/Netty allocator's 4 MiB pooled chunk size:

3,976 * 4 MiB = 15.53125 GiB

The captured process/container values were:

Java VmRSS: 15.544 GiB
Container memory: 15.65 GiB
Java VmSwap: 3.08 GiB

The near-exact match between pooled chunk capacity and RSS is strong evidence that the dominant resident memory was the gRPC Netty direct-memory pool, not Java heap alone.

The same histogram contains only one ManagedChannelImpl, one NettyClientTransport, one NettyClientStream, and one WriteQueue. This does not look like thousands of leaked channels; it looks like thousands of outbound frame/buffer objects retained around a terminated/current stream and the shared allocator.

Upstream release-4.1 retry path

The relevant code is also present in the upstream release-4.1 branch:

1. BaseCloudManagerService.processMsgPack() uses do { ... } while (!success).
2. MAX_SEND_UPLINK_ATTEMPTS is applied only when isGeneralMsg == true.
3. Timeseries batches (isGeneralMsg == false) therefore have no attempt limit.
4. sendUplinkMsgPack() calls orderedPendingMsgQueue.forEach(this::sendUplinkMsg).
5. EdgeGrpcClient.sendUplinkMsg() unconditionally calls inputStream.onNext(...) without checking whether that stream/session/event loop has already terminated.
6. When WriteQueue.enqueue() rejects the frame because the executor is terminated, the batch returns false and the Timeseries loop sends the remaining batch again.

Across the retained July 7-17 logs, the relevant counts were:

Event	Count
gRPC stream terminated	1,775
reconnect attempts	13,256
Cannot create Epoll EventLoopGroup	11,482
RejectedExecutionException: event executor terminated	1,551
failed Timeseries batch attempts	4,925
sampled ByteBuf.release() leak reports	4

Refined root-cause assessment

A network timeout is the trigger. The unbounded Timeseries retry loop then continues calling onNext after the gRPC stream/event executor has terminated. gRPC allocates compressed direct buffers before the write is rejected by the terminated executor, and the sampled error path does not release those buffers. Repeating an entire telemetry batch amplifies this into sustained direct-memory growth.

The Nashorn PropertyMap retention remains a separate confirmed heap-pressure issue and reduces available headroom. However, the Netty object counts, allocator arithmetic, RSS match, and allocation stack show that the uplink retry path itself is a concrete direct-memory leak/amplifier and should be fixed independently.

Suggested fix/acceptance criteria

• Mark the RPC session disconnected before scheduling reconnect and stop/cancel the active uplink batch.
• Do not call inputStream.onNext unless the exact session is still active.
• Associate callbacks with a connection generation so a delayed old onError cannot close a newer channel.
• Do not actively retry Timeseries batches while disconnected. Keep PostgreSQL offsets unchanged and resume after a successful connection.
• Bound active retries and use backoff; persisted events should remain available for later replay.
• Log the complete reconnect exception, including the nested cause.
• Regression test: disconnect during a compressed 2,000-message telemetry batch, hold the network down, and verify that direct memory reaches a plateau, no ByteBuf leak is reported, no sends target a terminated executor, and the persisted backlog resumes after reconnection without premature offset advancement.

The original heap dump and full logs contain customer payloads and should not be uploaded publicly. Sanitized histogram sections and selected stack traces can be provided if useful.