runtask NATS handler times out on all Windows agents (v2.10.0), runscript with wait works

[eroeinformatico]

Summary

func: runtask NATS handler on rmmagent v2.10.0 (Windows) consistently times out from server-side agent.nats_cmd({"func":"runtask","taskpk":N}, timeout=60) calls on every Windows agent, despite func: ping returning pong and func: runscript/runscriptfull (with wait=True) working correctly. This breaks the tasks/{id}/run/ API endpoint and prevents collector tasks (with custom_field binding) from being executed on demand.

Environment

• rmmagent: v2.10.0 (commit deaafa9, released 2025-01-14) — built from source github.com/amidaware/rmmagent
• TacticalRMM server: v1.4.0 (build 0.0.202)
• Backend Django: rmm container, Daphne via Unix socket
• NATS: bundled with TRMM, single-instance, port 4222 internal
• Affected agents: 8 Windows 10/11 endpoints, multiple OEM hardware (desktop + tower + small server form factors), Windows builds 19045.x and 26200.x
• Network: agents over private LAN, reverse proxy → server VM, NATS port internal only

Steps to reproduce

# Django shell on TRMM server
from agents.models import Agent
import asyncio

a = Agent.objects.filter(plat='windows').first()  # any online windows agent
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

# 1) ping → pong (NATS connectivity OK)
print(loop.run_until_complete(a.nats_cmd({"func": "ping"}, timeout=10)))
# Output: pong

# 2) runtask → ALWAYS timeout (with task_pk that exists and is synced)
print(loop.run_until_complete(a.nats_cmd({"func": "runtask", "taskpk": <existing_task_id>}, timeout=60)))
# Output: timeout

# 3) runscript via run_script() helper with wait=True → works
print(a.run_script(scriptpk=<any_script_id>, timeout=60, full=True, wait=True))
# Output: {'stdout': '...actual output...', 'stderr': '', 'retcode': 0, 'execution_time': 0.x}

loop.close()

Reproduced on 8/8 Windows agents (online, ping OK, scheduled tasks sync_status=synced, task.collector_all_output=True, task.custom_field=<set>).

Expected behavior

nats_cmd({"func": "runtask", "taskpk": N}) should:

1. Locate the Windows scheduled task TacticalRMM_<unique_name> and trigger immediate execution
2. Stream stdout/stderr back to server via NATS
3. Trigger TaskResult.save_collector_results() server-side, which populates the bound AgentCustomField

Actual behavior

NATS runtask request times out after timeout seconds. No data returned. TaskResult remains status=pending, stdout='', retcode=None. AgentCustomField never populated.

Impact

Severe for any TRMM deployment using collector pattern (asset inventory, BitLocker key escrow, Defender state, hardware serial, etc.) via tasks with collector_all_output=True + custom_field. The dashboard shows 0/N CF populated even though server reports sync_status=synced for all task instances.

Workaround applied

Server-side cron executing equivalent of:

# runs every 4h via crontab
for a in Agent.objects.filter(plat='windows'):
    if a.status != 'online': continue
    for script_id, cf_name in COLLECTOR_MAP.items():
        ret = a.run_script(scriptpk=script_id, timeout=60, full=True, wait=True)
        stdout = (ret.get('stdout','') if isinstance(ret, dict) else '').strip()
        if stdout and 'timeout' not in stdout.lower():
            cf = CustomField.objects.get(name=cf_name, model='agent')
            acf, _ = AgentCustomField.objects.get_or_create(agent=a, field=cf)
            acf.string_value = stdout.replace('\r','').strip()[:500]
            acf.save()

This bypasses runtask entirely by using run_script(wait=True) (which works) and writes CF directly to DB. Crontab line:

0 8,12,16,20 * * * cd /rmm/api/tacticalrmm && source /rmm/api/env/bin/activate && python manage.py shell < /path/to/cf-populate.py >> /var/log/cf-populate.log 2>&1

Result: 96/96 CF populated across 8 online Windows agents in <60s.

Hypothesis on root cause

Possibly in agent/agent_windows.go or wherever NATS runtask handler is implemented: the routine that locates and triggers the Windows scheduled task may be failing silently (Windows scheduler error, missing task name, permission denial, etc.) and never sends a NATS reply. From server side this manifests as timeout.

Notable: runscript (which executes the script directly inline without going through Windows Task Scheduler) works perfectly. This suggests the issue is specifically in the Task Scheduler interaction layer of rmmagent.

Suggested fix direction

1. Add explicit error handling in rmmagent runtask NATS handler to return error string instead of silently dropping the message
2. Verify Windows Task Scheduler API call (schtasks.exe /run /tn <taskname> or COM ITaskService::GetFolder("\\TacticalRMM").GetTask(name).Run()) handles missing/disabled tasks gracefully
3. Log to %ProgramFiles%\TacticalAgent\agent.log when runtask fails so users can self-diagnose

Related issues

• Linux agents not populating model/serial reliably #54 (Linux agents not populating model/serial reliably) — possibly related collector pattern issue, but Linux-specific
• serial number on linux not quite working as expected #42 (serial number on linux not quite working) — same area

Additional context

• 1/8 agents (Windows 11 build 26200.x) correctly executes the scheduled task at its native daily 08:00 UTC trigger and populates CF — so the scheduled execution path seems functional on some agents, but on-demand runtask is universally broken
• Will gladly provide more diagnostics, packet captures of NATS traffic, or test patches