Incorrect Timestamps for Linux Events

[MakoWish]

I have previously noticed an issue with what appeared to a massive delay in events being ingested from Linux devices to Elasticsearch, but I finally looked into it a bit more this morning and found that the documents are being written to Elasticsearch with the local time of the machine's local time, minus its timezone offset, instead of the actual UTC time. Since I am located in Colorado with timezone UTC-6, this is causing all events from my Ubuntu machine to appear six hours delayed.

I have attached a screenshot of the current local and UTC time of my computer (as well as OS version in case that info is helpful for the Devs), and one document from Elasticsearch showing that it is stored with my local time minus 6 offset instead of the UTC time. This is a pretty critical issue, as it affects event correlation.

linux_incorrect_timestamp_in_elasticsearch.json

Thank you,
Eric

Support Ticket: 02116258

[nicholasberlin]

This appears to be a boot time caching issue. It's likely restarting elastic-endpoint will fix the issue for the current boot, unless the time is changed.

Working on a fix.

[nicholasberlin]

@MakoWish First, thanks for the report and the debugging work. The @timestamp vs event.ingested is perfect evidence. We would appreciate some more debugging help to understand how this happened.

Sorry for the following wall of AI text, but I've read/edited it a bit, and I stand by it.

Working theory

The @timestamp vs. event.ingested delta being exactly your UTC-6 offset, while your live clock is NTP-synced and correct, points at a specific failure mode:

1. The system clock was ~6 hours behind true UTC when the machine booted — most likely because the (physical or virtual) RTC held local Denver time instead of UTC. Prime suspects: a Windows dual-boot rewriting the RTC to local time, or a VM whose hypervisor seeds the guest clock with the host's local time.
2. Endpoint started before NTP corrected the clock and cached the boot-time epoch (btime from /proc/stat) once, at startup putting everything off by 6 hours.
3. NTP then stepped the clock forward 6 hours, fixing everything live (which is why timedatectl looks correct now). The kernel also wrote correct UTC back to the RTC, hiding the original evidence.
4. Endpoint's boot time cache is never updated, so every event since is off and will be until Endpoint is restarted.

Diagnostics

If you can, please run these and share the output.

1. Look for the clock step near boot. chrony logs System clock wrong by 21600.x seconds and systemd-timesyncd logs an initial synchronization jump. A ~21600s (6h) step right after boot would confirm the theory.

journalctl -b | grep -iE "system clock|clock wrong|time (set|step|jump|change)|synchroniz"

2. Compare when the kernel thinks it booted vs. reality. uptime -s is derived from the current (corrected) clock, while the first journal lines of this boot were written with the original (possibly wrong) clock. If the first journal entries are timestamped ~6h before uptime -s, the clock was 6h behind at boot.

uptime -s
journalctl -b --output=short-iso | head -5

3. Restart the Endpoint. Endpoint re-reads the boot time at startup; since your clock is correct now, new events should get correct timestamps if the theory is right. This both confirms the diagnosis and fixes the machine until the next reboot with a wrong clock.

sudo systemctl restart ElasticEndpoint

Then generate some activity and check whether new documents in Kibana have correct timestamps.

Workarounds until a fix ships

• If the machine dual-boots Windows: either set Windows to keep the RTC in UTC (RealTimeIsUniversal=1 under HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation), or tell Ubuntu the RTC is local time: sudo timedatectl set-local-rtc 1 --adjust-system-clock.
• If it's a VM: configure the hypervisor to expose the RTC as UTC.
• Restarting elastic-agent any time after NTP sync corrects timestamps for the remainder of that boot.

[MakoWish]

Hi @nicholasberlin ,

Yes, this machine is dual boot. I rarely boot into Windows just for some compatibility stuff with my work, but I believe the previous reboot was actually from Windows going back into my Ubuntu daily driver.

Diag 1:

journalctl -b | grep -iE "system clock|clock wrong|time (set|step|jump|change)|synchroniz"
Jul 13 02:10:03 COMPUTER1 kernel: rtc_cmos 00:01: setting system clock to 2026-07-13T08:09:59 UTC (1783930199)
Jul 13 02:10:04 COMPUTER1 systemd[1]: Starting systemd-timesyncd.service - Network Time Synchronization...
Jul 13 02:10:04 COMPUTER1 systemd[1]: Started systemd-timesyncd.service - Network Time Synchronization.
Jul 13 02:10:04 COMPUTER1 systemd[1]: Reached target time-set.target - System Time Set.
Jul 13 02:10:07 COMPUTER1 smartd[1618]: Device: /dev/sda [SAT], SMART Prefailure Attribute: 3 Spin_Up_Time changed from 146 to 143
Jul 13 08:10:36 COMPUTER1 systemd-timesyncd[1248]: Initial clock synchronization to Mon 2026-07-13 08:10:36.217633 MDT.

Diag 2:

uptime -s
journalctl -b --output=short-iso | head -5
2026-07-13 08:09:57
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
2026-07-13T08:19:20-06:00 COMPUTER1 sudo[24334]: user1 : TTY=pts/2 ; PWD=/home/user1 ; USER=root ; COMMAND=/bin/bash
2026-07-13T08:19:20-06:00 COMPUTER1 sudo[24334]: pam_intune(sudo:session): No authtok available; password policies will fail: No module specific data is present
2026-07-13T08:19:20-06:00 COMPUTER1 sudo[24334]: pam_unix(sudo:session): session opened for user root(uid=0) by user1(uid=136631)
2026-07-13T09:17:17-06:00 COMPUTER1 sudo[33498]: user1 : TTY=pts/5 ; PWD=/home/user1 ; USER=root ; COMMAND=/usr/bin/apt update
2026-07-13T09:17:17-06:00 COMPUTER1 sudo[33498]: pam_unix(sudo:session): session opened for user root(uid=0) by user1(uid=136631)

Diag 3:
I restarted the ElasticEndpoint service, and the events are now flowing in with the correct timestamp. I believe you found the issue! I am a rare dual boot user within the company, so that could explain why it is only my computer showing the issue.

[nicholasberlin]

Cool, thanks for the update. Seems like the Linux command from above:

sudo timedatectl set-local-rtc 1 --adjust-system-clock

is the simplest "fix".

That will tell Linux to treat the real time clock as local instead of UTC, and will survive reboots.
The bots can tell you all about it, of course.

[MakoWish]

I did catch wind just two weeks ago of two developers who are requesting to dual-boot Windows and Ubuntu. Since I am aware of those two, I can apply that "fix" to those, but I may not catch wind of others in the future. Can a bugfix be submit for this so that it is permanent on the Elastic Endpoint side?

[nicholasberlin]

Can a bugfix be submit for this so that it is permanent on the Elastic Endpoint side?

We're actively working on a fix, it will be fixed in Endpoint. I added the bug label to this issue, and don't intend to close it until it's fixed in Endpoint.

[MakoWish]

Thank you @nicholasberlin !