Skip to content

Commit 372fe4c

Browse files
Carolyn135Carolyn135
authored
Merge pull request #310020 from EdB-MSFT/cef-syslog-troubleshooting-guide
cef-syslog-troubleshooting-guide
2 parents 968a81e + 25d6abb commit 372fe4c

4 files changed

Lines changed: 387 additions & 38 deletions

File tree

‎articles/sentinel/TOC.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -718,10 +718,12 @@
718718
href: cef-syslog-ama-overview.md
719719
- name: CEF and Syslog via AMA
720720
href: connect-cef-syslog-ama.md
721-
- name: CEF - configure security device
721+
- name: Device configuration for CEF via AMA
722722
href: unified-connector-cef-device.md
723723
- name: Syslog - configure security device
724724
href: unified-connector-syslog-device.md
725+
- name: Troubleshoot CEF and Syslog via AMA
726+
href: cef-syslog-ama-troubleshooting.md
725727
- name: Custom log sources (text files)
726728
items:
727729
- name: Collect logs from text files via AMA

‎articles/sentinel/cef-syslog-ama-troubleshooting.md‎

Lines changed: 310 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,310 @@
1+
---
2+
title: Troubleshoot CEF and Syslog via AMA connectors in Microsoft Sentinel
3+
description: Learn how to troubleshoot issues with CEF and Syslog data collection using the Azure Monitor Agent (AMA) in Microsoft Sentinel.
4+
author: EdB-MSFT
5+
ms.author: edbaynash
6+
ms.topic: troubleshooting
7+
ms.date: 01/12/2026
8+
9+
# cusomtomer intent: As a Microsoft Sentinel administrator, I want to troubleshoot issues with CEF and Syslog data collection using the Azure Monitor Agent (AMA) so that I can ensure logs are being ingested correctly.
10+
---
11+
12+
# Troubleshoot CEF and Syslog via AMA connectors
13+
14+
This article provides troubleshooting guidance for Common Event Format (CEF) and Syslog data collection using the Azure Monitor Agent (AMA) in Microsoft Sentinel. Use this guide to diagnose and resolve ingestion issues with your log forwarder machines. The commands and configurations should be run on the log forwarder machines where AMA and RSyslog/Syslog-ng are installed.
15+
16+
Before you begin troubleshooting, familiarize yourself with the following articles:
17+
18+
- [Ingest syslog and CEF messages to Microsoft Sentinel with the Azure Monitor Agent](connect-cef-syslog-ama.md)
19+
- [CEF via AMA data connector - Configure specific appliance or device](cef-syslog-ama-overview.md)
20+
- [Azure Monitor Agent overview](/azure/azure-monitor/agents/azure-monitor-agent-overview)
21+
22+
## Architecture overview
23+
24+
The following diagram illustrates the data flow from log sources to Microsoft Sentinel/log analytics workspaces via RSyslog and the Azure Monitor Agent.
25+
26+
:::image type="content"source="./media/cef-syslog-ama-troubleshooting/ama-flow.png" lightbox="./media/cef-syslog-ama-troubleshooting/ama-flow.png" alt-text="Diagram showing data flow from source to Log Analytics via RSyslog and AMA.":::
27+
28+
Key components:
29+
- **RSyslog/Syslog-ng**: Receives logs on port 514 and forwards them to AMA
30+
- **Azure Monitor Agent**: Processes logs according to Data Collection Rules (DCR)
31+
- **Data Collection Rule**: Defines which logs to collect and where to send them
32+
33+
## Initial verification steps
34+
35+
### Verify logs are being received
36+
37+
Logs can take up to 20 minutes to appear in Microsoft Sentinel after configuration.
38+
39+
1. Run tcpdump to verify logs are arriving at the forwarder:
40+
41+
```bash
42+
sudo tcpdump -i any port 514 -A -vv
43+
```
44+
45+
2. Verify your log source is configured to send messages to the correct forwarder IP address.
46+
47+
3. Check for infrastructure components that might affect connectivity:
48+
- Firewalls
49+
- Load balancers
50+
- Network security groups
51+
52+
### Check Azure Monitor Agent extension status
53+
54+
1. In the Azure portal, navigate to your log forwarder virtual machine.
55+
2. Select **Extensions + applications**.
56+
3. Select the **AzureMonitorLinuxAgent** extension.
57+
4. Verify that **Status** shows **Provisioning succeeded**.
58+
59+
### Verify agent version
60+
61+
1. In the **AzureMonitorLinuxAgent** extension blade, check the **Version** field.
62+
2. Ensure the version is one of the 2-3 most recent releases. See [AMA version details](/azure/azure-monitor/agents/azure-monitor-agent-manage#agent-versions) for the latest versions.
63+
64+
> [!NOTE]
65+
> New versions may take up to 5 weeks to roll out after initial release.
66+
67+
## Agent-level troubleshooting
68+
69+
Make sure that the agent and RSyslog services are running.
70+
```bash
71+
sudo systemctl status azuremonitoragent
72+
sudo systemctl status rsyslog
73+
sudo systemctl status syslog-ng.service # If using Syslog-ng
74+
```
75+
76+
### Verify RSyslog configuration
77+
78+
The RSyslog configuration consists of `/etc/rsyslog.conf` and files in `/etc/rsyslog.d/`.
79+
80+
1. Verify port configuration:
81+
82+
```bash
83+
grep -E 'imudp|imtcp' /etc/rsyslog.conf
84+
```
85+
86+
Expected output:
87+
88+
```
89+
module(load="imudp")
90+
input(type="imudp" port="514")
91+
module(load="imtcp")
92+
input(type="imtcp" port="514")
93+
```
94+
95+
2. Verify the AMA forwarding configuration exists:
96+
97+
```bash
98+
cat /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf
99+
```
100+
101+
The file should start with:
102+
103+
```
104+
# Azure Monitor Agent configuration: forward logs to azuremonitoragent
105+
```
106+
107+
### Verify port status
108+
109+
Check that the required ports are listening:
110+
111+
```bash
112+
sudo ss -lnp | grep -E "28330|514"
113+
```
114+
115+
Expected output:
116+
117+
```
118+
udp UNCONN 0 0 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=12289,fd=5))
119+
tcp LISTEN 0 10 127.0.0.1:28330 0.0.0.0:* users:(("mdsd",pid=1424,fd=1363))
120+
tcp LISTEN 0 25 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=12289,fd=7))
121+
```
122+
123+
This confirms:
124+
- RSyslog is listening on port 514 (TCP and UDP)
125+
- MDSD (AMA component) is listening on port 28330 (TCP)
126+
127+
### Verify Data Collection Rule configuration
128+
129+
Check if the DCR is properly configured on the agent.
130+
131+
For CEF logs:
132+
133+
```bash
134+
sudo grep -i -r "SECURITY_CEF_BLOB" /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks
135+
```
136+
137+
For Cisco ASA logs:
138+
139+
```bash
140+
sudo grep -i -r "SECURITY_CISCO_ASA_BLOB" /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks
141+
```
142+
143+
The output should show a JSON string containing the DCR configuration.
144+
145+
### Review firewall rules
146+
147+
Ensure firewall rules allow communication between:
148+
- Log source and RSyslog (port 514)
149+
- RSyslog and AMA (port 28330)
150+
- AMA and Azure endpoints
151+
152+
## Data Collection Rule configuration
153+
154+
### Enable all facilities for troubleshooting
155+
156+
For initial troubleshooting:
157+
158+
1. In the Azure portal, navigate to your Data Collection Rule.
159+
2. Enable all syslog facilities.
160+
3. Select all log levels.
161+
4. If available, enable collection of messages with no facility or severity.
162+
163+
For more information, see [Select facilities and severities](connect-cef-syslog-ama.md).
164+
165+
## Common Event Format (CEF) validation
166+
167+
### CEF format requirements
168+
169+
CEF uses Syslog as a transport mechanism with this structure:
170+
171+
```
172+
<Priority>Timestamp Hostname CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]
173+
```
174+
175+
Example:
176+
177+
```
178+
Jan 18 11:07:53 host CEF:0|Vendor|Product|1.0|100|EventName|5|src=10.0.0.1 dst=10.0.0.2
179+
```
180+
181+
### Common CEF formatting issues
182+
183+
**Incorrect header format**
184+
- Ensure the CEF version is present: `CEF:0|`
185+
- All header fields must be present and delimited by pipe (|) characters
186+
187+
**Improper character escaping**
188+
- Pipe characters (|) in header values must be escaped: `\|`
189+
- Backslashes (\) must be escaped: `\\`
190+
- Equal signs (=) in extensions must be escaped: `\=`
191+
192+
**Missing or unmapped values**
193+
- If a value can't be mapped to a standard field, it's stored in the `AdditionalExtensions` column
194+
- See [CEF and CommonSecurityLog field mapping](cef-name-mapping.md) for field mappings
195+
196+
For the complete CEF specification, search for "Implementing ArcSight Common Event Format (CEF)" documentation.
197+
198+
## Advanced troubleshooting
199+
200+
### Enable diagnostic tracing
201+
202+
> [!WARNING]
203+
> Enable trace flags only for troubleshooting sessions. Trace flags generate extensive logging that can fill disk space quickly.
204+
205+
1. Edit the AMA configuration file:
206+
207+
```bash
208+
sudo vim /etc/default/azuremonitoragent
209+
```
210+
211+
2. Add trace flags to the MDSD_OPTIONS line:
212+
213+
```bash
214+
export MDSD_OPTIONS="-A -c /etc/opt/microsoft/azuremonitoragent/mdsd.xml -d -r $MDSD_ROLE_PREFIX -S $MDSD_SPOOL_DIRECTORY/eh -L $MDSD_SPOOL_DIRECTORY/events -e $MDSD_LOG_DIR/mdsd.err -w $MDSD_LOG_DIR/mdsd.warn -o $MDSD_LOG_DIR/mdsd.info -T 0x2002"
215+
```
216+
217+
3. Restart the agent:
218+
219+
```bash
220+
sudo systemctl restart azuremonitoragent
221+
```
222+
223+
4. Reproduce the issue and wait a few minutes.
224+
225+
5. Review debug information in `/var/opt/microsoft/azuremonitoragent/log/mdsd.info`.
226+
227+
6. Remove the trace flag and restart the agent after troubleshooting.
228+
229+
### Monitor log processing in real-time
230+
231+
View incoming logs as they're processed:
232+
233+
```bash
234+
tail -f /var/opt/microsoft/azuremonitoragent/log/mdsd.info
235+
```
236+
237+
Filter for specific log types:
238+
239+
```bash
240+
sudo tail -f /var/opt/microsoft/azuremonitoragent/log/mdsd.* | grep -a "CEF"
241+
```
242+
243+
Review specific facility logs:
244+
245+
```bash
246+
grep local0.info /var/opt/microsoft/azuremonitoragent/log/mdsd.info
247+
```
248+
249+
### Verify successful log processing
250+
251+
When trace flags are enabled, you can verify that logs are being processed correctly by examining the debug output.
252+
253+
#### ASA log ingestion example
254+
255+
For Cisco ASA logs, successful processing appears in the logs as:
256+
257+
```
258+
2022-01-18T22:00:14.8650520Z: virtual bool Pipe::SyslogCiscoASAPipeStage::PreProcess(std::shared_ptr<CanonicalEntity>) (.../mdsd/PipeStages.cc +604) [PipeStage] Processing CiscoASA event '%ASA-1-105003: (Primary) Monitoring on 123'
259+
260+
2022-01-18T22:00:14.8651330Z: virtual void ODSUploader::execute(const MdsTime&) (.../mdsd/ODSUploader.cc +325) Uploading 1 SECURITY_CISCO_ASA_BLOB rows to ODS.
261+
262+
2022-01-18T22:00:14.8653090Z: int ODSUploader::UploadFixedTypeLogs(const string&, const string&, const std::function<void(bool, long unsigned int, int, long unsigned int)>&, int, uint64_t) (.../mdsd/ODSUploader.cc +691) Uploading to ODS with request 3333-44dd-555555eeeeee Host https://00001111-aaaa-2222-bbbb-3333cccc4444.ods.opinsights.azure.com for datatype SECURITY_CISCO_ASA_BLOB. Payload: {"DataType":"SECURITY_CISCO_ASA_BLOB","IPName":"SecurityInsights","ManagementGroupId":"00000000-0000-0000-0000-000000000002","sourceHealthServiceId":"2c2c2c2c-3333-dddd-4444-5e5e5e5e5e5e","type":"JsonData","DataItems":[{"Facility":"local0","SeverityNumber":"6","Timestamp":"2022-01-14T23:28:49.775619Z","HostIP":"127.0.0.1","Message":" (Primary) Monitoring on 123","ProcessId":"","Severity":"info","Host":"localhost","ident":"%ASA-1-105003"}]}. Uncompressed size: 443. Request size: 322
263+
```
264+
265+
Key indicators of successful processing:
266+
- The event is recognized as a CiscoASA event
267+
- The log is uploaded to ODS (Operations Data Service)
268+
- A request ID is generated for tracking
269+
- The payload contains properly formatted JSON data
270+
271+
#### CEF log ingestion example
272+
273+
For CEF logs, successful processing appears as:
274+
275+
```
276+
2022-01-14T23:09:13.9087860Z: int ODSUploader::UploadFixedTypeLogs(const string&, const string&, const std::function<void(bool, long unsigned int, int, long unsigned int)>&, int, uint64_t) (.../mdsd/ODSUploader.cc +691) Uploading to ODS with request 3333-44dd-555555eeeeee Host https://00001111-aaaa-2222-bbbb-3333cccc4444.ods.opinsights.azure.com for datatype SECURITY_CEF_BLOB. Payload: {"DataType":"SECURITY_CEF_BLOB","IPName":"SecurityInsights","ManagementGroupId":"00000000-0000-0000-0000-000000000002","sourceHealthServiceId":"2c2c2c2c-3333-dddd-4444-5e5e5e5e5e5e","type":"JsonData","DataItems":[{"Facility":"local0","SeverityNumber":"6","Timestamp":"2022-01-14T23:08:49.731862Z","HostIP":"127.0.0.1","Message":"0|device1|PAN-OS|8.0.0|general|SYSTEM|3|rt=Nov 04 2018 07:15:46 GMTcs3Label=Virtual","ProcessId":"","Severity":"info","Host":"localhost","ident":"CEF"}]}. Uncompressed size: 482. Request size: 350
277+
```
278+
279+
Key indicators of successful CEF processing:
280+
- The datatype is SECURITY_CEF_BLOB
281+
- The upload request includes a valid endpoint
282+
- The CEF message structure is preserved in the payload
283+
- Compression metrics show the data is being optimized for transfer
284+
285+
> [!IMPORTANT]
286+
> Remember to disable trace flags after completing your investigation to prevent excessive disk usage.
287+
288+
## Collect diagnostic information
289+
290+
Before opening a support case, collect the following information:
291+
292+
### Run the AMA troubleshooter
293+
294+
The script can be run with specific flags for different log types.
295+
- `--cef`: For Common Event Format logs
296+
- `--asa`: For Cisco ASA logs
297+
- `--ftd`: For Cisco Firepower Threat Defense logs
298+
299+
The output is saved to `/tmp/troubleshooter_output_file.log`.
300+
301+
```bash
302+
sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py && sudo python3 Sentinel_AMA_troubleshoot.py [--cef | --asa | --ftd]
303+
```
304+
305+
## Related content
306+
307+
- [Ingest syslog and CEF messages with Azure Monitor Agent](connect-cef-syslog-ama.md)
308+
- [CEF and Syslog via AMA connectors overview](cef-syslog-ama-overview.md)
309+
- [Troubleshoot the Azure Monitor Agent](/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm)
310+
- [Data collection rules in Azure Monitor](/azure/azure-monitor/essentials/data-collection-rule-overview)

0 commit comments

Comments
 (0)