You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/connectivity/dns/basic-troubleshooting-dns-resolution-problems.md
+43-39Lines changed: 43 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -94,7 +94,7 @@ To start the process, run tests from a test pod against each layer.
94
94
kubectl get pod --namespace kube-system --selector k8s-app=kube-dns --output wide
95
95
```
96
96
97
-
1. Connect to the test pod (using `kubectl exec -it aks-test -- bash`) and test the DNS resolution against each CoreDNS pod IP address by running the following commands:
97
+
1. Connect to the test pod using the `kubectl exec -it aks-test -- bash`command and test the DNS resolution against each CoreDNS pod IP address by running the following commands:
98
98
99
99
```bash
100
100
# Placeholder values
@@ -167,44 +167,46 @@ Examine the DNS server configuration of the virtual network, and determine wheth
167
167
168
168
##### Review the health and performance of CoreDNS pods
169
169
170
-
You can use kubectl commands to check the health and performance of CoreDNS pods. Start by verifying that the CoreDNS pods are running:
170
+
You can use `kubectl` commands to check the health and performance of CoreDNS pods. To do so, follow these steps:
171
171
172
-
```bash
173
-
kubectl get pods -l k8s-app=kube-dns -n kube-system
174
-
```
172
+
1. Verify that the CoreDNS pods are running:
175
173
176
-
Check whether the CoreDNS pods are overused:
174
+
```bash
175
+
kubectl get pods -l k8s-app=kube-dns -n kube-system
176
+
```
177
177
178
-
```bash
179
-
kubectl top pods -n kube-system -l k8s-app=kube-dns
180
-
```
178
+
2. Check if the CoreDNS pods are overused:
181
179
182
-
```output
183
-
NAME CPU(cores) MEMORY(bytes)
184
-
coredns-dc97c5f55-424f7 3m 23Mi
185
-
coredns-dc97c5f55-wbh4q 3m 25Mi
186
-
```
180
+
```bash
181
+
kubectl top pods -n kube-system -l k8s-app=kube-dns
182
+
```
187
183
188
-
Verify that the nodes that host the CoreDNS pods aren't overused. Also, get the nodes that are hosting the CoreDNS pods:
184
+
```output
185
+
NAME CPU(cores) MEMORY(bytes)
186
+
coredns-dc97c5f55-424f7 3m 23Mi
187
+
coredns-dc97c5f55-wbh4q 3m 25Mi
188
+
```
189
189
190
-
```bash
191
-
kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].spec.nodeName}'
192
-
```
190
+
3. Get the nodes that host the CoreDNS pods:
193
191
194
-
Check the usage of these nodes:
192
+
```bash
193
+
kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].spec.nodeName}'
194
+
```
195
195
196
-
```bash
197
-
kubectl top nodes
198
-
```
196
+
4. Verify that the nodes aren't overused:
199
197
200
-
Verify the logs for the CoreDNS pods:
198
+
```bash
199
+
kubectl top nodes
200
+
```
201
201
202
-
```bash
203
-
kubectl logs -l k8s-app=kube-dns -n kube-system
204
-
```
202
+
5. Verify the logs for the CoreDNS pods:
203
+
204
+
```bash
205
+
kubectl logs -l k8s-app=kube-dns -n kube-system
206
+
```
205
207
206
208
> [!NOTE]
207
-
> To see more debugging information, enable verbose logs in CoreDNS. To enable verbose logging in CoreDNS, see [Troubleshooting CoreDNS customizationsin AKS](/azure/aks/coredns-custom#troubleshooting).
209
+
> To get more debugging information, enable verbose logs in CoreDNS. To do so, see [Troubleshooting CoreDNS customizationin AKS](/azure/aks/coredns-custom#troubleshooting).
208
210
209
211
##### Review the health and performance of nodes
210
212
@@ -260,19 +262,20 @@ To get a better picture of resource usage at the pod and node level, you can als
260
262
261
263
#### Part 3: Analyze DNS traffic and review DNS resolution performance
262
264
263
-
Analyzing DNS traffic can help you understand how your AKS cluster is handling the DNS queries. Ideally, you want to reproduce the problem on a test pod while you capture the traffic from this test pod and on each of the CoreDNS pods.
265
+
Analyzing DNS traffic can help you understand how your AKS cluster handles the DNS queries. Ideally, you want to reproduce the problem on a test pod while you capture the traffic from this test pod and on each of the CoreDNS pods.
264
266
265
267
There are two main ways to analyze DNS traffic:
266
268
267
-
- Using real-time DNS analysis tools (e.g. [Inspektor Gadget](../../logs/capture-system-insights-from-aks.md#what-is-inspektor-gadget)) to analyze the DNS traffic in real time.
268
-
- Using traffic capture tools (e.g. [Retina Capture](https://retina.sh/docs/Troubleshooting/capture), [Dumpy](https://github.com/larryTheSlap/dumpy)) to collect the DNS traffic and analyze the traffic ina network packet analyzer tool, such as Wireshark.
269
+
- Using real-time DNS analysis tools, such as [Inspektor Gadget](../../logs/capture-system-insights-from-aks.md#what-is-inspektor-gadget), to analyze the DNS traffic in real time.
270
+
- Using traffic capture tools, such as [Retina Capture](https://retina.sh/docs/Troubleshooting/capture) and [Dumpy](https://github.com/larryTheSlap/dumpy), to collect the DNS traffic and analyze it with a network packet analyzer tool, such as Wireshark.
269
271
270
-
In both the approaches the goal would be to understand the health and performance of DNS responses using DNS response codes, response times, and other metrics. You are free to choose the approach that best fits your needs.
272
+
Both approaches aim to understand the health and performance of DNS responses using DNS response codes, response times, and other metrics. Choose the one that fits your needs best.
271
273
272
274
##### Real-time DNS traffic analysis
273
275
274
-
In this section, we will use [Inspektor Gadget](../../logs/capture-system-insights-from-aks.md#what-is-inspektor-gadget) to analyze the DNS traffic in real time. Follow this [guide](../../logs/capture-system-insights-from-aks.md#how-to-install-inspektor-gadget-in-an-aks-cluster) to install Inspektor Gadget to your cluster.
275
-
We can use the following command to trace DNS traffic across all namespaces
276
+
You can use [Inspektor Gadget](../../logs/capture-system-insights-from-aks.md#what-is-inspektor-gadget) to analyze the DNS traffic in real time. To install Inspektor Gadget to your cluster, see [How to install Inspektor Gadget in an AKS cluster](../../logs/capture-system-insights-from-aks.md#how-to-install-inspektor-gadget-in-an-aks-cluster).
277
+
278
+
To trace DNS traffic across all namespaces, use the following command:
276
279
277
280
```bash
278
281
# Get the version of Gadget
@@ -281,7 +284,8 @@ GADGET_VERSION=$(kubectl gadget version | grep Server | awk '{print $3}')
281
284
kubectl gadget run trace_dns:$GADGET_VERSION --all-namespaces --fields "src,dst,name,qr,qtype,id,rcode,latency_ns"
282
285
```
283
286
284
-
Where `--fields` is a comma-separated list of fields to be displayed. The following table describes the fields that are used in the command:
287
+
Where `--fields` is a comma-separated list of fields to displayed. The following list describes the fields that are used in the command:
288
+
285
289
- `src`: The source of the request with Kubernetes information (`<kind>/<namespace>/<name>:<port>`).
286
290
- `dst`: The destination of the request with Kubernetes information (`<kind>/<namespace>/<name>:<port>`).
287
291
- `name`: The name of the DNS request.
@@ -291,7 +295,7 @@ Where `--fields` is a comma-separated list of fields to be displayed. The follow
291
295
- `rcode`: The response code of the DNS request.
292
296
- `latency_ns`: The latency of the DNS request.
293
297
294
-
The output of the commandwill look like the following:
p/kube-system/coredns-57d886c994-r2… p/default/aks-test:56921 db.contoso.com. R A 6574 NameErr… 0ns
304
308
```
305
309
306
-
Here you can use `ID` to identify if a query has a response or not. The `RCODE` field will show you the response code of the DNS request. The `LATENCY_NS` field will show you the latency of the DNS request in nanoseconds. Together, these fields can help you understand the health and performance of DNS responses.
307
-
For more information about real-time DNS analysis, see [Troubleshoot DNS failures across an AKS cluster in real time](troubleshoot-dns-failures-across-an-aks-cluster-in-real-time.md)
310
+
You can use the `ID`field to identify if a query has a response or not. The `RCODE` field shows you the response code of the DNS request. The `LATENCY_NS` field shows you the latency of the DNS request in nanoseconds. These fields can help you understand the health and performance of DNS responses.
311
+
For more information about real-time DNS analysis, see [Troubleshoot DNS failures across an AKS cluster in real time](troubleshoot-dns-failures-across-an-aks-cluster-in-real-time.md).
308
312
309
313
##### Capture DNS traffic
310
314
311
-
In this section, we use Dumpy as an example of how to collect DNS traffic captures from each CoreDNS pod and a client DNS pod (in this case, the `aks-test` pod).
315
+
This section demonstrates how to use Dumpy to collect DNS traffic captures from each CoreDNS pod and a client DNS pod (in this case, the `aks-test` pod).
312
316
313
-
To collect the captures from the test client pod, run the following Dumpy command:
317
+
To collect the captures from the test client pod, run the following command:
314
318
315
319
```bash
316
320
kubectl dumpy capture pod aks-test -f "-i any port 53" --name dns-cap1-aks-test
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/connectivity/dns/troubleshoot-dns-failures-across-an-aks-cluster-in-real-time.md
+7-4Lines changed: 7 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ ms.date: 08/09/2024
8
8
---
9
9
# Real-time DNS traffic analysis to troubleshoot DNS resolution failures in AKS
10
10
11
-
This article discusses how to troubleshoot DNS failures across an Azure Kubernetes Service (AKS) cluster in real time.
11
+
This article discusses how to troubleshoot Domain Name System (DNS) failures across an Azure Kubernetes Service (AKS) cluster in real time.
12
12
13
13
> [!NOTE]
14
14
> This article is complementary to [Basic troubleshooting of DNS resolution problems in AKS](basic-troubleshooting-dns-resolution-problems.md) guide.
@@ -79,12 +79,14 @@ Here are some causes of unsuccessful DNS responses:
79
79
80
80
- The DNS name being resolved has a typo.
81
81
- The upstream DNS nameservers experience issues.
82
-
- A misconfigured networking configuration (netpol, firewall, NSG rules etc.) is blocking DNS traffic.
83
-
- The DNS name becomes invalid after expansion. To understand how DNS queries might be expanded using a pod's `/etc/resolv.conf`, see [Namespaces of Services](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#namespaces-of-services).
82
+
- A misconfigured networking configuration (such as NetworkPolicy (NetPol), firewall or NSG rules) blocks DNS traffic.
83
+
- The DNS name becomes invalid after expansion.
84
+
85
+
To understand how DNS queries might be expanded using the `/etc/resolv.conf` file of a pod, see [Namespaces of Services](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#namespaces-of-services).
84
86
85
87
## Step 2: Identify slow DNS queries across the cluster
86
88
87
-
You can use the DNS gadget to identify all the slow DNS queries across the cluster. To perform this check, trace DNS packets on all the nodes and filter slow responses.
89
+
You can use the DNS gadget to identify all the slow DNS queries across the cluster. To perform this check, trace DNS packets on all the nodes and filter slow responses.
88
90
89
91
In the following example, you're using a latency value of `5ms` to define slow packets. You can change it to a desired value, for example, `5μs`, `20ms`, `1s`:
90
92
@@ -96,6 +98,7 @@ kubectl gadget run trace_dns:$GADGET_VERSION \
96
98
```
97
99
98
100
Here are the explanations of the command parameters:
101
+
99
102
-`--all-namespaces`: Shows data from pods in all namespaces.
100
103
-`--fields k8s.node,src,dst,name,qtype,rcode,latency_ns`: Shows only Kubernetes information, source and destination of the DNS packets, DNS name, DNS query type, DNS response code, and latency in nanoseconds.
101
104
-`--filter "latency_ns_raw>5000000"`: Matches only DNS responses with latency greater than `5ms` (`5000000` nanoseconds).
0 commit comments