You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the following example, the MQTT `topic` property is mapped to the `origin_topic` field in the output:
197
197
198
-
```yaml
199
-
inputs:
200
-
- $metadata.topic
201
-
output: origin_topic
202
-
```
198
+
| Input | Output |
199
+
|-------|--------|
200
+
|`$metadata.topic`|`origin_topic`|
203
201
204
202
If the user property `priority` is present in the MQTT message, the following example demonstrates how to map it to an output field:
205
203
206
-
```yaml
207
-
inputs:
208
-
- $metadata.user_property.priority
209
-
output: priority
210
-
```
204
+
| Input | Output |
205
+
|-------|--------|
206
+
|`$metadata.user_property.priority`|`priority`|
211
207
212
208
### Write to metadata
213
209
@@ -217,27 +213,21 @@ Setting a metadata field to an empty value (`()`) removes it. For user propertie
217
213
218
214
You can also map metadata properties to an output header or user property. In the following example, the MQTT `topic` is mapped to the `origin_topic` field in the output's user property:
If the incoming payload contains a `priority` field, the following example demonstrates how to map it to an MQTT user property:
227
221
228
-
```yaml
229
-
inputs:
230
-
- priority
231
-
output: $metadata.user_property.priority
232
-
```
222
+
| Input | Output |
223
+
|-------|--------|
224
+
|`priority`|`$metadata.user_property.priority`|
233
225
234
226
The same example for Kafka:
235
227
236
-
```yaml
237
-
inputs:
238
-
- priority
239
-
output: $metadata.header.priority
240
-
```
228
+
| Input | Output |
229
+
|-------|--------|
230
+
|`priority`|`$metadata.header.priority`|
241
231
242
232
Metadata fields are supported in map, filter, and branch rules. They aren't available in window (accumulate) rules.
243
233
@@ -314,28 +304,11 @@ JSON objects and arrays are preserved as-is when fields are copied without an ex
314
304
315
305
## Dot notation and escaping
316
306
317
-
Dot notation is widely used to reference nested fields. A standard dot-notation sample looks like this:
307
+
Dot notation is widely used to reference nested fields. A standard dot-notation path looks like `Person.Address.Street.Number`.
318
308
319
-
```yaml
320
-
- inputs:
321
-
- Person.Address.Street.Number
322
-
```
323
-
324
-
In a data flow, a path described by dot notation might include strings and some special characters without needing escaping:
325
-
326
-
```yaml
327
-
- inputs:
328
-
- Person.Date of Birth
329
-
```
309
+
In a data flow, a path described by dot notation might include strings and some special characters without needing escaping, such as `Person.Date of Birth`.
The previous example, among other special characters, contains dots within the field name. Without escaping, the field name would serve as a separator in the dot notation itself.
311
+
In other cases, escaping is necessary, for example: `nsu=http://opcfoundation.org/UA/Plc/Applications;s=RandomSignedInt32`. This path, among other special characters, contains dots within the field name. Without escaping, the field name would serve as a separator in the dot notation itself.
339
312
340
313
While a data flow parses a path, it treats only two characters as special:
341
314
@@ -344,56 +317,18 @@ While a data flow parses a path, it treats only two characters as special:
344
317
345
318
Any other characters are treated as part of the field name. This flexibility is useful in formats like JSON, where field names can be arbitrary strings.
346
319
347
-
The path definition must also adhere to the rules of YAML. When a character with special meaning is included in the path, proper quoting is required in the configuration. Consult the YAML documentation for precise rules. Here are some examples that demonstrate the need for careful formatting:
348
-
349
-
```yaml
350
-
- inputs:
351
-
- ':Person:.:name:' # ':' cannot be used as the first character without single quotation marks
352
-
- '100 celsius.hot' # numbers followed by text would not be interpreted as a string without single quotation marks
353
-
```
320
+
The path definition must also adhere to the rules of the configuration format. When a character with special meaning is included in the path, proper quoting is required. For example, field names that start with a colon (like `:Person:.:name:`) or that begin with a number followed by text (like `100 celsius.hot`) need quoting in the configuration to be interpreted correctly as strings.
354
321
355
322
### Escaping
356
323
357
-
The primary function of escaping in a dot-notated path is to accommodate the use of dots that are part of field names rather than separators:
358
-
359
-
```yaml
360
-
- inputs:
361
-
- 'Payload."Tag.10".Value'
362
-
```
363
-
364
-
The outer single quotation marks (`'`) are necessary because of YAML syntax rules, which allow the inclusion of double quotation marks within the string.
365
-
366
-
In this example, the path consists of three segments: `Payload`, `Tag.10`, and `Value`.
324
+
The primary function of escaping in a dot-notated path is to accommodate the use of dots that are part of field names rather than separators. For example, the path `Payload."Tag.10".Value` consists of three segments: `Payload`, `Tag.10`, and `Value`. The double quotation marks around `Tag.10` prevent the dot from acting as a separator.
367
325
368
326
### Escaping rules in dot notation
369
327
370
-
* **Escape each segment separately:** If multiple segments contain dots, those segments must be enclosed in double quotation marks. Other segments can also be quoted, but it doesn't affect the path interpretation:
***Escape each segment separately:** If multiple segments contain dots, those segments must be enclosed in double quotation marks. Other segments can also be quoted, but it doesn't affect the path interpretation. For example: `Payload."Tag.10".Measurements."Vibration.$12".Value`
377
329
378
330
379
-
* **Proper use of double quotation marks:** Double quotation marks must open and close an escaped segment. Any quotation marks in the middle of the segment are considered part of the field name:
380
-
381
-
382
-
```yaml
383
-
- inputs:
384
-
- 'Payload.He said: "Hello", and waved'
385
-
```
386
-
387
-
This example defines two fields: `Payload`and `He said: "Hello", and waved`. When a dot appears under these circumstances, it continues to serve as a separator:
388
-
389
-
390
-
```yaml
391
-
- inputs:
392
-
- 'Payload.He said: "No. It is done"'
393
-
```
394
-
395
-
396
-
In this case, the path is split into the segments `Payload`, `He said: "No`, and `It is done"` (starting with a space).
331
+
***Proper use of double quotation marks:** Double quotation marks must open and close an escaped segment. Any quotation marks in the middle of the segment are considered part of the field name. For example, the path `Payload.He said: "Hello", and waved` defines two fields: `Payload` and `He said: "Hello", and waved`. When a dot appears under these circumstances, it continues to serve as a separator. For example, the path `Payload.He said: "No. It is done"` is split into the segments `Payload`, `He said: "No`, and `It is done"` (starting with a space).
397
332
398
333
### Segmentation algorithm
399
334
@@ -406,11 +341,9 @@ In many scenarios, the output record closely resembles the input record, with on
406
341
407
342
Let's consider a basic scenario to understand the use of asterisks in mappings:
408
343
409
-
```yaml
410
-
- inputs:
411
-
- '*'
412
-
output: '*'
413
-
```
344
+
| Input | Output |
345
+
|-------|--------|
346
+
|`*`|`*`|
414
347
415
348
This configuration shows a basic mapping where every field in the input is directly mapped to the same field in the output without any changes. The asterisk (`*`) serves as a wildcard that matches any field in the input record.
This revised mapping accurately captures the necessary fields. It correctly specifies the paths to include the nested `Mid` object, which ensures that the asterisks work effectively across different levels of the JSON structure.
598
510
599
511
### Specialization and second rules
600
512
601
513
When you use the previous example from multi-input wildcards, consider the following mappings that generate two derived values for each property:
This mapping is intended to create two separate calculations (`Avg` and `Diff`) for each property under `ColorProperties`. This example shows the result:
618
521
@@ -639,19 +542,10 @@ Here, the second mapping definition on the same inputs acts as a *second rule* f
639
542
640
543
Now, consider a scenario where a specific field needs a different calculation:
An empty `output` field in the second definition implies not writing the fields in the output record (effectively removing `Opacity`). This setup is more of a `Specialization` than a `Second Rule`.
562
+
An empty output field in the second definition implies not writing the fields in the output record (effectively removing `Opacity`). This setup is more of a `Specialization` than a `Second Rule`.
677
563
678
564
Resolution of overlapping mappings by data flows:
679
565
@@ -699,19 +585,15 @@ Contextualization datasets can be used with wildcards. Consider a dataset named
699
585
700
586
In an earlier example, we used a specific field from this dataset:
This mapping copies `BaseSalary` from the context dataset directly into the `Employment` section of the output record. If you want to automate the process and include all fields from the `position` dataset into the `Employment` section, you can use wildcards:
709
593
710
-
```yaml
711
-
- inputs:
712
-
- '$context(position).*'
713
-
output: 'Employment.*'
714
-
```
594
+
| Input | Output |
595
+
|-------|--------|
596
+
|`$context(position).*`|`Employment.*`|
715
597
716
598
This configuration allows for a dynamic mapping where every field within the `position` dataset is copied into the `Employment` section of the output record:
0 commit comments