Processing pipelines on Keyword-based detections ?

[EzLucky]

I'm trying to create a processing pipeline to apply a default field to items of a keyword-based detection.

But it seems that all conditions and transformations are applying on field-based detections (or attributes not related to detection).

Let's take the following sigma detection :

detection:
    keywords:
      - a
      - b
      - c
    condition: keywords

It produces, in Splunk for example : a OR b OR c
I was aiming at applying a pipeline to give the following output : my_field=a OR my_field=b OR my_field=c.

I even just tried to throw a SigmaTransformationError using the detection_item_failure transformation pipeline when in input I have a rule not using fields, but one more time I could not add a condition related to the keyword-based detection.

Any advice on making something work only using pipelines ?

[M3NIX]

You could try to work with the replace post processing step.
It is not a very elegant solution but that's what I got working quite fast for your example, but might fail for more complex queries.

sigconverter.io link

rule.yml

title: Test
logsource:
    category: process_creation
    product: windows    
detection:
    keywords:
      - a
      - b
      - c
    selection:
        already_a_field: test
    condition: keywords and selection

pipeline.yml

---
name: test
postprocessing:
- type: replace
 pattern: '(?<!\=)"([^"]*)"'
 replacement: 'my_new_field="\\\1"'
- type: replace
 pattern: '"\\'
 replacement: '"'

query:

my_new_field="a" OR my_new_field="b" OR my_new_field="c" already_a_field="test"

[EzLucky]

Nice workaround thank you @M3NIX !

The disadvantage of this solution is that the regex is SIEM-dependent.

[EzLucky]

I let the discussion opened as it may lead to a keyword-based transformation pipeline type 🤞

[EzLucky]

I corrected your answer @M3NIX to include wildcard in the generated SPL, I also included Lucene if someone needs this workaround :

SPLUNK

pipeline.yml :

name: splunk_keyword_replacement
postprocessing:
- type: replace
  pattern: '(?<!\=)"([^"]+?)"'
  replacement: 'my_new_field="*\\\1*"'
- type: replace
  pattern: '="\*\\'
  replacement: '="*'

result :

my_new_field="*a*" OR my_new_field="*b*" OR my_new_field="*c*"

LUCENE

pipeline.yml :

name: lucene_keyword_replacement
postprocessing:
- type: replace
  pattern: '(?<!\:)(\*[^*]+?\*)'
  replacement: 'my_new_field:\\\1'
- type: replace
  pattern: ':\\\*'
  replacement: ':*'

result :

my_new_field:*a* OR my_new_field:*b* OR my_new_field:*c*

update :
advanced Lucene regex to deal with the case already_a_field: a*b*c --> (?<!\:)(?:^|(?<=(?:\(|\s)))(\*[^*]+?\*)

[EzLucky]

Update : this workaround is not working when "already_a_field" is used with "| contains" and multivalues

title: Test
logsource:
    category: process_creation
    product: windows    
detection:
    keywords:
      - a
      - b
      - c
    selection:
        already_a_field|contains:
          - test1
          - test2
    condition: keywords and selection

[thomaspatzke]

Valid use case, should be easy to implement with the already existing field name mappings by allowing to map null values to something. I transfer this to an issue.

[EzLucky]

This pipeline is working (tested here) :

name: Default field name
transformations:
  - id: field_mapping
    type: field_name_mapping
    mapping:
      null:
        - my_field_name