Add built-in functions by aannleax · Pull Request #781 · knowsys/nemo

aannleax · 2026-04-08T14:38:56Z

This PR adds new built-in functions. Closes #642 (I moved aggregate functions to a separate issue #782) and closes #769.

String functions

REPLACE(str, pattern, replacement[, flags]) : regex-based string replacement
langMatches(tag, range): language tag range matching per SPARQL/RFC 4647
STRTRIM(str): remove leading and trailing whitespace
STRTRIMSTART(str): remove leading whitespace only
STRTRIMEND(str): remove trailing whitespace only
STRDT(lex, datatype): construct a typed literal from a lexical value and datatype IRI
Hashing functions: MD5, SHA1, SHA256, SHA384, SHA512

Date/time functions

YEAR, MONTH, DAY, HOURS, MINUTES, SECONDS: component extraction from xsd:dateTime, xsd:date, and xsd:time values
TIMEZONE: timezone as xsd:dayTimeDuration
TZ: timezone as a plain string

Nondeterministic functions

RAND(): pseudo-random double in [0, 1)
UUID() : fresh IRI-form UUID
STRUUID() : fresh string-form UUID
NOW() : current date/time as xsd:dateTime

Changes to existing functions

REGEX now accepts flags as an optional third parameter

mmarx

I've not yet reviewed everything, but I don't think this is ready to merge without a major overhaul. There's several issues both with respect to code quality and with respect to correctness:

we're pulling in chrono for handling timestamps, but then re-implement the timestamp parsing ourselves – why?
I'm not impressed with the code quality – there's functions that are defined identically in multiple files (and then inlined in other places as well), one-line functions that are only called once, redundant function calls like Some(value.to_plain_string_unchecked()) which could be a simple value.to_plain_string() instead, …
the hash functions are implemented for plain strings and language tagged strings, yet the spec says that they take plain strings and simple literals
the nondeterministic functions take an arbitrary number of arguments, even though they really should not take any arguments at all

mmarx · 2026-04-08T14:56:29Z

+fn int_type_propagation() -> FunctionTypePropagation {
+    FunctionTypePropagation::KnownOutput(StorageTypeName::Int64.bitset())
+}
+
+fn double_type_propagation() -> FunctionTypePropagation {
+    FunctionTypePropagation::KnownOutput(StorageTypeName::Double.bitset())
+}
+
+fn string_type_propagation() -> FunctionTypePropagation {
+    FunctionTypePropagation::KnownOutput(
+        StorageTypeName::Id32
+            .bitset()
+            .union(StorageTypeName::Id64.bitset()),
+    )
+}


Why are these separate functions? double_type_propagation is even only called once.

mmarx · 2026-04-08T15:01:37Z

+        if parameter_first.value_domain() != ValueDomain::PlainString {
+            return None;
+        }
+        if parameter_second.value_domain() != ValueDomain::Iri {
+            return None;
+        }
+
+        let lexical = parameter_first.to_plain_string_unchecked();
+        let datatype = parameter_second.to_iri_unchecked();
+
+        AnyDataValue::new_from_typed_literal(lexical, datatype).ok()


Suggested change

if parameter_first.value_domain() != ValueDomain::PlainString {

return None;

}

if parameter_second.value_domain() != ValueDomain::Iri {

return None;

}

let lexical = parameter_first.to_plain_string_unchecked();

let datatype = parameter_second.to_iri_unchecked();

AnyDataValue::new_from_typed_literal(lexical, datatype).ok()

AnyDataValue::new_from_typed_literal(

parameter_first.to_plain_string()?,

parameter_second.to_iri()?

).ok()

mkroetzsch · 2026-04-09T06:23:52Z

Since "simple literals" came up in the discussion: These should not be implemented as a separate type but identified with xsd:string. This is a relic from the depth of RDF history that is vanishing as specs get updated.

mmarx · 2026-04-09T17:57:11Z

+
+        let regex_str = if parameters.len() == 3 {
+            let flags = LangTaggedString::try_from(parameters[2].clone()).ok()?;
+            format!("(?{}){}", flags.string, lang_pattern.string)


This has, in principle, similar problems to the REPLACE implementation. It's somewhat mitigated by the regex crate not implementing backreferences – which SPARQL requires, so that's a different kind of problems. We probably want to use fancy-regex instead.

Ah so the regex crate doesnt implement those, because they are hard to implement efficiently. But I guess we don't mind that.

Depends on what we want – if we want to go for SPARQL compatibility, a different regex implementation is required (fancy-regex uses a hybrid approach, where the efficient NFA-based implementation is used if possible, so backreferences, lookaround, etc. become pay-as-you-go). In any case, we still shouldn't just inject flags into the pattern here, certainly not without validating them first. It's also better to use the RegexBuilder to set the options instead of splicing them into the pattern.

Oxigraph does this correctly: https://github.com/oxigraph/oxigraph/blob/5a9a8cd81f9afe5d64187aa9912705cfade833d0/lib/spareval/src/expression.rs#L1777

mmarx · 2026-04-09T17:57:44Z

+/// An optional third parameter may provide regex flags (e.g. `"i"` for case-insensitive),
+/// corresponding to the SPARQL `regex(string, pattern [, flags])` function.


This also doesn't validate the flags, and furthermore doesn't support the q flag.

mmarx · 2026-04-09T18:01:00Z

+/// Returns a language tagged string if the first parameter has a language tag.
+/// Otherwise, return a plain string.


It would be nice if we could keep the grammar consistent and not use returns and return in the same paragraph.

mmarx · 2026-04-09T18:05:58Z

+        // With flags: case-insensitive match
+        let string_flags = AnyDataValue::new_plain_string("Hello".to_string());
+        let pattern_flags = AnyDataValue::new_plain_string("hello".to_string());
+        let flags = AnyDataValue::new_plain_string("i".to_string());
+        let result_flags = AnyDataValue::new_boolean(true);
+        let actual_result_flags = StringRegex.evaluate(&[string_flags, pattern_flags, flags]);
+        assert!(actual_result_flags.is_some());
+        assert_eq!(result_flags, actual_result_flags.unwrap());
    }


i is the most boring flag to test here. What we should have test for are the q flag (since that requires special handling), though that should rather be an integration test, backreferences in pattern (again an integration test), and that invalid flags lead to None.

mmarx · 2026-04-14T12:25:02Z

+                    if *parameter_count == 0 {
+                        current_height += 1;
+                    } else {
+                        current_height -= parameter_count - 1;
+                    }


This is now a more complicated way to write the same thing as before, why?

Suggested change

if *parameter_count == 0 {

current_height += 1;

} else {

current_height -= parameter_count - 1;

}

current_height -= parameter_count - 1;

parameter count is a usize

True. It's still the most(?) convoluted way to write

Suggested change

if *parameter_count == 0 {

current_height += 1;

} else {

current_height -= parameter_count - 1;

}

current_height -= parameter_count;

current_height += 1;

(alternatively, make those isize here)

mmarx · 2026-04-14T13:52:50Z

We really need to push the nondeterministic functions into SPARQL queries. Consider the following program:

@import nemo :- sparql{ endpoint = <https://query.wikidata.org/sparql>, query = "SELECT ?x ?y WHERE { BIND(?x AS ?y) }"} .
@import sparql :- sparql{ endpoint = <https://query.wikidata.org/sparql>, query = "SELECT ?x WHERE { BIND(RAND() AS ?x) }"} .

foo(?x) :- nemo(RAND(), ?x) .
bar(?x) :- sparql(?x) .

bar correctly turns into a single SPARQL query, whereas foo will just produce an endless list of queries.

mmarx · 2026-04-15T12:20:03Z

+pub struct FuncUuid;
+impl NullaryFunction for FuncUuid {
+    fn evaluate(&self) -> Option<AnyDataValue> {
+        let iri = format!("urn:uuid:{}", Uuid::new_v4());


It's probably better to use v7 UUIDs here: a v7 UUID is always lexicographically greater than all the v7 UUIDs generated before it, since they include the current time in addition to randomness. SPARQL leaves the version implementation-defined: “The variant and version of the UUID is implementation dependent.”

mmarx · 2026-04-15T12:20:20Z

+pub struct FuncStruuid;
+impl NullaryFunction for FuncStruuid {
+    fn evaluate(&self) -> Option<AnyDataValue> {
+        Some(AnyDataValue::new_plain_string(Uuid::new_v4().to_string()))


see above, v7 might be more appropriate.

aannleax added 11 commits April 6, 2026 11:21

add replace, langMatches, and hash functions

791b004

add strdt

cee4d92

add datetime functions

25fdd82

cargo fmt

6ac5051

restore functions

002193b

add nondeterministic functions

f5fb34e

add trim

b285150

adjust tokio dependency to allow tests to be run on nemo-physical

cc10b57

cargo fmt

3249911

cargo clippy

0ebab92

add optional flag parameter for REGEX builtin

a071ce4

github-project-automation Bot added this to nemo Apr 8, 2026

github-project-automation Bot moved this to Todo in nemo Apr 8, 2026

aannleax added the builtins Issue related to built-in functions label Apr 8, 2026

aannleax added this to the Release 0.10.0 milestone Apr 8, 2026

aannleax requested a review from mmarx April 8, 2026 14:40

aannleax added 2 commits April 8, 2026 16:45

add feature flag for uuid

85f6ea7

fix uuid on wasm

3bd13a9

mmarx requested changes Apr 8, 2026

View reviewed changes

github-project-automation Bot moved this from Todo to In Progress in nemo Apr 8, 2026

aannleax added 3 commits April 8, 2026 22:50

review comments

a1edb2d

cargo fmt

5b46957

add nullary function case

6fe2681

aannleax requested a review from mmarx April 9, 2026 08:50

mmarx requested changes Apr 9, 2026

View reviewed changes

mmarx modified the milestones: Release 0.10.0, Release 0.11.0 Apr 13, 2026

mmarx reviewed Apr 14, 2026

View reviewed changes

mmarx reviewed Apr 15, 2026

View reviewed changes

		/// An optional third parameter may provide regex flags (e.g. `"i"` for case-insensitive),
		/// corresponding to the SPARQL `regex(string, pattern [, flags])` function.

		/// Returns a language tagged string if the first parameter has a language tag.
		/// Otherwise, return a plain string.

Uh oh!

Conversation

aannleax commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmarx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkroetzsch commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmarx commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aannleax commented Apr 8, 2026 •

edited

Loading

mmarx commented Apr 14, 2026 •

edited

Loading