Separate prefix for untagged nodes with --iri-prefix-for-untagged-nodes (#120)

As of this writing, the complete OSM data has 11 B objects, around 90% of which are untagged nodes, that is, nodes with zero tags (key-value pairs). Most queries involve a join of a set of tagged objects with their geometry, a minimal example being `?object osmkey:amenity "post_box" ; geo:hasGeometry ?geometry .` This is a join of a relatively small table (400 k rows in this case) with a very large table (11 B rows). In most queries, this will be a merge join, which is efficient in principle. However, the matching rows are typically "dense" in the large table because IRIs for tagged and untagged nodes both start with `https://www.openstreetmap.org/node/` followed by a number that has nothing to do with whether the node is tagged or untagged. Then the merge join has to scan through most of the very large table, which is expensive due to its sheer size.

With `--iri-prefix-for-untagged-nodes`, a distinct prefix can be given to untagged nodes, for example `http://www.openstreetmap.org/node/` (which has the nice side property of still working in a browser). The matching rows in the merge join from the example above are then all contained in a range of only around 1 B rows, which for an engine such as QLever (which is smart enough to figure out that it can skip the remaining 90% of the rows) speeds up the merge join by a factor of around 10.

When `--iri-prefix-for-untagged-nodes` is used with a prefix other than `https://www.openstreetmap.org/node/`, then also the objects of `geo:hasGeometry` and the subjects of `geo:asWKT` are given names such that the prefix for tagged and untagged nodes is different, for example, `osm2rdfgeom:osmnode_tagged_1` and `osm2rdfgeom:osmnode_untagged_137`. That way, the same speedups can be achieved for joins with the `geo:asWKT` predicate. This is important because most OSM queries involve a join with `geo:hasGeometry/geo:asWKT`, so both of the mentioned predicates.

The modified node IRIs appear in three kinds of triples in the output of `osm2rdf`. First, as subject and object of `geo:hasGeometry` . Second, as subject of `geo:asWKT`. Third, as object of `osmway:member_id`. Using the modified IRI in the first two places is trivial because when we write those triples, we have seen the node for the first time and we know whether its tagged or untagged. For the third place, it's more tricky because this happens later in the output and we need to remember whether a node is tagged or untagged. So far, only the coordinates of each node is remembered. We remember the additional bit by using the most significant bit of the `y` coordinate. That way, no more space than is used than so far, which is important because of the sheer number of nodes.

NOTE 1: To implement this, we need a modified version of the headers `Location.h` and `NodeLocationsForWays.h` from `libosmium`. According to osmcode/libosmium#395, this change cannot be made in `libosmium` itself, at least for now. We therefore use modified copies. These potentially have to be updated when pulling newer version of `libosmium`.

NOTE 2: There is still a bug in that the precomputed geometric relations (`ogc:sfContains`, etc.) don't use the modified IRIs when tagged and untagged nodes have different prefixes. This is easy to fix and will be done in a subsequent PR.

Co-authored-by: Patrick Brosi <[email protected]>

Author: 2 people

CMakeLists.txt

@@ -57,9 +57,6 @@ add_compile_options(-march=native)
 
 set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fsanitize=address,undefined")
 
-# Enable fast-math
-add_compile_options(-ffast-math)
-
 # ----------------------------------------------------------------------------
 # Configure dependencies
 # ----------------------------------------------------------------------------
@@ -95,6 +92,7 @@ find_package(POPL REQUIRED)
 include_directories(SYSTEM ${POPL_INCLUDE_DIR})
 find_package(Protozero REQUIRED)
 include_directories(SYSTEM ${PROTOZERO_INCLUDE_DIR})
+set(OSMIUM_INCLUDE_DIR "${CMAKE_SOURCE_DIR}/vendor/osmcode/libosmium/include")
 find_package(Osmium REQUIRED COMPONENTS pbf xml)
 include_directories(SYSTEM ${OSMIUM_INCLUDE_DIRS})
 

include/osm2rdf/config/Config.h

@@ -22,31 +22,26 @@
 
 #include <filesystem>
 #include <string>
+#include <thread>
 #include <unordered_set>
 #include <vector>
-#include <thread>
 
 #include "osm2rdf/config/Constants.h"
+#include "osm2rdf/ttl/Constants.h"
 #include "osm2rdf/ttl/Format.h"
 #include "osm2rdf/util/OutputMergeMode.h"
 
 namespace osm2rdf::config {
 
-enum GeoTriplesMode {
-  none = 0,
-  full = 1
-};
+enum GeoTriplesMode { none = 0, full = 1 };
 
 enum CompressFormat {
   NONE = 0,
   BZ2 = 1,
   GZ = 2,
 };
 
-enum SourceDataset {
-  OSM = 0,
-  OHM = 1
-};
+enum SourceDataset { OSM = 0, OHM = 1 };
 
 struct Config {
   // Select what to do
@@ -87,6 +82,11 @@ struct Config {
 
   bool addSpatialRelsForUntaggedNodes = true;
 
+  bool separate = true;
+
+  std::string iriPrefixForUntaggedNodes =
+      osm2rdf::ttl::constants::IRI_PREFIX__OSM_NODE_UNTAGGED;
+
   int numThreads = std::thread::hardware_concurrency();
 
   // Default settings for data

include/osm2rdf/config/Constants.h

@@ -209,6 +209,14 @@ const static inline std::string NO_UNTAGGED_NODES_OPTION_LONG =
 const static inline std::string NO_UNTAGGED_NODES_OPTION_HELP =
     "Do not output untagged nodes";
 
+const static inline std::string IRI_PREFIX_FOR_UNTAGGED_NODES_INFO =
+    "IRI prefix for untagged nodes: ";
+const static inline std::string IRI_PREFIX_FOR_UNTAGGED_NODES_OPTION_SHORT = "";
+const static inline std::string IRI_PREFIX_FOR_UNTAGGED_NODES_OPTION_LONG =
+    "iri-prefix-for-untagged-nodes";
+const static inline std::string IRI_PREFIX_FOR_UNTAGGED_NODES_OPTION_HELP =
+    "IRI prefix for untagged nodes";
+
 const static inline std::string NO_UNTAGGED_WAYS_INFO =
     "Do not output untagged ways";
 const static inline std::string NO_UNTAGGED_WAYS_OPTION_SHORT = "";

include/osm2rdf/osm/FactHandler.h

@@ -29,6 +29,8 @@
 
 namespace osm2rdf::osm {
 
+class LocationHandler;
+
 enum DateTimeType {
   invalid = 0,
   date_yyyy = 1,
@@ -42,6 +44,7 @@ class FactHandler {
  public:
   FactHandler(const osm2rdf::config::Config& config,
               osm2rdf::ttl::Writer<W>* writer);
+  void setLocationHandler(osm2rdf::osm::LocationHandler* locationHandler);
   // Add data
   void area(const osm2rdf::osm::Area& area);
   void node(const osm2rdf::osm::Node& node);
@@ -106,6 +109,7 @@ class FactHandler {
 
   const osm2rdf::config::Config _config;
   osm2rdf::ttl::Writer<W>* _writer;
+  osm2rdf::osm::LocationHandler* _locationHandler;
 };
 
 }  // namespace osm2rdf::osm