[WIP] API Changes

CLAUDE.md

@@ -0,0 +1,80 @@
+# CLAUDE.md - Notes for Claude Code
+
+## Project Overview
+Closyr is a symbolic regression tool that uses genetic algorithms to find mathematical formulas that fit data points.
+
+## Key Architecture
+
+### Symja Math Library
+- Uses `org.matheclipse.core` (Symja) for symbolic math
+- **Critical**: Parser creates different symbol objects than `F/x` or `F/Dummy "x"`
+- The system uses `ops-common/sym-x` which is `(F/Dummy "x")` as the variable
+- When parsing formula strings, the parser creates its own `x` symbol objects
+
+### Phenotype Structure
+A phenotype is a map with:
+- `:sym` - The ISymbol used in the expression (should be `ops-common/sym-x`)
+- `:expr` - The IExpr mathematical expression
+- `:util` - ExprEvaluator instance
+- `:id` - UUID
+- `:score` - Double score from evaluation
+
+### Evaluation Flow
+1. `expr->fn` wraps expression: `Function({sym}, expr)`
+2. `eval-phenotype-on-expr-args` evaluates the function at input values
+3. **Critical**: The `:sym` in the phenotype MUST match the symbol used inside `:expr`
+
+### Thread Safety
+- **Critical**: Symja's `ExprEvaluator` is NOT thread-safe
+- Has internal mutable state (ArrayDeque stacks) that corrupts under concurrent access
+- When using `pmap` for parallel scoring, ALWAYS create a fresh `ExprEvaluator` per evaluation
+- Never share `ExprEvaluator` instances across threads
+
+## Keep Going Feature (Implemented)
+
+### Overview
+Allows continuing evolution from a completed/stopped job's results by seeding the new population with previous formulas.
+
+### Implementation
+1. **Formula Parsing** (`src/closyr/ops/initialize.clj`):
+   - `parse-formula->phenotype` - Parses formula string to phenotype
+   - `seeded-phenotypes` - Creates population seeded from formulas (default 80% seeded, 20% fresh)
+   - Uses recursive tree-walk to replace parser's `x` symbol with `ops-common/sym-x`
+
+2. **API Endpoint** (`src/closyr/web/handlers/api.clj`):
+   - `POST /api/jobs/:id/continue` - Continue from a completed/stopped job
+   - Accepts `xs`, `ys`, and `config` (including `freshPercent`)
+
+3. **Frontend** (`resources/public/js/solver-history.js`):
+   - "Keep Going" button in history items
+   - `keepGoingFromHistory(index)` function
+
+### Key Technical Solution
+The Symja parser creates its own Symbol instances for "x" that are NOT equal to `F/x` or `(F/Dummy "x")`.
+Solution: Recursive tree-walk that replaces any symbol named "x" with `ops-common/sym-x`:
+
+```clojure
+(defn- make-tree-replacer [^ISymbol target-sym]
+  (ops-common/as-function
+    (fn tree-replace [^IExpr ie]
+      (cond
+        (and (.isSymbol ie) (= "x" (str ie))) target-sym
+        (instance? IAST ie) (.map ^IAST ie (make-tree-replacer target-sym))
+        :else ie))))
+```
+
+### Tests
+- `test/closyr/seeded_evolution_test.clj` - Comprehensive tests for formula parsing and seeded evolution
+
+### Error Handling
+Robust error handling at multiple levels prevents crashes during evolution:
+1. **Formula validation** (`api.clj`): `valid-seed-formula?` filters corrupted formulas before seeding
+2. **Scoring** (`ga.clj`): `with-score` catches exceptions, returns min-score
+3. **Competition** (`ga.clj`): `compete` catches exceptions, keeps parents unchanged
+4. **Evaluation** (`eval.clj`): Returns infinity vectors on failure instead of throwing
+
+## Build/Test Commands
+- `lein test` - Run all tests
+- `lein test :only ns/test-name` - Run specific test
+- `lein run --web 3000` - Start the web server
+- `lein run` - Start the java GUI

doc/depstree.txt

@@ -1,5 +1,5 @@
- [ch.qos.logback/logback-classic "1.5.0"]
-   [ch.qos.logback/logback-core "1.5.0"]
+ [ch.qos.logback/logback-classic "1.5.23"]
+   [ch.qos.logback/logback-core "1.5.23"]
  [clojure-complete "0.2.5" :exclusions [[org.clojure/clojure]]]
  [flames "0.5.0"]
    [http-kit "2.6.0"]
@@ -26,30 +26,30 @@
    [org.swinglabs/swingx "1.6.1" :scope "runtime"]
      [com.jhlabs/filters "2.0.235" :scope "runtime"]
      [org.swinglabs/swing-worker "1.1" :scope "runtime"]
- [metosin/malli "0.14.0"]
+ [metosin/malli "0.20.0"]
    [borkdude/dynaload "0.3.5"]
-   [borkdude/edamame "1.3.23"]
-   [fipp "0.6.26"]
+   [borkdude/edamame "1.4.32"]
+   [fipp "0.6.29"]
      [org.clojure/core.rrb-vector "0.1.2"]
    [mvxcvi/arrangement "2.1.0"]
    [org.clojure/test.check "1.1.1"]
  [nrepl "0.8.3" :exclusions [[org.clojure/clojure]]]
- [org.apache.logging.log4j/log4j-api "2.22.1"]
- [org.apache.logging.log4j/log4j-core "2.22.1"]
- [org.clojure/clojure "1.11.1"]
-   [org.clojure/core.specs.alpha "0.2.62"]
-   [org.clojure/spec.alpha "0.3.218"]
- [org.clojure/core.async "1.6.681"]
-   [org.clojure/tools.analyzer.jvm "1.2.3"]
-     [org.clojure/core.memoize "1.0.253"]
-       [org.clojure/core.cache "1.0.225"]
-         [org.clojure/data.priority-map "1.1.0"]
-     [org.clojure/tools.analyzer "1.1.1"]
-     [org.clojure/tools.reader "1.3.6"]
+ [org.apache.logging.log4j/log4j-api "2.25.3"]
+ [org.apache.logging.log4j/log4j-core "2.25.3"]
+ [org.clojure/clojure "1.12.4"]
+   [org.clojure/core.specs.alpha "0.4.74"]
+   [org.clojure/spec.alpha "0.5.238"]
+ [org.clojure/core.async "1.8.741"]
+   [org.clojure/tools.analyzer.jvm "1.3.2"]
+     [org.clojure/core.memoize "1.1.266"]
+       [org.clojure/core.cache "1.1.234"]
+         [org.clojure/data.priority-map "1.2.0"]
+     [org.clojure/tools.analyzer "1.2.0"]
+     [org.clojure/tools.reader "1.5.0"]
      [org.ow2.asm/asm "9.2"]
  [org.clojure/data.csv "1.1.0"]
- [org.clojure/tools.cli "1.1.230"]
- [org.knowm.xchart/xchart "3.8.7"]
+ [org.clojure/tools.cli "1.2.245"]
+ [org.knowm.xchart/xchart "3.8.8"]
    [com.madgag/animated-gif-lib "1.4"]
    [de.erichseifert.vectorgraphics2d/VectorGraphics2D "0.13"]
    [de.rototor.pdfbox/graphics2d "3.0.1"]
@@ -65,57 +65,60 @@
          [org.junit.jupiter/junit-jupiter-engine "5.10.1" :scope "runtime"]
            [org.junit.platform/junit-platform-engine "1.10.1" :scope "runtime"]
          [org.junit.jupiter/junit-jupiter-params "5.10.1"]
- [org.matheclipse/matheclipse-core "3.1.0-20240219.223701-64" :exclusions [[org.slf4j/slf4j-api] [com.fasterxml.jackson.core/jackson-core]]]
-   [com.esotericsoftware/kryo "5.6.0"]
+ [org.matheclipse/matheclipse-core "3.1.0-SNAPSHOT" :exclusions [[org.slf4j/slf4j-api] [com.fasterxml.jackson.core/jackson-core]]]
+   [com.esotericsoftware/kryo "5.6.2"]
      [com.esotericsoftware/minlog "1.3.1"]
      [com.esotericsoftware/reflectasm "1.11.9"]
-     [org.objenesis/objenesis "3.3"]
+     [org.objenesis/objenesis "3.4"]
    [com.github.freva/ascii-table "1.8.0"]
    [com.univocity/univocity-parsers "2.9.1"]
+   [commons-io "2.18.0"]
    [de.labathome/AdaptiveQuadrature "1.0.2"]
-   [it.unimi.dsi/fastutil-core "8.5.13"]
-   [org.apache.commons/commons-csv "1.10.0"]
-   [org.apache.commons/commons-lang3 "3.14.0"]
-   [org.apfloat/apfloat "1.13.0"]
-   [org.choco-solver/choco-solver "4.10.15-20240118.152009-24"]
+   [it.unimi.dsi/fastutil-core "8.5.15"]
+   [org.apache.commons/commons-csv "1.13.0"]
+     [commons-codec "1.17.2"]
+   [org.apache.commons/commons-lang3 "3.17.0"]
+   [org.apfloat/apfloat "1.14.1-SNAPSHOT"]
+   [org.choco-solver/choco-solver "4.10.18" :exclusions [[org.knowm.xchart/xchart]]]
      [dk.brics.automaton/automaton "1.11-8"]
      [net.sf.trove4j/trove4j "3.0.3"]
      [org.ehcache/sizeof "0.4.3"]
    [org.codehaus.janino/janino "3.1.12"]
      [org.codehaus.janino/commons-compiler "3.1.12"]
    [org.gavaghan/geodesy "1.1.3"]
-   [org.hipparchus/hipparchus-clustering "3.1-20240212.230522-23"]
-     [org.hipparchus/hipparchus-core "3.1-20240212.230522-23"]
-     [org.hipparchus/hipparchus-stat "3.1-20240212.230522-23"]
-   [org.hipparchus/hipparchus-fft "3.1-20240212.230522-23"]
-   [org.hipparchus/hipparchus-fitting "3.1-20240212.230522-23"]
-   [org.hipparchus/hipparchus-ode "3.1-20240212.230522-23"]
-   [org.hipparchus/hipparchus-optim "3.1-20240212.230522-23"]
-   [org.jgrapht/jgrapht-io "1.5.3-20240118.050300-19"]
+   [org.hipparchus/hipparchus-clustering "4.0-20250316.160502-27"]
+     [org.hipparchus/hipparchus-stat "4.0-20250316.160502-27"]
+   [org.hipparchus/hipparchus-fft "4.0-20250316.160502-27"]
+   [org.hipparchus/hipparchus-fitting "4.0-20250316.160502-27"]
+   [org.hipparchus/hipparchus-ode "4.0-20250316.160502-27"]
+   [org.hipparchus/hipparchus-optim "4.0-20250316.160502-27"]
+   [org.jgrapht/jgrapht-io "1.5.3-SNAPSHOT"]
      [org.antlr/antlr4-runtime "4.13.1"]
      [org.apache.commons/commons-text "1.11.0"]
-     [org.jgrapht/jgrapht-core "1.5.3-20240118.050300-19"]
+     [org.jgrapht/jgrapht-core "1.5.3-SNAPSHOT"]
        [org.jheaps/jheaps "0.14"]
-   [org.logicng/logicng-j11 "2.4.3"]
-   [org.matheclipse/matheclipse-external "3.1.0-20240219.223701-64"]
-     [com.fasterxml.jackson.core/jackson-databind "2.16.1"]
-       [com.fasterxml.jackson.core/jackson-annotations "2.16.1"]
-     [com.google.guava/guava "33.0.0-jre"]
+   [org.logicng/logicng "2.6.0"]
+   [org.matheclipse/matheclipse-external "3.1.0-SNAPSHOT"]
+     [com.fasterxml.jackson.core/jackson-databind "2.18.3"]
+       [com.fasterxml.jackson.core/jackson-annotations "2.18.3"]
+     [com.google.guava/guava "33.4.0-jre"]
        [com.google.code.findbugs/jsr305 "3.0.2"]
-       [com.google.errorprone/error_prone_annotations "2.23.0"]
+       [com.google.errorprone/error_prone_annotations "2.36.0"]
        [com.google.guava/failureaccess "1.0.2"]
        [com.google.guava/listenablefuture "9999.0-empty-to-avoid-conflict-with-guava"]
-       [com.google.j2objc/j2objc-annotations "2.8"]
-       [org.checkerframework/checker-qual "3.41.0"]
-     [io.pebbletemplates/pebble "3.2.2"]
+       [com.google.j2objc/j2objc-annotations "3.0.0"]
+       [org.checkerframework/checker-qual "3.43.0"]
+     [io.pebbletemplates/pebble "3.2.3"]
        [org.unbescape/unbescape "1.1.6.RELEASE"]
-   [org.matheclipse/matheclipse-parser "3.1.0-20240219.223701-64"]
+     [org.hipparchus/hipparchus-core "4.0-20250316.160502-27"]
+   [org.matheclipse/matheclipse-parser "3.1.0-SNAPSHOT"]
    [org.organicdesign/Paguro "3.10.3"]
      [org.jetbrains/annotations "23.0.0"]
- [org.matheclipse/matheclipse-gpl "3.1.0-20240219.223701-64" :exclusions [[org.slf4j/slf4j-api] [com.fasterxml.jackson.core/jackson-core]]]
-   [org.apache.logging.log4j/log4j-1.2-api "2.22.1"]
- [org.slf4j/jcl-over-slf4j "2.0.12"]
- [org.slf4j/slf4j-api "2.0.12"]
+   [us.hebi.matlab.mat/mfl-core "0.5.15"]
+ [org.matheclipse/matheclipse-gpl "3.1.0-SNAPSHOT" :exclusions [[org.slf4j/slf4j-api] [com.fasterxml.jackson.core/jackson-core]]]
+   [org.apache.logging.log4j/log4j-1.2-api "2.24.3"]
+ [org.slf4j/jcl-over-slf4j "2.0.17"]
+ [org.slf4j/slf4j-api "2.0.17"]
  [seesaw "1.5.0"]
    [com.fifesoft/rsyntaxtextarea "2.5.6"]
    [com.jgoodies/forms "1.2.1"]

perf-profile/README.md

@@ -0,0 +1,7 @@
+# How to run
+
+In project root:
+
+```sh
+lein run -m clojure.main ./perf-profile/profile_test.clj # or other profile class
+```

perf-profile/benchmark.clj

@@ -0,0 +1,104 @@
+(ns benchmark
+  (:require [closyr.ops.common :as ops-common]
+            [closyr.ops.eval :as ops-eval]
+            [closyr.ops.initialize :as ops-init]
+            [closyr.ops :as ops]
+            [closyr.ga :as ga]
+            [closyr.util.prng :as prng]))
+
+(defn benchmark-evolution
+  "Benchmark evolution with optional deterministic mode.
+   When random-seed is provided, runs in deterministic (single-threaded) mode."
+  [pop-size n-iters max-leafs xs-max & {:keys [random-seed]}]
+  (binding [ga/*deterministic-mode* (some? random-seed)]
+    (when random-seed
+      (prng/set-random-seed! random-seed))
+    (let [phenos (ops-init/initial-phenotypes pop-size)
+          muts (ops-init/initial-mutations)
+          xs (vec (range 0.1 xs-max 0.2))
+          ys (mapv #(+ (* % %) (Math/sin %)) xs)
+          input-xs-list (ops-common/exprs->exprs-list (ops-common/doubles->exprs xs))
+          run-args {:input-xs-list  input-xs-list
+                    :input-xs-count (count xs)
+                    :input-xs-vec   xs
+                    :input-ys-vec   ys
+                    :input-ys-arr   (double-array ys)}
+          run-config {:max-leafs max-leafs}
+          score-fn (partial ops/score-fn run-args run-config)
+          mut-fn (partial ops/mutation-fn run-config muts)
+          cross-fn (partial ops/crossover-fn run-config muts)
+          ga-config (ga/initialize phenos score-fn mut-fn cross-fn)]
+
+      ;; Warmup
+      (dotimes [_ 3] (ga/evolve ga-config))
+
+      ;; Timed run
+      (let [start (System/nanoTime)
+            result (loop [config ga-config, i 0]
+                     (if (< i n-iters)
+                       (recur (ga/evolve config) (inc i))
+                       config))
+            elapsed-ms (/ (- (System/nanoTime) start) 1e6)]
+        {:pop-size       pop-size
+         :iters          n-iters
+         :elapsed-ms     elapsed-ms
+         :ms-per-iter    (/ elapsed-ms n-iters)
+         :phenos-per-sec (/ (* pop-size n-iters 1000) elapsed-ms)
+         :deterministic  (some? random-seed)}))))
+
+(println "\n=== Evolution Benchmark ===\n")
+
+(def results* (atom []))
+
+(doseq [pop-size [10 100 500 1000 10000]]
+  (swap! results* concat ["\n"])
+  (doseq [xs-max [5.0 20.0]]
+    (doseq [max-leafs [40 120]]
+      (let [result (benchmark-evolution pop-size 10 max-leafs xs-max)]
+        (swap! results* concat [(format "Pop %4d, XsMax %4f, MaxLeafs %4d: %6.1f ms/iter, %8.0f phenos/sec\n"
+                                        pop-size
+                                        xs-max
+                                        max-leafs
+                                        (:ms-per-iter result)
+                                        (:phenos-per-sec result))])))))
+
+(println "\n=== Parallel Benchmark Done ===")
+(println "\n=== Results (Parallel Mode) ===")
+(println (apply str @results*))
+
+;; Deterministic benchmark
+(println "\n=== Deterministic Benchmark (single-threaded) ===\n")
+
+(def deterministic-results* (atom []))
+
+(doseq [pop-size [10 100 500 1000]]
+  (swap! deterministic-results* concat ["\n"])
+  (doseq [xs-max [5.0 20.0]]
+    (doseq [max-leafs [40 120]]
+      (let [result (benchmark-evolution pop-size 10 max-leafs xs-max :random-seed 12345)]
+        (swap! deterministic-results* concat [(format "Pop %4d, XsMax %4.1f, MaxLeafs %4d: %6.1f ms/iter, %8.0f phenos/sec (deterministic)\n"
+                                                      pop-size
+                                                      xs-max
+                                                      max-leafs
+                                                      (:ms-per-iter result)
+                                                      (:phenos-per-sec result))])))))
+
+(println "\n=== Results (Deterministic Mode) ===")
+(println (apply str @deterministic-results*))
+
+;; Comparison summary
+(println "\n=== Parallel vs Deterministic Comparison ===\n")
+
+(doseq [pop-size [100 1000 5000]]
+  (let [parallel-result (benchmark-evolution pop-size 10 40 5.0)
+        deterministic-result (benchmark-evolution pop-size 10 40 5.0 :random-seed 42)
+        speedup (/ (:phenos-per-sec parallel-result) (:phenos-per-sec deterministic-result))]
+    (println (format "Pop %4d: Parallel %8.0f phenos/sec, Deterministic %8.0f phenos/sec, Speedup: %.2fx"
+                     pop-size
+                     (:phenos-per-sec parallel-result)
+                     (:phenos-per-sec deterministic-result)
+                     speedup))))
+
+(println "\n=== All Benchmarks Complete ===")
+
+(System/exit 0)

perf-profile/eval_comparison.clj

@@ -0,0 +1,49 @@
+(ns eval-comparison
+  (:require [closyr.ops.common :as ops-common]
+            [closyr.ops.eval :as ops-eval])
+  (:import [org.matheclipse.core.expression F]
+           [org.matheclipse.core.interfaces IExpr IAST ISymbol]))
+
+(defn compare-eval []
+  (let [x ops-common/sym-x
+        expr (F/Plus (F/Sin x) (F/Times x x))
+        util (ops-common/new-util)
+        pheno {:sym x :util util :expr expr :id (java.util.UUID/randomUUID)}
+
+        xs (vec (range 0.1 10.0 0.1))
+        n (count xs)
+        xs-list (ops-common/exprs->exprs-list (ops-common/doubles->exprs xs))
+        run-args {:input-xs-list xs-list :input-xs-count n}
+
+        ;; Pre-compute for isolated tests
+        fn-expr (ops-common/expr->fn pheno)
+        ast (F/ast xs-list fn-expr)]
+
+    (println "\n=== Evaluation Time Breakdown ===\n")
+    (println "Points:" n)
+
+    ;; Warmup
+    (dotimes [_ 50] (ops-eval/eval-vec-pheno pheno run-args))
+
+    (println "\n1. Current optimized eval-vec-pheno (1000x):")
+    (time (dotimes [_ 1000] (ops-eval/eval-vec-pheno pheno run-args)))
+
+    (println "\n2. Just the .eval call (1000x):")
+    (time (dotimes [_ 1000] (.eval util ast)))
+
+    (println "\n3. Just result extraction from pre-computed result (1000x):")
+    (let [^IAST result (.eval util ast)]
+      (time (dotimes [_ 1000]
+              (loop [i (int 0), acc (transient [])]
+                (if (< i n)
+                  (let [^IExpr e (.getArg result (inc i) F/Infinity)
+                        v (if (.isReal e)
+                            (ops-common/expr->double e)
+                            Double/POSITIVE_INFINITY)]
+                    (recur (unchecked-inc-int i) (conj! acc v)))
+                  (persistent! acc))))))
+
+    (println "\nConclusion: .eval is ~95% of eval time, extraction is ~5%")
+    (println "\n=== Done ===")))
+
+(compare-eval)