If a pipeline has more than 9 stages, the saved stages cannot be loaded. I think the code adds a single digit suffix for the path. Please check out the logs below.
Saved model at: [./target/test_models/2.3.1/regextokenizer-...-indextostring]
Loading model from: [./target/test_models/2.3.1/regextokenizer-...-indextostring]
File target/test_models/2.3.1/regextokenizer-...-indextostring/stages/0_regexTok_a4348c4a1fd1/metadata/part-00000 does not exist
java.io.FileNotFoundException: File target/test_models/2.3.1/regextokenizer-...-indextostring/stages/0_regexTok_a4348c4a1fd1/metadata/part-00000 does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:539)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:752)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:529)
However, the files are there, as shown below:
samik@samik-lap:~/git/spark-ml-serving/target/test_models/2.3.1/regextokenizer-...-indextostring/stages$ l
total 76K
drwxr-xr-x 19 samik samik 4.0K Sep 5 18:07 .
drwxr-xr-x 4 samik samik 4.0K Sep 5 18:07 ..
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 00_regexTok_a4348c4a1fd1
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 01_stopWords_4061e58af19e
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 02_ngram_3eebcba67f89
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 03_ngram_00c144c7006b
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 04_ngram_c8cda5f15e23
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 05_hashingTF_da56ada55a20
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 06_hashingTF_426b130af12d
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 07_hashingTF_3fdadf69e9c4
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 08_hashingTF_5e81e16aa1df
drwxr-xr-x 4 samik samik 4.0K Sep 5 18:07 09_idf_91a0d9e2bd25
drwxr-xr-x 4 samik samik 4.0K Sep 5 18:07 10_idf_ff7e432c7377
drwxr-xr-x 4 samik samik 4.0K Sep 5 18:07 11_idf_297287d8b400
drwxr-xr-x 4 samik samik 4.0K Sep 5 18:07 12_idf_45a0afd1fe51
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 13_vecAssembler_7a2ee0c762c5
drwxr-xr-x 4 samik samik 4.0K Sep 5 18:07 14_strIdx_b7fbad214c7e
drwxr-xr-x 9 samik samik 4.0K Sep 5 18:07 15_oneVsRest_a54f608345aa
drwxr-xr-x 3 samik samik 4.0K Sep 5 18:07 16_idxToStr_0def5421e41c
The test code is available in my fork: https://github.com/samikrc/spark-ml-serving
If a pipeline has more than 9 stages, the saved stages cannot be loaded. I think the code adds a single digit suffix for the path. Please check out the logs below.
However, the files are there, as shown below:
The test code is available in my fork: https://github.com/samikrc/spark-ml-serving