PKCS12 not found by Identity and Server

[WORMSTweaker]

Hello,
I'm trying to deploy Stasis, so far it's been.... quite difficult to say the least...
Right now, my issue is that the container can't find my .p12, but I confirmed the file is loaded in the container properly
What could be the issue?

Inside the container (stasis-identity), checking for the existence of the file:

demiourgos728@e6b1487bb463:/opt/docker$ ls
bin  config  lib  secrets
demiourgos728@e6b1487bb463:/opt/docker$ ls secrets/
identity.p12  identity-signature-key.jwk.json
demiourgos728@e6b1487bb463:/opt/docker$

Docker container logs:

identity-1              | SLF4J(W): A number (3) of logging calls during the initialization phase have been intercepted and are
identity-1              | SLF4J(W): now being replayed. These are subject to the filtering rules of the underlying logging system.
identity-1              | SLF4J(W): See also https://www.slf4j.org/codes.html#replay
identity-1              | [2024-11-26 16:32:23,179] [INFO ] [o.a.p.event.slf4j.Slf4jLogger]: Slf4jLogger started
identity-1              | Exception in thread "main" java.lang.ExceptionInInitializerError
identity-1              |       at stasis.identity.Main.main(Main.scala)
identity-1              | Caused by: java.io.FileNotFoundException: Store [/opt/docker/secrets/identity.p12] with type [PKCS12] was not found
identity-1              |       at stasis.layers.security.tls.EndpointContext$.loadStore(EndpointContext.scala:110)
identity-1              |       at stasis.layers.security.tls.EndpointContext$.$anonfun$apply$3(EndpointContext.scala:41)
identity-1              |       at scala.Option.map(Option.scala:242)
identity-1              |       at stasis.layers.security.tls.EndpointContext$.apply(EndpointContext.scala:40)
identity-1              |       at stasis.layers.security.tls.EndpointContext$.apply(EndpointContext.scala:32)
identity-1              |       at stasis.identity.service.Service$Config$.apply(Service.scala:291)
identity-1              |       at stasis.identity.service.Service.$init$(Service.scala:53)
identity-1              |       at stasis.identity.Main$.<clinit>(Main.scala:5)
identity-1              |       ... 1 more
server-1                | SLF4J(W): A number (3) of logging calls during the initialization phase have been intercepted and are
server-1                | SLF4J(W): now being replayed. These are subject to the filtering rules of the underlying logging system.
server-1                | SLF4J(W): See also https://www.slf4j.org/codes.html#replay
server-1                | [2024-11-26 16:32:23,693] [INFO ] [o.a.p.event.slf4j.Slf4jLogger]: Slf4jLogger started
server-1                | Exception in thread "main" java.lang.ExceptionInInitializerError
server-1                |       at stasis.server.Main.main(Main.scala)
server-1                | Caused by: java.io.FileNotFoundException: Store [/opt/docker/secrets/server.p12] with type [PKCS12] was not found
server-1                |       at stasis.layers.security.tls.EndpointContext$.loadStore(EndpointContext.scala:110)
server-1                |       at stasis.layers.security.tls.EndpointContext$.$anonfun$apply$3(EndpointContext.scala:41)
server-1                |       at scala.Option.map(Option.scala:242)
server-1                |       at stasis.layers.security.tls.EndpointContext$.apply(EndpointContext.scala:40)
server-1                |       at stasis.layers.security.tls.EndpointContext$.apply(EndpointContext.scala:32)
server-1                |       at stasis.server.service.Service$Config$.apply(Service.scala:547)
server-1                |       at stasis.server.service.Service.$init$(Service.scala:74)
server-1                |       at stasis.server.Main$.<clinit>(Main.scala:5)
server-1                |       ... 1 more

Stasis page (not loading probably because it can't find the p12 file):

Unrelated to this issue, Stasis documentation and deployment instructions are a bit confusing and lack-luster in some area, and I'm not even sure I set things up properly or even generated the keys as wanted by Stasis. Some more details could be added with examples.

In any case, thank you in advance for the support

[sndnv]

Hi,

Thanks for trying it out!

Let me address the documentation/setup issues first and then we can dig into the two specific problems.

Difficult setup and documentation

It most definitely is a pain to configure everything and get things going the first time; there are a lot of moving pieces and they all have their own security considerations. It would be great if you could remember some specific examples of what went wrong (or was difficult/not obvious) so that I can either update the documentation, the automation or (if possible) simplify things.

As for the documentation, could you point me to the parts that were not as expected or confusing; for sure I've made many assumptions when writing the docs so it would be great if they can be improved.

Also, did you have a look at the dev and prod docs? It's not mentioned anywhere but I would suggest to go through the dev setup first as it includes a lot of automation around generating secrets, artifacts, preparing the services and running some basic tests to ensure the setup is working.

Missing certificates

It's hard to tell the exact reason at this point but it is most likely a volume mount or a permissions issue; I've had this happen many times 😅. Could you try to list the files in the container but with a few more details- ls -aln secrets/; if you could do the same for the directory on the host where those files are, that would also be great (make sure to hide/remove any personal information, if it shows up ;) ).

I'm looking for issues with either the user(s) or with the permissions of the file. Also, could you provide the following info:

• Host OS - is it macos, linux, windows?
• Stasis version - I assume it's the latest but just to be clear
• Container management - are you using docker, podman or something else?
• Volume mounts - how are you mounting the volumes/files? like this or in some other way?

Page not found

I haven't seen that specific flutter/js error before 🤔. Could you share the exact URL that is causing this and what (if anything) happened before you saw the 404? As in, did you get redirected to this page as part of the login process or is it the first/only thing that you see.

It would also help to know what you have configured and on which port. For example, in the dev compose file the identity service is using ports 10000 and 10001, the identity-ui service is using 8080. The server has a bunch of ports and finally server-ui uses 9090.

[WORMSTweaker]

Hi, thanks for the quick response!

Difficult setup and documentation

So since I tried to configure it today, things are still fresh and I can point specific examples! I did go through the prod documentation but not the dev one, that's good to know and I'll go through it when I'll have some time (expect me to talk about things that may be already answered in the dev doc!)

• Environment files are provided (and wow there's a lot), but unless you go with the variable names, it is not completely clear which is what, what does what, and what is optional/necessary
• Still on the env files, I feel that some files could be united in a single .env files (ie: db-identity-exporter.env.example and db-server-exporter.env.example ), more of a potential improvement rather than a complain
• Provide TLS certificates and signature keys: Figured out that you can generate those using the python script, but it does require a CA certificate and its private key, and it could good to either generate those via the same script (since, well, might as well do those at the same time) or provide the commands to generate the CA certificate with the expected parameters for Stasis
• Enable bootstrap: Admittedly I should have taken less time to figure that the documentation was talking about the STASIS_IDENTITY_BOOTSTRAP_ENABLED and STASIS_SERVER_SERVICE_BOOTSTRAP_ENABLED env variables in the docker-compose file, but maybe this could be clarified to avoid future confusions
• PEKKO_HTTP_CORS_ALLOWED_ORIGINS: More of a question than a feedback, would setting this as 0.0.0.0 be considered valid? Or does it need to be the domain/server address for things to work properly?

I also checked out the Wiki but it's a bit empty, or at least I didn't find information that could help me

Missing certificates

Sure! Here's the output on the host:

/Stasis$ ls -aln secrets/
total 66
drwxr-xr-x 2 1000 1000   19 Nov 26 15:20 .
drwxr-xr-x 7 1000 1000    8 Nov 26 17:32 ..
-rw-r--r-- 1 1000 1000   81 Nov 26 14:29 db-identity-exporter.env
-rw-r--r-- 1 1000 1000   79 Nov 26 14:29 db-identity.env
-rw-r--r-- 1 1000 1000   81 Nov 26 14:29 db-server-exporter.env
-rw-r--r-- 1 1000 1000   79 Nov 26 14:29 db-server.env
-rw-r--r-- 1 1000 1000    0 Nov 26 15:20 identity-signature-key.jwk.json
-rw-r--r-- 1 1000 1000   82 Nov 26 15:00 identity-ui.env
-rw-r--r-- 1 1000 1000 1103 Nov 26 15:00 identity.bootstrap.env
-rw-r--r-- 1 1000 1000 1350 Nov 26 15:20 identity.cert.pem
-rw-r--r-- 1 1000 1000  135 Nov 26 14:28 identity.env
-rw------- 1 1000 1000 3268 Nov 26 15:20 identity.key.pem
-rw------- 1 1000 1000 3875 Nov 26 15:20 identity.p12
-rw-r--r-- 1 1000 1000   78 Nov 26 15:00 server-ui.env
-rw-r--r-- 1 1000 1000  486 Nov 26 15:00 server.bootstrap.env
-rw-r--r-- 1 1000 1000 1350 Nov 26 15:19 server.cert.pem
-rw-r--r-- 1 1000 1000  657 Nov 26 15:00 server.env
-rw------- 1 1000 1000 3268 Nov 26 15:19 server.key.pem
-rw------- 1 1000 1000 3875 Nov 26 15:19 server.p12

and in the container (stasis-server to be exact):

/opt/docker$ ls -aln secrets/
total 21
drwxr-xr-x 2    0    0 4096 Nov 26 16:32 .
drwxr-xr-x 1 1001    0 4096 Nov 26 16:32 ..
-rw------- 1 1000 1000 3875 Nov 26 14:20 identity.p12
-rw------- 1 1000 1000 3875 Nov 26 14:19 server.p12

The file not found error happen in both the identity and server container so I assume the issue is the same for both, I don't see any issues myself in those permissions but it might be because it's lacking the +x permission? Or actually it might be because it's not 644 (-rw-r--r--) like the other certificates and if it's that whoops, should have checked
UPDATE: Yup, that was it... I should have known 🤦
Although it's complaining about an invalid UUID string now, which, well indeed that not really a valid UUID formatted string ahah

SLF4J(W): A number (3) of logging calls during the initialization phase have been intercepted and are
server-1                | SLF4J(W): now being replayed. These are subject to the filtering rules of the underlying logging system.
server-1                | SLF4J(W): See also https://www.slf4j.org/codes.html#replay
server-1                | [2024-11-27 00:45:08,815] [INFO ] [o.a.p.event.slf4j.Slf4jLogger]: Slf4jLogger started
server-1                | [2024-11-27 00:45:10,589] [ERROR] [stasis.server.Main$]: Service startup failed: [IllegalArgumentException - Invalid UUID string: stasis_wormstweaker]
server-1                | [2024-11-27 00:45:10,594] [INFO ] [stasis.server.Main$]: Service stopping...
server-1                | [2024-11-27 00:45:10,742] [INFO ] [o.a.p.actor.CoordinatedShutdown]: Running CoordinatedShutdown with reason [ActorSystemTerminateReason]
server-1                | [2024-11-27 00:45:10,957] [INFO ] [stasis.server.Main$]: Service stopping...

For the host info, here you go:

• Host OS - Linux 6.11.5, Debian 12 Bookworm
• Stasis version - It should be the latest yes, 1.2.0
• Container management - Docker v27.3.1
• Volume mounts - I'm using the provided docker-compose.yml so yes, like this

Page not found

This happens when trying to access identity.stasis.internal:42600 (or, to be more exact, the edited URL that I set up, because in a local environment, without changing any of the *.stasis.internal URLs, I would get sent to that and of course this domain doesn't exist... Trying it out with an ip on my local network gives me CORS issues, so for now I tried setting it up on my public domain. I can give you the full docker-compose in private if you want to see for yourself all of what I edited), this happens after a redirection to the /login/callback page, which not long after gives me that 404

Since I'm using the prod docker-compose, and have yet to edit the ports, you can base yourself on this file for the ports I'm using

The only things modified in that compose are the values cited earlier, the values required to be changed and detailed in the deployment process, and commenting out prometheus and grafana as those were spamming the output, which wasn't helping with debug

That should be all!

If you need more information or details, feel free to ask and I'll be happy to provide them to you, and again thank you for the fast answer

[sndnv]

Hi, that's all very useful feedback, thanks! I'll be updating the docs in the coming days/weeks.

Environment files - I can see how the .env files can be very confusing, with no explanation at all 😅 those will be getting some updates as well. What you could do for your deployment is to check the dev compose file and see what values are set there for the env vars in the .env files. Hopefully, this will also help with your Invalid UUID string issue; it's probably the STASIS_SERVER_AUTHENTICATORS_INSTANCE_CLIENT_ID parameter (the code expects a UUID but I see no good reason for that; will definitely check what's going on here).

As a side note, the DB-related .env files are split because in principle you don't have to use identity as your OAuth provider so you might want to omit that service; also, I prefer having the identity and server databases completely separate (it's a reason but not necessarily a good one 😂).

TLS certificates and signature keys - indeed, there's not enough information; I don't remember exactly how I generated my root CA as it wasn't related to this project but a quick search gives me this (and it looks about right):

openssl genrsa -aes256 -out rootCA.key 4096 # for the private key
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt # for the cert

As soon as I have some time to dig into it and confirm that it is correct, I'll update the docs to include this info.

Enable bootstrap - yeah, docs need to be improved; the gist is that the bootstrap process creates enough objects in the database to be able to log in and start managing the services

CORS - I'm not sure if 0.0.0.0 is going to work as I didn't try it myself; if you don't intend to make your setup available to the outside world, it should be fine to just set it to * (it's already the case for the dev setup; don't forget to configure nginx as well).

Page not found - it looks like there's a mismatch between what is coming into the bootstrap/identity.conf config (via the .env files) and what is in the redirect URIs in the compose file. Also, port 42600 is for the server-ui but you mention identity so maybe that's the cause. This bit definitely needs to be better documented as I don't remember all of the details but the redirect URIs for these two clients should match the respective URIs here and here (it looks like those env vars should not be in the .env files anyway).

---

All in all, the production setup has a steep learning curve so you will probably experience more issues there that aren't properly documented. Do let me know if anything else comes up!

The development setup might be a bit easier as it is more automated but you will have to replace the docker images in the compose file for sure (unless you want to setup all the dev tools to be able to build the images yourself).

One more note, regarding the post-bootstrap setup, the docs only mention the scripts that can be used but they are not particularly straightforward. Since that documentation was written, the server-ui was implemented so that should be easier to use (with Chrome or Chromium-based browsers; seems like flutter support for Firefox/Safari has gotten worse recently).

[WORMSTweaker]

Env files - Okay, I checked all of the provided .env files fields, and well... None of them match the ones present in the compose file
STASIS_SERVER_AUTHENTICATORS_INSTANCE_CLIENT_ID was set by me, since I didn't know it expected an UUID I set it to a random string, will unset it and see how it goes. Un-setting it leads to the same error I gave earlier except that it's just an empty string... So yeah something is up

TLS certificates and signature keys - Okay, well that's pretty much what I found and used, except that I used another type of key algorithm, shouldn't be too problematic (I hope)

CORS - Changed back all of the *.stasis.internal field to the local ip of the machine and corresponding port, and tested with * on my local network, CORS issues still happening
Tried setting it to the machine IP, same issue, it might be because self-signed certificates are usually not very happy in a local network environment (unless you import them in your browser and devices, but I think that's quite problematic for Stasis case...)
SSL Certificates are complicated and hard, I don't like them, but alas they exist for a good reason 😅

Page not found - Well seems to be another misconfiguration from my part again, I just noticed that those files were not present and instead were created by docker as folders... I think my main issue in all of this was that I used the compose file outside the cloned repo, so many expected files aren't present when by default they should be (could be a doc improvement to explain that cloning the repo is expected 😅 )
But still, after putting them in the correct place, with * for the CORS, 404 using the public domain 🤔 At least I'm not getting it on my local network so that's good?

I might retry the whole process using the dev setup and see how it goes, maybe I'll have more luck with it
I'll keep in mind the info about Firefox compatibility and will try more stuff on Chrome instead

[sndnv]

Sounds like there is some slow and painful progress 😅

Env files - specifically about STASIS_SERVER_AUTHENTICATORS_INSTANCE_CLIENT_ID, this one is required and it has to be a UUID at the moment (a bug that needs fixing); this is the UUID of one of the clients created via this bootstrap config. What you could do is only deploy the identity services - identity, identity-ui and db-identity (and the exporter, if you care about the metrics). That should be relatively straightforward and in the end you get the ability to see/manage the clients via the UI. From there you can see for sure what redirect URIs are set and what the client IDs are.

At this point, go ahead and disable password derivation, it's another layer of complexity and headaches 😅

• for server - set STASIS_SERVER_BOOTSTRAP_DEVICES_PARAMETERS_SECRETS_DERIVATION_AUTHENTICATION_ENABLED to false
• for server-ui - set SERVER_UI_PASSWORD_DERIVATION_ENABLED to false
• for identity-ui - set IDENTITY_UI_PASSWORD_DERIVATION_ENABLED to false

(Just for information, this feature enables additional hashing of user passwords before they leave the device/browser; the cost of the improved security is that it makes debugging authentication issues more difficult)

TLS certificates and signature keys - yeah, this part is always a pain and we have to suffer through it, unfortunately. That's why I went for the root CA approach; I have it set as trusted on my devices so no need to mess around with certs too much. If you decide to try the Android client, it's the only way to get it to work with self-signed certs anyway.

CORS - the * should be the most permissive option so I suspect it might be the untrusted certs that are tripping it up. If you don't have the root CA setup as trusted, what you could do is hit each endpoint manually in the browser and "accept" the self-signed certs/warnings. Then, do the login process from the beginning and it should work; the endpoints to hit would be https://<identity-ip-or-name>:42100, https://<identity-UI-ip-or-name>:42200 and https://<server-UI-ip-or-name>:42600 (and eventually, after login, the server APIs will need to be hit on ports 42300 and 42302 but at least at that point the basic login process should work).

To be honest, I didn't try very hard to get CORS to work properly with my local setup and I ended up using * everywhere 😂 It looks like I'll have to spend the time to configure it correctly and document it but I guess that doesn't really help you right now 😅.

404 - Did I get it right? On your local network, you don't get any CORS/404 issues but with your public domain, it's still failing? Is it maybe the redirect URIs that don't match the public domain names/ports?

---

In the end, doing this setup is definitely too complex as it is right now. And you are correct, the repo shouldn't have to be cloned to get all the config/example files so that's another point to add to the (now very long) list of improvements.

I do appreciate you trying to get this going! Hopefully you won't get too frustrated/discouraged so remember that you are trying to solve at least 3 very challenging problems - TLS, OAuth and CORS 😄 ... at the same time.

[WORMSTweaker]

Okay, so some updates

Env files - Okay! Disabled password derivation, and the unnecessary containers (db-identity, identity, and identity-ui are the only ones left)
Before keeping just those, for the UUID issue, I tried cheating a little and generated a random UUID to see how it would react with all the services enabled
And well, it (half) loaded! Printed out my build and config, but hanged afterward on an "Invalid UUID string", at least that's proof that it sorta works, just need a valid UUID, which I've yet to get because of the domain/CORS issues

server-1       | Exception in thread "main" java.lang.ExceptionInInitializerError
server-1       |        at stasis.server.Main.main(Main.scala)
server-1       | Caused by: java.lang.IllegalArgumentException: Invalid UUID string: 
server-1       |        at java.base/java.util.UUID.fromString1(UUID.java:282)
server-1       |        at java.base/java.util.UUID.fromString(UUID.java:260)
server-1       |        at stasis.server.service.Bootstrap$.scheduleFromConfig(Bootstrap.scala:158)
server-1       |        at stasis.server.service.Bootstrap$.$anonfun$run$5(Bootstrap.scala:63)
server-1       |        at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
server-1       |        at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
server-1       |        at scala.collection.convert.JavaCollectionWrappers$JListWrapper.map(JavaCollectionWrappers.scala:138)
server-1       |        at stasis.server.service.Bootstrap$.run(Bootstrap.scala:63)
server-1       |        at stasis.server.service.Service.$init$(Service.scala:452)
server-1       |        at stasis.server.Main$.<clinit>(Main.scala:5)
server-1       |        ... 1 more

It also allowed the identity service to proceed a bit further, just enough to try and read the JWK key file (mind you, this file is empty, am I supposed to generate it? Examples online gives me an idea of how this file should be filled but I'm unsure, and since I see in the stasis/layers that there's functions to generate it, maybe something is not right)

identity-1     | SLF4J(W): A number (3) of logging calls during the initialization phase have been intercepted and are
identity-1     | SLF4J(W): now being replayed. These are subject to the filtering rules of the underlying logging system.
identity-1     | SLF4J(W): See also https://www.slf4j.org/codes.html#replay
identity-1     | [2024-11-28 09:53:16,944] [INFO ] [o.a.p.event.slf4j.Slf4jLogger]: Slf4jLogger started
identity-1     | [2024-11-28 09:53:18,938] [ERROR] [stasis.identity.Main$]: Service startup failed: [JoseException - Parsing error: org.jose4j.json.internal.json_simple.parser.ParseException: Unexpected token END OF FILE at position 0.]
identity-1     | [2024-11-28 09:53:18,943] [INFO ] [stasis.identity.Main$]: Identity service stopping...
identity-1     | [2024-11-28 09:53:19,003] [INFO ] [o.a.p.actor.CoordinatedShutdown]: Running CoordinatedShutdown with reason [ActorSystemTerminateReason]
identity-1     | [2024-11-28 09:53:19,126] [INFO ] [stasis.identity.Main$]: Identity service stopping...

Trying to access the server service like this with the placeholder UUID, as expected it can't load the JWK

server-1       | [2024-11-28 10:00:33,796] [ERROR] [s.l.s.keys.RemoteKeyProvider]: Failed to load keys from JWKs endpoint [https://identity:42100/jwks/jwks.json]: [StreamTcpException - Tcp command [Connect(identity/<unresolved>:42100,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused]

TLS certificates and signature keys - Agreed, always a pain... Maybe it could be interesting to explore things like Let's Encrypt certbot, or since we're in a container environment, things that implement certbot automatically like LinuxServer docker-swag? Although that would be for a production environment more than a local/testing one

CORS - Since I'm using the same certificate for all of the services (for now, might change it later), I double checked and my browser didn't ask for me to accept it again when visiting all the services on the corresponding ports, so things should be good on that front, yet I'm still getting a CORS error 🤔
I think it's just really unhappy to be redirected from, for example, https://192.168.1.26:42200 to https://192.168.1.26:42100, that change of port might not be allowed in CORS guidelines

404 - Oh nope! I think you got it backward, on my local network I get the CORS issues (like the example given just above), on my public domain I get the weird 404-but-not-really-404 flutter error
I think it might be better to ignore the public domain issue for now, because the way I set it up is quite janky to say the least
Basically, since my domain is behind Clouflare, I'm using their cloudflared service, which allows me to expose some of my local services as subdomains of my main domain, but who knows if this doesn't mess with Stasis redirections or some other things

---

Yeah, it's a bit complex, sorry for putting so much work on your plate, it's quite a lot all at once 😅

I'll try not to be discouraged! Just taking it slow and trying to resolve one issue at a time, and reporting my findings/issues as I progress

[sndnv]

Env files - that exception is from the bootstrap process, in this case it's trying to create one of the schedules and it's getting an empty string for the UUID. You can remove those from the server's bootstrap file but some of the other stuff is needed. For example, in the nodes section, you have to specify the node IDs (any UUID will do, as long as they are all unique) and the directories where the data is to be stored (kinda important 😅). You can tell which particular object type is failing by the function - stasis.server.service.Bootstrap$.scheduleFromConfig; in this case, it's the schedule.

JWKs - that is another huge oversight 😨 indeed, there's no (easy) way to generate the production ones (to be fixed) but for testing you can set STASIS_IDENTITY_TOKENS_ACCESS_SIGNATURE_KEY_TYPE to generated. Then identity will generate a new set on startup; the drawback here is that if you login and then restart the services, a new set will be generated so all old tokens will be invalid (and you have to login again). Fine for testing, not great for production.

TLS - nice! I'll see if I can integrate that with the prod compose file

CORS - that's weird though; an * in both ui services and both nginx containers should allow anything 🤔 What is the exact error message that your browser giving you?

404 - if we focus on identity and identity-ui for now, can you double-check that this and this have the same values and are matching with whatever "name' the services use. For example, if you want to access the identity and identity-ui at 192.168.1.26, then those URIs should both say https://192.168.1.26:42200/login/callback.

I'm not very familiar with cloudflare but what I find a bit odd in that screenshot is that entries 13 and 15 both say idstasis (which I assume is for identity or the identity-ui) but the port is 42600; in the compose file, that's the port for server-ui.

---

No worries, it's quite a lot to fix but they are valid issues that need fixing 😁

[WORMSTweaker]

Env files - Aaaah right, that makes sense since I didn't set the values in server.bootstrap.env, guess I'll comment out the schedules for now, since I'm not even going to use the server service for testing, although I'm going to set some UUIDs for the nodes, just in prevision for later... Do you think a simple "stasis/" for the parent-directory will be enough? 😅

JWKs - Well, as long as they are generated once, and that you set it back to stored, things should be fine right? Could be part of the bootstrap process in the docs, generate once run forever (and refresh the token once in while)
Set it to generated, identity-signature-key.jwk.json didn't get populated though.. 🤔 Maybe I did something wrong? Or it's not starting properly?

---

I don't know which one of those changes did it, but it's quite unhappy with my current database, clearing it and starting fresh seems to produce the same issue, it's complaining about duplicate keys and already existing tables, see the attached logs
stasis_logs.txt before clear
stasis_logs_db.txt after clear

CORS - Those previous logs made me realize that my NGINX_SERVER_NAME probably wasn't set correctly since it was still on my public domain, but changing it to the local IP didn't seem to change anything... Here's what both browser tell me, which just seem to be a SSL issue... Quite vague...
Firefox

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://192.168.1.26:42100/oauth/authorization?response_type=code&client_id=c7b3c6e0-d5de-46d1-a70a-2f855fa7c7fc&redirect_uri=https%3A%2F%2F192.168.1.26%3A42200%2Flogin%2Fcallback&code_challenge=8DUqDRrsx6U3Rr0ZyZQ8wvKqYcb5p-AjVfRyUlO7rYw&code_challenge_method=S256&state=72bw5qsZdrP6ZoCKew8n2yBMvjQo64j3GGGLS2HdGSbI0TlX9z2EXEMXYXV9NhJo&scope=urn%3Astasis%3Aidentity%3Aaudience%3Amanage-identity. (Reason: CORS request did not succeed). Status code: (null).

Chrome(ium)

An SSL certificate error occurred when fetching the script.
flutter.js:3 Exception while loading service worker: SecurityError: Failed to register a ServiceWorker for scope ('https://192.168.1.26:42200/') with script ('https://192.168.1.26:42200/flutter_service_worker.js?v='): An SSL certificate error occurred when fetching the script.
GET https://192.168.1.26:42100/oauth/authorization?response_type=code&client_id=c7b3c6e0-d5de-46d1-a70a-2f855fa7c7fc&redirect_uri=https%3A%2F%2F192.168.1.26%3A42200%2Flogin%2Fcallback&code_challenge=gBf-WKCN2-VSd4yU2m8rW0KhqmttIcjjanPcqaW2Aqg&code_challenge_method=S256&state=dmearNEBRtYzfWU7EvtUUzj0J21DQMysRIGE8SrtxKzrTm8z6Tya90b3a6DQu9XT&scope=urn%3Astasis%3Aidentity%3Aaudience%3Amanage-identity net::ERR_CONNECTION_REFUSED

404 - If the file take from the env, it should be pointing to that uri, but just to be safe I'll "hardcode" it in the config
I'm not sure how I could be checking if the env value is applied correctly for the bootstrap file, since going in the container and opening the file just shows it has the ${placeholder} values, which is expected for me
Changing the ${placeholder} value directly didn't seem to change anything sadly, and considering we're dropping the public domain idea, I'm not getting the 404 anymore

Okay maybe showing you this will clear your confusion, basically the default server.stasis.internal and identity.stasis.internal are what are being replaced, each sub-domain AND path redirects to a specific port!

      SERVER_UI_SERVER_API: "https://srvstasis.[redacted_domain]" # server.stasis.internal:42300
      SERVER_UI_BOOTSTRAP_API: "https://srvstasis.[redacted_domain]/bootstrap" # server.stasis.internal:42302
      SERVER_UI_AUTHORIZATION_ENDPOINT: "https://idstasis.[redacted_domain]/endpoint/login/authorize" # identity.stasis.internal:42200/login/authorize
      SERVER_UI_TOKEN_ENDPOINT: "https://idstasis.[redacted_domain]/identity/oauth/token" # identity.stasis.internal:42100/oauth/token
      SERVER_UI_REDIRECT_URI: "https://idstasis.[redacted_domain]/redirect/login/callback" # identity.stasis.internal:42600/login/callback

Think of it as some sort of redirection from the sub-domain & path to a local port, via a tunnel, it's an interesting service and is relatively simple to set up

---

We're getting there little by little, plus this troubleshooting will help streamlining the deployment (and even dev!) process
I think Stasis is an interesting piece of software, and the backup solution I've been searching for a while, so if I can get it running that would be awesome!

[sndnv]

Env files - yeah, you can point it to some directory and it should create them; even something like /tmp/stasis/store-a and /tmp/statis/store-b should work.

JWKs - using generate means it's only generated in memory but there's no reason not to have a 3rd option like generate-if-not-exists and then write it for future use =]

DB - the error seems to be correct actually; it first says it created your user, then it tries to create it again:

identity-1     | [2024-11-29 11:34:12,862] [INFO ] [stasis.identity.Main$]: Resource owner [wormstweaker] added
identity-1     | [2024-11-29 11:34:12,869] [ERROR] [stasis.identity.Main$]: Failed to add ... ERROR: duplicate key ... "
identity-1     |   Detail: Key ("USERNAME")=(wormstweaker) already exists.]

Is it possible that the env vars got mixed up? At least by default, the identity bootstrap config has two users (one for you as the admin and another one for the server to manage devices/credentials) but those should have unique names. If that all looks ok, then can you enable debug logging by setting STASIS_IDENTITY_LOGLEVEL to DEBUG to see what it is actually trying to do there.

I'm not sure if it's mentioned anywhere in the docs but as soon as the bootstrap succeeds, you should set STASIS_IDENTITY_BOOTSTRAP_ENABLED back to false. Otherwise it will try again on the next restart.

CORS / 404 - this thing... it looks like identity on port 42100 is refusing the connection this time (chromium has a better error message here, I guess); what I get from the this MDN page is:

The error is not directly related to CORS, but is a fundamental network error of some kind.

So let's try to get identity to start properly, do the bootstrap and load those JWKs 😅. Then we can untangle this CORS/404 stuff.

[WORMSTweaker]

Env - Argh, undocumented env variables strike again, I set STASIS_IDENTITY_BOOTSTRAP_OWNERS_DEFAULT_USER and STASIS_IDENTITY_BOOTSTRAP_OWNERS_SERVER_MANAGEMENT_USER to the same user, that's on me tbh I shouldn't have given them the same name
Clearing the db (because it was complaining that the API table already existed) and starting it, now the identity server seems to be up and running! Will set the bootstrap to false before I forget about it
Restarting the service a few times, no issues anymore so hey that's a win!

JWKs - Oh, yeah generate kinda sounds like it would write it somewhere, maybe instead of adding generate-if-not-exists you could rename the current behavior to memory, and make it so generate actually writes it? Then you switch to stored if you want to use the newly made file

CORS / 404 - Progress on that front too, since the identity server is running! Still giving me a CORS error on firefox, but on Chrome, I'm getting an invalid_request

{
  "error": "invalid_request",
  "error_description": "The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.",
  "state": "QGVhkilmjlPrt39EWQkgTnu9LJHoi4a1cL924TNB8XqDo9VbQ0vsqmqVAPzZkgJ7"
}

From what I'm reading in the identity service logs, it might be because STASIS_IDENTITY_BOOTSTRAP_CLIENTS_IDENTITY_UI_REDIRECT_URI wasn't set, but again quite unsure because setting it to the expected URL doesn't change anything, same thing when hardcoding the URL in identity.conf 🤔 Maybe I'm targeting the wrong value

identity-1     | [2024-11-29 15:43:32,017] [WARN ] [s.i.a.o.PkceAuthorizationCodeGrant]: Redirect URI [] for client [c7b3c6e0-d5de-46d1-a70a-2f855fa7c7fc] did not match URI provided in request: [Some(https://192.168.1.26:42200/login/callback)]
identity-1     | [2024-11-29 15:43:32,782] [WARN ] [s.i.a.o.PkceAuthorizationCodeGrant]: Redirect URI [] for client [c7b3c6e0-d5de-46d1-a70a-2f855fa7c7fc] did not match URI provided in request: [Some(https://192.168.1.26:42200/login/callback)]

We're getting there!

[sndnv]

Ah yes, progress! 🥳 What that last error is saying is that client c7b3c6e0-d5de-46d1-a70a-2f855fa7c7fc is configured to have an empty string for the redirect URI. In the request itself, a URI is being provided so it looks like this value in the bootstrap config was empty when the bootstrap process was done. Even if you set it afterwards, if the bootstrap is not performed again, nothing will happen.

If that's the issue, then you have two options - wipe the DB and bootstrap again or go in the DB and set the value manually for that client.

[WORMSTweaker]

Just to be clean, I wiped the database again, enabled bootstrapping with the URL set correctly, and success! Finally logged and and accessed Stasis identity ui! 🥳

Although, trying to access any other tab results in a [WARN ] [stasis.identity.api.Manage]: Rejecting [GET] request for [https://192.168.1.26:42100/manage/tokens] with invalid credentials from [unknown]: [AuthenticationFailure - Resource owner [] not found] warning, and me getting kicked out of the UI 😅

My first guess was that the env variable was being messed up by special characters in my password, but that can't be the case because entering an incorrect password when logging in doesn't let me in, so the issue must be somewhere else 🤔

Now that I got the identity UI working, I suppose the server is next? Because I don't think I can do any backups as-is with Stasis with half of the backend missing 😂

[sndnv]

Almost there 😁 It's refusing the request because the STASIS_IDENTITY_BOOTSTRAP_OWNERS_DEFAULT_USER_ID env var was probably empty during bootstrap. The code loading these items should be checking if it's empty and then not set it but that's not the case right now. Setting a value here will help with the server setup, so go ahead give it some random UUID and set this one to the same value.

---

Yeah, it's the server that does all the work :D identity is just the OAuth provider. To make it easier for now:

• this you can set to some random value
• the rest of the bootstrap values we discussed earlier but, basically, give the nodes and schedules some random UUIDs and it should be fine 🤞
• SERVER_UI_CLIENT_ID here should be the same UUID as STASIS_IDENTITY_BOOTSTRAP_CLIENTS_SERVER_UI_CLIENT_ID here
• these env vars should be the same UUID and secret that are configured for this client.
• this and this don't have to be valid URIs but the standard requires them to be set 🤷‍♂️ I have them pointing to localhost.
• STASIS_SERVER_AUTHENTICATORS_NODES_AUDIENCE here should be the same UUID as the server's node client here. Long story short, at the data storage level, the server(s) and clients are able to form a P2P network to exchange data so each node requires credentials. Currently, only the clients are allowed to send data to the server so this bit is only used for checking the JWTs that the clients send.
• and (hopefully) finally, these user credentials should be the same as these.

I probably missed something, so just let me know 😅 the next "milestone" would be to be able to log into server-ui and add/remove users, devices, etc.