Host data in S3 and use DuckDB for data ingest#67
Conversation
…versions for heatmap-client
…kspace, working on bumping version from 2021 to 2024
… DOM with new leptos version
…ve database connection
…fore processing data
…urrent request to finish before rendering first request
…nt any other data from loading in that request
d53db50 to
dae5456
Compare
kalebvonburris
left a comment
kalebvonburris
left a comment
There was a problem hiding this comment.
Mostly nitpicks. I'd definitely double check with more Clippy groups on and cargo fmt.
| leptos::task::spawn_local(load_data_async( | ||
| self.event_loop_proxy.clone(), | ||
| self.active_requests, | ||
| self.set_active_requests, | ||
| self.connection.clone(), | ||
| self.ingest_queue.clone(), | ||
| self.ingest_flag.clone(), | ||
| self.ingest_filter.clone(), | ||
| )) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| async fn load_data_async( | ||
| event_loop_proxy: EventLoopProxy<UserMessage<'static>>, | ||
| active_requests: leptos::prelude::ReadSignal<u32>, | ||
| set_active_requests: leptos::prelude::WriteSignal<u32>, | ||
| connection: Rc<AsyncDuckDBConnection>, | ||
| ingest_queue: Rc<Mutex<VecDeque<String>>>, | ||
| ingest_flag: Rc<RefCell<AtomicBool>>, | ||
| ingest_filter: Rc<RefCell<Filter>>, | ||
| ) { |
There was a problem hiding this comment.
Could you not pass in a DataLoader? Or a &DataLoader?
There was a problem hiding this comment.
I need to mutate the data loader fields and load_data_async is called by a method implemented on DataLoader. I tried that initially but ran into a bunch of errors with trying to move self, I can revisit this though and see if I can make something work because the giant argument list is not my favorite!
There was a problem hiding this comment.
Trying to pass a reference to self into load_data_async like this:
leptos::task::spawn_local(load_data_async(self))
Gives this error:
borrowed data escapes outside of method
argument requires that `'1` must outlive `'static`
I briefly tried wrapping self in an Rc but that didnt change anything. If you have ideas for a solution there I am open to hearing them otherwise I think I am stuck with lots of Rc wrapped arguments.
|
For what I mentioned before, if you add this to you #![warn(
clippy::all,
clippy::restriction,
clippy::pedantic,
clippy::nursery,
clippy::cargo,
)]Though, nursery and cargo can probably be removed. |
3 participants
Bumped project dependencies
Python ingest script moved to archive-heatmaps-v2 repository
Removed microservice that served data in favor of using DuckDB to read files hosted in an S3 bucket.