Skip to content

feat(*): stagger imports to avoid overloading#463

Open
yannisbarlas wants to merge 2 commits into
stagingfrom
453-import-out-of-memory
Open

feat(*): stagger imports to avoid overloading#463
yannisbarlas wants to merge 2 commits into
stagingfrom
453-import-out-of-memory

Conversation

@yannisbarlas

@yannisbarlas yannisbarlas commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Addresses the second part of #453

On the server:

  • there is a lock that prevents imports from happening when an import for that group is already in progress. this was set and released immediately, so in essence was not doing its job. fixed now.
  • updated the form-data dependency to make a node error go away
  • added a 30s timeout to the import api calls so that there is no possibility of them hanging
  • in biorxiv import, I added a check to stop recursion if a collection result is empty
  • restructured biorxiv full text import a bit, so that it saves in batches instead of accumulating all results
  • chunked requests to pubmed and added headers to the request (it was failing with status 400)

On the client:

  • dropped react-toastify in favor of using ant's toast notification
  • fixed the import subscription not firing
  • fixed the manuscripts table not being refreshed until reload after importing finishes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant