Add USE_HYPER helper-bot transfers, HyperUpload, wzgram, and faster H… - #1
Open
dianemill16 wants to merge 7 commits into
Open
Add USE_HYPER helper-bot transfers, HyperUpload, wzgram, and faster H…#1dianemill16 wants to merge 7 commits into
dianemill16 wants to merge 7 commits into
Conversation
…yperDL - Add USE_HYPER config: when enabled (with HELPER_TOKENS + LEECH_DUMP_CHAT), helper bots accelerate both Telegram downloads and uploads - HyperUpload: rotate the least-loaded helper bot per file with load tracking; files are still sent strictly one-by-one in natural sort order so series episodes and split parts always upload in sequence - Fall back to the default client automatically if a helper bot cannot post in the upload chat; skip inter-file flood delay when rotating bots - Gate existing HyperDL on USE_HYPER; show (HyperDL)/(HyperUL) in status - Optimize HyperDL: parts now write directly at their byte offset into a single preallocated file (removes per-part temp files and the full second concatenation pass); per-part progress rollback on retry instead of resetting the whole task's progress - Raise max_concurrent_transmissions for helper bots (HYPER_THREADS, default 8) for higher per-client throughput - Integrate wzgram (github.com/rjriajul/wzgram) as the pyrogram library: drop-in replacement with Rust crypto (warpcrypto), replacing pyroblack + tgcrypto-pyroblack; update stats display accordingly - Add USE_HYPER to bot settings toggles, sample_config and help texts Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF
Schedule the coroutine on bot_loop so bot commands are actually registered with Telegram at startup and after session restarts. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF
Helper bots now pre-upload several files concurrently (client.save_file) while a single sender posts each finished file with raw SendMedia in strict natsorted order, so aggregate upload speed scales with the number of HELPER_TOKENS while series episodes and split parts still land in sequence. Falls back to the classic per-file upload for >2GB files (premium session) and on any raw-send failure. Also report the real Telegram library (wzgram) in the status engine tag instead of the hardcoded 'Pyro' label. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF
- Add ARIA2_MAX_DL_SPEED config to cap aria2's aggregate download rate (--max-overall-download-limit) so fast mirrors can't saturate the VPS disk/NIC and freeze the machine - Use --file-allocation=none to avoid preallocation stalls - Answer status callbacks through a QueryIdInvalid-safe helper so stale button presses after a stall stop spamming the log Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF
The ordered sender assigned and cleaned up self._up_path, which prefetch workers concurrently reassign under _prep_lock. After posting a file the sender's cleanup could delete the NEXT episode's file (whatever self._up_path pointed to by then), making its pre-upload fail and silently dropping episodes from the sequence (e.g. E18, E20, E22). The sender now works exclusively on its own local path; the classic fallback holds _prep_lock while it uses self._up_path; and a vanished file before pre-upload is logged loudly instead of skipped silently. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF
HyperDL (download):
- Fix off-by-one chunk count that zero-filled part tails when a range
ended exactly on a 1MiB boundary (silent file corruption)
- Propagate cancellation into part tasks (stop_transmission now sets the
cancel event instead of only stopping the progress reporter)
- Bound per-chunk retries and rebuild dead pooled media sessions on
outer retry; resume mid-stream instead of restarting from part 1
- Re-raise real errors so the plain-download fallback actually runs
- Honor FloodWait in file-ref fetch; lazy cleaner task; stop
half-started sessions on retry
HyperUL (upload):
- Backpressure: cap pre-uploaded-but-unposted files at slots+2
- Stop pre-uploading once helper posting is disabled mid-task
- FloodWaits no longer burn retry attempts (posting a pre-uploaded file
waits out floods instead of re-uploading); sleeps are cancellable
- Progress accounting: roll back only own bytes on retries so
concurrent workers' progress isn't erased (no more >100% or negative)
- Raw-send fallback recomputes premium flag and balances helper loads
- Skip md5 hashing unless the caption template uses {md5_hash}
- Wire up MEDIA_GROUP config (was hardcoded off)
Infra:
- Status overview: recognize wzgram engine tag, guard speed parsing
(fixes AttributeError crash on the Overview button during leech)
- Re-assert ARIA2_MAX_DL_SPEED over the DB aria2 options snapshot and
validate its format; aria2 RPC no longer listens on all interfaces
- Cache engine label at module load; keep set_bot_commands task
referenced with error logging; drop duplicate handler registration on
session restart; remove stale asyncio backport dep; pin wzgram
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF
The Restarting... notice was sent as a reply to the original /restart command, which may already be deleted (e.g. DELETE_LINKS). send_message then raised on None and returned the error text as a string, and restart_message.chat.id crashed confirm_restart after every client was stopped but before the re-exec, leaving the bot down until the host supervisor revived it. Fall back to the chat id for the notice, treat a string result as no message, and never let the .restartmsg write abort the re-exec. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Cb6XCXVFbzL77mWMhcGqPL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…yperDL
Claude-Session: https://claude.ai/code/session_01FZvodgJdwGsuvw1GEMzgjF