Skip to content

log metrics for metis uploads #1481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
amadeovezz wants to merge 1 commit into from
Closed

log metrics for metis uploads #1481

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions metis/db/migrations/029_add_upload_bandwidth_tracking.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Sequel.migration do
change do
alter_table(:uploads) do
add_column :started_at, DateTime
end
end
end
18 changes: 13 additions & 5 deletions metis/lib/models/upload.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,19 @@ def finish!
f.data_block = data_block
end
file.update(folder: folder, author: author, data_block: data_block)
Metis::DataBlockLedger.log_create(
file,
data_block,
author
)
Metis::DataBlockLedger.log_create(file, data_block, author)
Metis::DataBlockLedger.log_link(file, data_block, author)

if started_at
duration = Time.now - started_at.to_time
bps = duration > 0 ? (data_block.size / duration).round : 0
Metis.instance.logger.warn(
"upload_complete project=#{project_name} bucket=#{bucket.name} " \
"file=#{file_name} size_bytes=#{data_block.size} " \
"duration_seconds=#{'%.3f' % duration} bytes_per_sec=#{bps}"
)

@graft graft Mar 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems to just send this to stdout rather than to the ledger? Those logs are pretty ephemeral.

@amadeovezz amadeovezz Mar 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to an a upload_started_at column to the ledger model just for this use case. I thought you were just testing one off use cases that you would be manually monitoring?

If we want something query-able we will need a different strategy

@graft graft Mar 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought we were putting this in the ledger so we could monitor upload speeds in general, i can always time client-side for one-off use cases. It should be pretty straightforward to just send the same content to the ledger instead of stdout, no?

end

return file
end

Expand All @@ -90,6 +97,7 @@ def self.fetch(params)
current_byte_position: 0,
next_blob_size: -1,
next_blob_hash: '',
started_at: DateTime.now,
)
end
end
Expand Down
Loading