HRI task (now SM1-5)#447
Merged
Merged
Conversation
* Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <[email protected]>
* vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]>
* Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black
SM1-4 works correctly, however during SM4 it doesn't look at the seated guests
Collaborator
Author
|
This solves issue #418 |
Aldrich-Fernandes
added a commit
that referenced
this pull request
Jun 19, 2026
* Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <[email protected]> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <[email protected]> Co-authored-by: Ma'ayan Armony <[email protected]>
Fadi-Mostefai
added a commit
that referenced
this pull request
Jun 23, 2026
* HRI task (now SM1-5) (#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <[email protected]> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <[email protected]> Co-authored-by: Ma'ayan Armony <[email protected]> * Added REID fallback Also made minor changes to accomodate gpu * Test HRI on GPU * Updated HRI to say attributes Also added in a SafeGoToLocation and moved StartDoorSM to skills * Debugging HRI task * Added gpu requirements for lasr_vision_reid * Testing of HRI with GPU * Fixing bugs * Fixing of EyeTracker * Working eye tracker, yolo is bottleneck * Fully fixed eye tracker * Adding in host * Fixed introduce for host * Formatted files * Merge branch 'ros2-hri-sm6' into ros2-gpu-and-reid
Fadi-Mostefai
added a commit
that referenced
this pull request
Jun 26, 2026
* Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * WIP: Follow Person Experimental * Merge branch 'ros2-hri-testing' into ros2-hri-sm6 * WIP: Planning SM6 * update waitforperson to use blackboard data as well * WIP: debugging stuck in loop * ran black formatting * Merge branch 'ros2' into ros2-hri-sm6 * Fixed broken merge * WIP: Debugging tracking * testing tf_listener instead of amcl pose for robot pose * Fixed tracking * Fixed following navigation * Fixed inconsistant following logic * Fixed bugs related to handle_sigint and service * Seperated updateable navigation to prevent conflicts * Fixed merge error and removed duplicate file * Fixed blocking logic * Added debug config for rviz * Ported additional States * HRI task (now SM1-5) (#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <[email protected]> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <[email protected]> Co-authored-by: Ma'ayan Armony <[email protected]> * Added Follow Host to HRI * Fixed recovery states not being reached * Fixed looking at person while tracking * Fixed Detect_keypoints and started implementing place_bag * Added drop point detection * WIP: Place bag MVP * Removed potentially breaking handle_sigints * Updated GetPersonPoint logic and added goal forgiving * Updated wait times and removed redundent check * Merging fixed gpu version of HRI (#454) * HRI task (now SM1-5) (#447) * Ros2 Port Language Package * Introduction testing * Modifications. fix init before final push * Message filter * Launch file * State machine and setup.py modifications * Fix register face and recognise * Port HRI introduce states to YASMIN * Fix YOLO detections in introduce. * Initial code for pick and place * Bug fixes * Fixes after testing on the robot * Black formatting * HRI Task - SM1-3 fully tested and working on the robot (#443) * Testing HRI SM2 * LLM package with venv * Fixed SM3 * RVIZ debug for HRI testing * Fixed lasr_llm by removing broken llama_cpp_python * Successfull SM2 of HRI * WIP - SM2 to SM3 * WIP: CHECKS * WIP - SM1 -> SM3 Final checks to see if it works consistently needs to be carried out * Ported recieve Object * Temporary detect3d fix * Working SM1-3 of HRI --------- Co-authored-by: Aldrich-Fernandes <[email protected]> * New VLM service to be used in HRI (#444) * vlm service * service update * Added back license and readme * Formatted files with black --------- Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> * VLM service for HRI task (#445) * Working VLM Replaced old vqa for describing people with new vlm service * Formatted files with black * Fixed accidental merge errors * WIP - Setting up and Testing HRI task * Fix introduce and remove testing code * WIP - HRI task semi-working SM1-4 works correctly, however during SM4 it doesn't look at the seated guests * Change LLM to Ollama got HRI get anem and drink * Updated Seat Guest (SM3) * WIP - HRI SM4 being refactored * Verbal introduction is correct * Debugging SM4 face person errors * Formatted files with black --------- Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: Yara Alkhelaiwi <[email protected]> Co-authored-by: rajhirym <[email protected]> Co-authored-by: Yara Alkhelaiwi <Yarakmk> Co-authored-by: Aldrich-Fernandes <[email protected]> Co-authored-by: Ma'ayan Armony <[email protected]> * Added REID fallback Also made minor changes to accomodate gpu * Test HRI on GPU * Updated HRI to say attributes Also added in a SafeGoToLocation and moved StartDoorSM to skills * Debugging HRI task * Added gpu requirements for lasr_vision_reid * Testing of HRI with GPU * Fixing bugs * Fixing of EyeTracker * Working eye tracker, yolo is bottleneck * Fully fixed eye tracker * Adding in host * Fixed introduce for host * Formatted files * Merge branch 'ros2-hri-sm6' into ros2-gpu-and-reid * Added in host logic to SM * Testing HRI * Working HRI * Bug fixes * HRI MVP * Formatted files * Merge branch 'ros2' into ros2-hri-sm6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.