[GSoC] Demo/integration with openhands#4320
Conversation
mzegla
added
the
GSoC
| ["Llama3"]="hermes3" | ||
| ["llama3"]="hermes3" | ||
| ["Mistral"]="hermes3" | ||
| ["mistral"]="hermes3" |
There was a problem hiding this comment.
This mapping is not going to work. We have multiple tool parsers, so for example "Llama3" should map to llama3 parser
| └── models/ | ||
| └── <model-id>/ | ||
| ├── openvino_model.xml | ||
| ├── openvino_model.bin |
There was a problem hiding this comment.
You could add some dots here to indicate those are not the only files expected in model catalog
| | Qwen 3 | `hermes3` | Strong coding performance, various sizes | | ||
| | Llama 3 | `hermes3` | Good general instruction following | | ||
| | Mistral | `hermes3` | Efficient inference | |
There was a problem hiding this comment.
wrong tool parsers for llama3 and mistral
| | Llama 3 | `hermes3` | Good general instruction following | | ||
| | Mistral | `hermes3` | Efficient inference | | ||
|
|
||
| > **Note:** Documentation examples use OpenVINO-exported models from the OpenVINO Hugging Face organization. The documented workflow is validated using `OpenVINO/qwen3-0.6b-int8-ov`. Other compatible models may also be used. The example model is chosen because it provides a lightweight validation path. |
There was a problem hiding this comment.
Note that for the final version we would liken to replace qwen3-0.6b model here with a one that will be capable of handling agentic workloads
| Tool parsers enable structured output for function calling. When OpenHands needs to execute a tool (e.g., running code, reading files), it expects the LLM to return structured JSON that specifies the tool name and arguments. The tool parser converts model outputs into this format. | ||
|
|
||
| - **Without tool parser:** The model may hallucinate tool calls or return unstructured text |
There was a problem hiding this comment.
That's not exactly true. Tool parsers do not influence model generation. So if model hallucinates or simply does not decide to generate structured output, tool parser will not help. The purpose of tool parsers is to interpret model output, and if there are some tools generated there, extracts them from structured output.
| └── graph.pbtxt # MediaPipe LLM graph configuration | ||
| ``` | ||
|
|
||
| ### Using Helper Scripts (Optional) |
There was a problem hiding this comment.
Marked as optional. Does it mean we can move forward without it?
If so, I think I would remove it from the main, user facing README to make it straightforward.
Just keep here steps that have to be executed to not overextend the volume of the demo.
|
|
||
| ## 5. Deployment Workflow | ||
|
|
||
| This demo provides two equivalent paths to deploy OVMS and OpenHands. Both achieve the same result—choose based on your preference. |
There was a problem hiding this comment.
Same as above. Two paths of deployment seem like too much for the main readme.
Maybe you could select one that you would recommend. If you'd like to keep information about optional steps or another path, then we could have another markdown file with such additional information.
In my opinion this README should be as simple and short as we can possibly make it.
| # Specify device, parser, or cache directory | ||
| ./scripts/deploy_model_ovms.sh OpenVINO/qwen3-0.6b-int8-ov \ | ||
| --device CPU \ | ||
| --parser qwen \ |
There was a problem hiding this comment.
Hmm, why not just pass hermes3. Maybe if we pass tool parsers names directly we can avoid having mappings?
|
|
||
| --- | ||
|
|
||
| ## 6. Understanding `docker-compose.yml` (Reference) |
There was a problem hiding this comment.
Informative section, but it's long and I would move it to a separate document, similarly to previous comments.
This readme should be quick to read and follow and details should be available elsewhere, for those who want to dive deeper into it.
|
|
||
| --- | ||
|
|
||
| ## 10. Screenshots |
There was a problem hiding this comment.
Not sure if we need such specific section. Maybe just put screenshots in appropriate places in earlier sections along with the story.
2 participants
No description provided.