LLM logging does not work when running agent remotely

[adityasoni9998]

It seems that LLM logging (by setting log_completions=True in LLM class) does not save any logs on user's machine when running this example. If I understand agent-sdk correctly, when running remotely (via Docker and/or remote runtime), the user's machine just talks to a server in the sandbox where all of agent-sdk's code is run and the LLM calls are made and logged. If this is true, then the log_dir in Telemetry will point to a path in the remote container, and not on user's machine which is probably why this happens?
When using the default log_completions_folder, I see a log entry from Docker container (I set DEBUG=1) saying

[DOCKER] openhands/sdk/llm/utils/telemetry.py:301: UserWarning: Telemetry logging failed: log_dir does not exist: logs/completions

When I set log_completions_folder to a /tmp/logs/completions and make this directory in remote sandbox using workspace.execute_command("mkdir -p /tmp/logs/completions") , I see that the JSON files are being logged inside the docker container at the above location, but they are all empty, and the logs now say:

[DOCKER] openhands/sdk/llm/utils/telemetry.py:301: UserWarning: Telemetry logging failed: Circular reference detected

[enyst]

@OpenHands investigate and find the problem here. I believe you are able to install docker yourself, but I suggest to do this in two stages: first understand the code and test it with a "remote" workspace which has a host, but it's localhost; and when you find the issue propose a fix and open a PR.

Then, you can continue and if you estimate that you need docker, install it, but it's up to you: if you are confident your fix covers the entire problem, you can say so and don't complicate it.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[openhands-ai]

Final summary for issue #1027: “LLM logging does not work when running agent remotely”

What I changed

• Root cause: When running remotely (e.g., in Docker), telemetry logging sometimes tried to JSON-serialize complex Pydantic objects (e.g., tool/spec objects in the log context) via dict, creating circular references and empty/failed logs with “Circular reference detected.”
• Fix: Make telemetry serialization robust for Pydantic models.
  • Updated openhands-sdk/openhands/sdk/llm/utils/telemetry.py:_safe_json to:
    • Keep special handling for litellm ModelResponse/ResponsesAPIResponse via model_dump(mode="json", exclude_none=True)
    • NEW: serialize any Pydantic BaseModel via model_dump(mode="json", exclude_none=True), respecting excluded fields and avoiding cycles
    • Fallbacks to dict → str(obj) remain for non-Pydantic types
• Tests: Added a targeted test simulating the remote case
  • tests/sdk/llm/test_llm_telemetry.py::test_log_completion_with_pydantic_objects_in_context
  • Creates a self-referencing Pydantic model with an excluded field (executor), ensures no “Circular reference detected” warnings, and verifies a valid JSON log is produced

Notes

• The default log directory behavior is unchanged: if you use a remote server, logs write to the server’s filesystem. If the default directory doesn’t exist, a warning is emitted (this is expected and preserves current behavior). Mount a host directory or set log_completions_folder accordingly if you want logs accessible from the host.

Checklist

• Investigated telemetry/logging and remote workspace/conversation behavior
• Implemented minimal, backward-compatible fix
• Added unit test covering the remote logging failure mode
• Ran pre-commit hooks on modified files (ruff format/lint, pycodestyle, pyright)
• Validated telemetry tests including new case
• Committed only relevant files (telemetry.py and its tests)
• Created a branch and opened a PR that closes the issue

Branch and PR

• Branch: openhands/fix-remote-llm-telemetry-logging
• PR: Fix: robust LLM telemetry logging for remote agent/server runs #1028
• Commit message includes: “Fixes LLM logging does not work when running agent remotely #1027” and “Co-authored-by: openhands openhands@all-hands.dev”

Confidence and next steps

• I’m confident this resolves the “Circular reference detected” and empty log files when running remotely.
• If you want me to also verify the Docker example end-to-end, I can do that. If you’d like auto-creation of the default log directory in remote environments, I can propose that as a separate, backward-compatible enhancement.

_{View full conversation}

[adityasoni9998]

Hi @enyst ! Thanks for looking into this issue. If the messages are only logged inside the remote container, they are not suitable for manual inspection/reuse by the end user. Can there be a modular way of copying these LLM messages from remote workspace to user's machine in the same relative path as that of log_dir?