HarmonyError: Unexpected EOS while waiting for message header to complete

[Arunprakash-A]

Summary:
Running the example code from the openai_harmony repository results in a HarmonyError when calling parse_messages_from_completion_tokens(...).

Steps to Reproduce:

from openai_harmony import (
    load_harmony_encoding,
    HarmonyEncodingName,
    Role,
    Message,
    Conversation,
    DeveloperContent,
    SystemContent,
)
enc = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
convo = Conversation.from_messages([
    Message.from_role_and_content(
        Role.SYSTEM,
        SystemContent.new(),
    ),
    Message.from_role_and_content(
        Role.DEVELOPER,
        DeveloperContent.new().with_instructions("Talk like a pirate!")
    ),
    Message.from_role_and_content(Role.USER, "Arrr, how be you?"),
])
tokens = enc.render_conversation_for_completion(convo, Role.ASSISTANT)
print(tokens)

#fails here <--
parsed = enc.parse_messages_from_completion_tokens(tokens, role=Role.ASSISTANT)
print(parsed)

It raised the following error while executing the last statement in the above example `parsed = enc.parse_messages_from_completion_tokens(tokens, role=Role.ASSISTANT).

Output and Traceback

[200006, 17360, 200008, 3575, 553, 17554, 162016, 11, 261, 4410, 6439, 2359, 22203, 656, 7788, 17527, 558, 87447, 100594, 25, 220, 1323, 19, 12, 3218, 279, 30377, 289, 25, 14093, 279, 2, 13888, 18403, 25, 8450, 11, 49159, 11, 1721, 13, 21030, 2804, 413, 7360, 395, 1753, 3176, 13, 200007, 200006, 77944, 200008, 2, 68406, 279, 37992, 1299, 261, 96063, 0, 200007, 200006, 1428, 200008, 8977, 81, 11, 1495, 413, 481, 30, 200007, 200006, 173781]
Traceback (most recent call last):
  File "/home/arun/experiments/gpt_oss.py", line 25, in <module>
    parsed = enc.parse_messages_from_completion_tokens(tokens, role=Role.ASSISTANT)
  File "/home/arun/miniconda3/envs/thcr/lib/python3.13/site-packages/openai_harmony/__init__.py", line 525, in parse_messages_from_completion_tokens
    raw_json: str = self._inner.parse_messages_from_completion_tokens(
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        list(tokens), None if role is None else str(role.value)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
openai_harmony.HarmonyError: Unexpected EOS while waiting for message header to complete

I checked the source code; however, the variable self._inner is bound to the Rust-side variable inner.

[phial3]

The error occurred here, the method process_next in encoding.rs:

StreamState::Header { header_tokens } => {
    let msg_tok = self
        .encoding
        .render_formatting_token(FormattingToken::Message)?;
    match token {
        Some(token) if token == msg_tok => {
            // Clone the tokens and next_role, then clear the state before parsing
            let header_tokens_cloned = header_tokens.clone();
            let next_role_cloned = next_role_clone;
            // Set state to dummy to drop mutable borrow
            self.state = StreamState::ExpectStart;
            let header =
                self.parse_header_from_tokens(&header_tokens_cloned, next_role_cloned)?;
            self.next_role = None;
            self.state = StreamState::Content {
                header,
                content_tokens: Vec::new(),
            };
        }
        Some(token) => {
            header_tokens.push(token);
        }
        None => {
			// Use the test code above, matched an empty token value
			// But it works normally when I comment this out.
			// I'm not sure why this should be bail
            anyhow::bail!("Unexpected EOS while waiting for message header to complete");
        }
    }
}

[phial3]

This is my running result：

[200006, 17360, 200008, 3575, 553, 17554, 162016, 11, 261, 4410, 6439, 2359, 22203, 656, 7788, 17527, 558, 87447, 100594, 25, 220, 1323, 19, 12, 3218, 279, 30377, 289, 25, 14093, 279, 2, 13888, 18403, 25, 8450, 11, 49159, 11, 1721, 13, 21030, 2804, 413, 7360, 395, 1753, 3176, 13, 200007, 200006, 77944, 200008, 2, 68406, 279, 37992, 1299, 261, 96063, 0, 200007, 200006, 1428, 200008, 8977, 81, 11, 1495, 413, 481, 30, 200007, 200006, 173781]
[Message(author=Author(role=<Role.ASSISTANT: 'assistant'>, name=None), content=[TextContent(text='You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\n\nReasoning: medium\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.')], channel=None, recipient='<|start|>system', content_type=None), Message(author=Author(role=<Role.DEVELOPER: 'developer'>, name=None), content=[TextContent(text='# Instructions\n\nTalk like a pirate!')], channel=None, recipient=None, content_type=None), Message(author=Author(role=<Role.USER: 'user'>, name=None), content=[TextContent(text='Arrr, how be you?')], channel=None, recipient=None, content_type=None)]

[phial3]

issues/38

[scut2sjtu]

I also encountered the same issue，but how can I solve it?
@phial3

[phial3]

I also encountered the same issue，but how can I solve it? @phial3

StreamState::Header { header_tokens } => {
    let msg_tok = self
        .encoding
        .render_formatting_token(FormattingToken::Message)?;
    match token {
        Some(token) if token == msg_tok => {
            // Clone the tokens and next_role, then clear the state before parsing
            let header_tokens_cloned = header_tokens.clone();
            let next_role_cloned = next_role_clone;
            // Set state to dummy to drop mutable borrow
            self.state = StreamState::ExpectStart;
            let header =
                self.parse_header_from_tokens(&header_tokens_cloned, next_role_cloned)?;
            self.next_role = None;
            self.state = StreamState::Content {
                header,
                content_tokens: Vec::new(),
            };
        }
        Some(token) => {
            header_tokens.push(token);
        }
        None => {
			// The stream ended while we were parsing a header.
            // This means the message is incomplete. Instead of crashing,
            // we discard the partial header and reset the state to wait for a new message.
            // This makes the parser more robust to truncated input.
			// (1) If receiving EOS in the Header state is a possible valid,
			// it can be regarded as a signal indicating the need to reset to the initial state.
			// self.state = StreamState::ExpectStart;
            // self.tokens.clear();
            // self.next_role = None;
            // self.undecoded_tokens.clear();
            // self.last_content_delta = None;
			// (2) if think it's a mistake, the previous was okay.
            anyhow::bail!("Unexpected EOS while waiting for message header to complete");
        }
    }
}

[dkundel-openai]

Hey! The parser handles the completion tokens so sending the same tokens you rendered in will mismatch with what the parser considers a valid input for a completion from an assistant.

[Arunprakash-A]

Hey! The parser handles the completion tokens so sending the same tokens you rendered in will mismatch with what the parser considers a valid input for a completion from an assistant.

Thanks. That makes sense. Could you update the code with some dummy completion tokens? That would be helpful.

[Howe829]

Hey! The parser handles the completion tokens so sending the same tokens you rendered in will mismatch with what the parser considers a valid input for a completion from an assistant.

Thanks. That makes sense. Could you update the code with some dummy completion tokens? That would be helpful.

Hey, I think you can check this pr openai/openai-cookbook#2038

[aleksanderhan]

I have the same issue. Something is not right or clearly explained.