enable validationmode for eagle3

[songbell]

eagle 3 validation mode for multinomial sampling
to be merged after #2740

[Copilot]

Pull Request Overview

This PR enables validation mode for Eagle 3 speculative decoding, specifically adding support for multinomial sampling. The change integrates EAGLE 3 model capabilities into the continuous batching pipeline with specialized handling for hidden state management and token validation.

• Adds eagle3_mode flag to llm_bench for enabling EAGLE 3 speculative decoding
• Implements comprehensive EAGLE 3 support with model transformations, hidden state extraction, and specialized sampling
• Extends tokenization to support EAGLE 3 mode with conditional special token handling

Reviewed Changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tools/llm_bench/benchmark.py	Adds command-line flag for EAGLE 3 mode
tools/llm_bench/task/text_generation.py	Implements conditional tokenization for EAGLE 3 mode
tools/continuous_batching/accuracy/continuous_batching_eagle_decoding.cpp	Adds new EAGLE decoding accuracy testing tool
tools/continuous_batching/accuracy/CMakeLists.txt	Builds the new EAGLE decoding test executable
src/cpp/src/speculative_decoding/	Core EAGLE 3 implementation with model transforms and pipeline logic
src/cpp/src/continuous_batching/	Pipeline integration and model runner enhancements for EAGLE mode
src/cpp/src/safe_tensor_wrapper.*	Utility for parsing SafeTensor format files
src/cpp/include/openvino/genai/	Header updates for EAGLE mode support

Comments suppressed due to low confidence (1)

src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp:1

• The magic number 0.9f should be defined as a named constant to improve code readability and maintainability.

// Copyright (C) 2023-2025 Intel Corporation

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The comment mentions 'compress rate' but doesn't explain what compression is being referred to or why disabling special tokens improves it. Consider adding a more detailed comment explaining the relationship between EAGLE 3 mode and special token handling.

Suggested change

	# eagle3 needs to disable special tokens to ensure compress rate
	# In EAGLE 3 mode, the model uses a custom tokenization and compression scheme that achieves a higher compression rate
	# (i.e., fewer tokens per input) when special tokens (such as BOS/EOS) are omitted. Disabling special tokens here ensures
	# that the input is tokenized in a way that is compatible with EAGLE 3's expectations and maximizes compression efficiency.

[Copilot]

Typo in comment: 'speulative' should be 'speculative'.

Suggested change

	// sampling param for speulative decoding
	// sampling param for speculative decoding

[Copilot]

Missing space before '<<std::endl;'. Should be consistent with spacing pattern used elsewhere in the file.

Suggested change

	std::cout << "Request was ignored due to lack of memory." <<std::endl;
	std::cout << "Request was ignored due to lack of memory." << std::endl;

[Copilot]

Missing space before '<<std::endl;'. Should be consistent with spacing pattern used elsewhere in the file.

Suggested change

	std::cout << "Request was aborted." <<std::endl;
	std::cout << "Request was aborted." << std::endl;

[Copilot]

Using std::cout for logging in production code is not recommended. Consider using a proper logging framework or at least make these debug-only outputs that can be conditionally compiled out.

[Copilot]

Commented-out code should be removed rather than left in the codebase. If this is for future use, consider adding a TODO comment explaining the purpose.

Suggested change

	// m_initial_hidden_states.clear();
	//auto key = std::make_pair(request_id, seq_grouped_id);