feat: add ImpitHttpClient http-client client using the impit library

[Mantisus]

Description

• add ImpitHttpClient http-client client using the impit library

Issues

• Relates: Integrate impit as HTTP client #1079

Testing

Added tests for ImpitHttpClient. ImpitHttpClient is enabled for all tests using http-client

[Mantisus]

For now, I suggest adding impit as an additional dependency, as it still needs some tweaking before it's ready to replace httpx.

Awaiting a decision - apify/impit#123

[Copilot]

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

pyproject.toml:63

• The version requirement for impit in the main dependencies (>=0.1.0) differs from the one in the adaptive-crawler section (>=0.2.0), which may lead to dependency conflicts. Consider aligning them to a consistent version.

"impit>=0.1.0",

[Copilot]

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

[Mantisus]

Python binding Impit has all the basic functionality to integrate into Crawlee.

The _get_client method is implemented based on ImpitHttpClient. However, this looks inefficient, especially when working without a proxy, but using a SessionPool of size greater than 1, because the client will be created anew for each request. I think we should improve this on the impit side. @barjin, maybe you'll have some ideas.

Replacing httpx with impit as the main client, I propose to do in a separate PR

[Pijukatel]

I see you are following established pattern about adding new test file for the new client, but maybe now is the time to refactor the tests and have only one test file for all clients and parametrize all tests with client.

For example:

@pytest.mark.parametrize("http_client", [
    CurlImpersonateHttpClient(http_version=CurlHttpVersion.V1_1),
    ImpitHttpClient(),
    HttpxHttpClient(http2=False)])
async def test_http_1(http_client: HttpClient, server_url: URL) -> None:
    response = await http_client.send_request(str(server_url))
    assert response.http_version == 'HTTP/1.1'

Maybe we would need 3xclient factories instead and parametrize by that, but regardless of that I think it would be great to reduce code duplication and ensure that we have exactly the same tests for all clients and that they can all work in exactly the same way.

But this is just a suggestion. Maybe it can be done in separate PR as well to not mix new implementation with pure refactoring.

[Mantisus]

That's a great idea. But yes, I think it should be done in a separate PR.

Also, we could take the same approach for beautifulsoup and parsel crawlers. Since the tests for them are also completely duplicated.

[Pijukatel]

#1292

[Pijukatel]

I will wait for this PR to be merged first, and then we can refactor: #1299

[vdusek]

LGTM

[Mantisus]

Not merging this PR until we resolve the test issue

[vdusek]

Test issue

[gw1] node down: Not properly terminated
[gw1] [ 98%] FAILED tests/unit/test_service_locator.py::test_storage_client_conflict 
replacing crashed worker gw1
tests/unit/crawlers/_parsel/test_parsel_crawler.py::test_enqueue_links_selector[curl] 

[vdusek]

LGTM