Shouldn't list remote file server each time max-fetch-size is reached

[karim789]

In what version(s) of Spring Integration are you seeing this issue?

all versions up to 6.5.x

Describe the bug

AbstractInboundFileSynchronizingMessageSource shoud list the remote server only once per poll.
However what happens here is each time you download max-fetch-size files, and max-message-per-poll isn't reached, then instead of working on the initial list until, it will list the remote server again, which is unnecessary and costs a lot of time.

Currently if you need a jdbc transaction per file downloaded and keep it to the end of the message flow, you need to use max-fetch-size=1, but then you end up having the remote server listed before downloading each single file, which slow down things considerably.

Chatgpt told me that AbstractRemoteFileStreamingMessageSource could be a solution to this, because it was maintaining a queue that was filled from remote file server list only when queue is empty.
But here I see the queue is cleared when maxFetchSize is reach, therefore it will trigger a remote file server list right after that

		if (maxFetchSize > 0 && this.fetched.get() >= maxFetchSize) {
			this.toBeReceived.clear();
			this.fetched.set(0);
		}

(

spring-integration/spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java

Line 219 in 2315423

if (maxFetchSize > 0 && this.fetched.get() >= maxFetchSize) {

)

Therefore I see no solution to have a transaction per downloaded file and it's emitted message without having bad performances.

To Reproduce

Use a max-fetch-size=1

Expected behavior

Work on the remote file list until exhausted, before polling the server for a file list again.

Sample

Uncessary as code analysis shows the issue.

[artembilan]

Can we understand why ChatGPT is involved here? Does it really give you a solution?

What is initial list in your meaning, since the reason of max fetch is to not fetch from remote server more.

We will look in details tomorrow.