Skip to content

Streaming remote files with a single file filter breaks fair rotation of sources #3315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dturanski opened this issue Jun 29, 2020 · 0 comments · Fixed by #3318
Closed

Streaming remote files with a single file filter breaks fair rotation of sources #3315

dturanski opened this issue Jun 29, 2020 · 0 comments · Fixed by #3318
Assignees
@garyrussell
Labels
in: file type: bug
Milestone
5.4 M2

Comments

@dturanski
Copy link
Contributor

dturanski commented Jun 29, 2020 •
edited by artembilan
Loading

Describe the bug
https://github.com/spring-projects/spring-integration/blame/master/spring-integration-file/src/main/java/org/springframework/integration/file/remote/AbstractRemoteFileStreamingMessageSource.java#L262. There may have been some changes here that affect the behavior of the fair rotation when streaming in the sftp source. Since we have SftpPersistentAcceptOnceFileListFilter configured, this supports single file filtering so maxFetch is ignored.
To Reproduce

Steps to reproduce the behavior.

Comment out this line https://github.com/dturanski/stream-applications-1/blob/sftp-source/functions/supplier/sftp-supplier/src/main/java/org/springframework/cloud/fn/supplier/sftp/SftpSupplierConfiguration.java#L160

Then run this test
https://github.com/dturanski/stream-applications-1/blob/sftp-source/functions/supplier/sftp-supplier/src/test/java/org/springframework/cloud/fn/supplier/sftp/SftpSupplierApplicationTests.java#L199

Expected behavior

The test should pass, verifying fair rotations

Sample

See above
@dturanski dturanski added type: bug status: waiting-for-triage The issue need to be evaluated and its future decided labels Jun 29, 2020
@artembilan artembilan added backport 5.2.x in: file and removed status: waiting-for-triage The issue need to be evaluated and its future decided labels Jun 29, 2020
@artembilan artembilan added this to the 5.4 M2 milestone Jun 29, 2020
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Jul 1, 2020
@garyrussell
spring-projectsGH-3315: Fix (S)FTP Stream with Fair Rotation
281edd9 
Resolves spring-projects#3315

`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.

Honor `maxFetchSize`, even with filters that support single file filtering.

**cherry-pick to 5.3.x, 5.2.x**
garyrussell added a commit to garyrussell/spring-integration that referenced this issue Jul 1, 2020
@garyrussell
spring-projectsGH-3315: Fix (S)FTP Stream with Fair Rotation
572ec5b 
Resolves spring-projects#3315

`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.

Honor `maxFetchSize`, even with filters that support single file filtering.

**cherry-pick to 5.3.x, 5.2.x**
@garyrussell garyrussell mentioned this issue Jul 1, 2020
Merged
artembilan pushed a commit that referenced this issue Jul 1, 2020
@garyrussell @artembilan
GH-3315: Fix (S)FTP Stream with Fair Rotation
f3d61ec 
Resolves #3315

`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.

Honor `maxFetchSize`, even with filters that support single file filtering.

**cherry-pick to 5.3.x, 5.2.x**
artembilan pushed a commit that referenced this issue Jul 1, 2020
@garyrussell @artembilan
GH-3315: Fix (S)FTP Stream with Fair Rotation
9cd8a4a 
Resolves #3315

`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.

Honor `maxFetchSize`, even with filters that support single file filtering.

**cherry-pick to 5.3.x, 5.2.x**
artembilan pushed a commit that referenced this issue Jul 1, 2020
@garyrussell @artembilan
GH-3315: Fix (S)FTP Stream with Fair Rotation
6d60b90 
Resolves #3315

`maxFetchSize` ignored with filters supporting single file filtering (default).
This breaks "fair" rotation with the `RotatingServerAdvice`.

Honor `maxFetchSize`, even with filters that support single file filtering.

**cherry-pick to 5.3.x, 5.2.x**

# Conflicts:
#	spring-integration-ftp/src/test/java/org/springframework/integration/ftp/inbound/RotatingServersTests.java
@dependabot dependabot bot mentioned this issue Mar 15, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@garyrussell garyrussell

Labels
in: file type: bug
Projects
None yet
Milestone
5.4 M2
Development

Successfully merging a pull request may close this issue.

GH-3315: Fix (S)FTP Stream with Fair Rotation garyrussell/spring-integration
3 participants
@garyrussell @dturanski @artembilan