bug: CatalogRequest call fails if number of entries in policy or contractdefinition or contract negotiation is more than edc.sql.fetch.size limit #3640
Comments
|
CatalogRequest fails if any of the sql stores has entries > edc.sql.fetch.size. |
ndr-brt
added
bug_report
|
Hey, thanks for reporting, I tried to reproduce using the I tried your proposed solution but it still throws the error. |
|
I tried to replicate the error in an automated test: I tried both with Testcontainer and with a postgresql instance running in docker separately but that error is not thrown. |
May be can you try this. return StreamSupport.stream(Spliterators.spliteratorUnknownSize(new ResultSetIterator(connection, resultSetMapper, sql, configuration, arguments), 0), false).toList().stream(); This is not ideal and correct fix but may be good hint what might be an issue. For me it looks like
|
|
to be honest I think that we shouldn't just "try", we need to be able to replicate the bug in a deterministic way through an automated test and then go with the fix. |
|
@suh-rao could you please update EDC to the latest version and try it again, and also provide a way to reproduce the bug in an automated way (likely with a test)? |
|
Hello @ndr-brt you mentioned that with EndToEndTransferPostgresqlTest you were able to reproduce the error. I guess then it is reproducible with latest EDC version? Can't this test considered as automated way. |
|
I done it by hacking stuff here and there but there will be the need of a reproducible way, that could be done providing a test that breaks out of the box, could you provide that? |
|
@ndr-brt : I will try it and update you |
|
Hello @ndr-brt I tried but was unable to reproduce it . One thing I observed was, it looks like postgres was not respecting fetchSize value. Instead always returns everything, in resultset. This was not case when executed without automation. ResultSet will have exactly number of fetchSize. Attached screenshot incase of Automation.
Attached screenshot incase of actual execution Do you know how to how to make postgres test container respect fetchSize |
|
In my example I set fetchSize to 5. |
|
wrong button 😅 |
Initial thought on seeing those mail was issue was resolved 😅😅. Nice that you found the issue. Will be interesting to see what is root cause. |
|
Just for the record: here's the test replicated (needs updated |
ndr-brt
added
bug
|
@ndr-brt : Same issue is there in other entities(Assets, Policy, ContractDefinition etc) too. } Same issue happens. You can follow same for other entities too. Not sure whether this issue created refers to same #3682 |
Here I mentioned same, that issue occurs in other entities too |
|
Yes, those will be solved by #3682 |
Bug Report
Describe the Bug
CatalogRequest call fails with edcexception if number entries in policy or contractdefinition or contract negotiation tables are more than limit mentioned in edc.sql.fetch.size.
This value will be set before executeQuery --> preparedStatement.setFetchSize(fetchSize)
Issue occurs since ResultSet is closed before all (More than fetch limit) data retrieved.
From postgres databse we get error "portal 'c_n' does not exist".
Expected Behavior
Irrespective of fetchSize limit set all data has to be streamed and retrieved.
Observed Behavior
resultset, connection, statement gets closed before all data retrieved.
Steps to Reproduce
Steps to reproduce the behavior:
Exception occurs.
Context Information
Detailed Description
Logs:
Possible Implementation
In class: SqlQueryExecutor.
And new iterator class
The text was updated successfully, but these errors were encountered: