AzureBlobDataSource to AzureTableApi: Out of Memory

[GuillaumeVadeBe]

We are trying to move data from an AzureBlobDataSource to AzureTableApi and noticed that the tool keeps on increasing in memory. If we look at the source code (AzureTableAPIDataSinkExtension), it turns out that a task is being created for every single dataItem. The completion of the tasks is only awaited after the entire enumeration of the dataItems which causes the memory to increase depending on the amount of items in the source.

Would it be possible to fix this so the memory usage remains stable no matter the blob size?

[pdehne-steidle]

We have the same issue. In our case we are using AzureTableAPI as the data source and data sink. It's currently not possible to copy bigger tables.