[cloud_firestore]: Firestore whereNotIn Query with Chunking Produces Incorrect Results

[Zubairpv]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

No response

Which platforms are affected?

No response

Description

Description
I encountered an issue while using Firestore's whereNotIn query with chunking to handle the 10-item limit for exclusions. The problem arises when chunking doesn't properly exclude all the necessary productIds across multiple queries. This leads to incorrect results being fetched.

Reproducing the issue

Steps to Reproduce

• Use Firestore with a collection containing products.
• Attempt to exclude a list of product IDs (productIds) longer than 10 using whereNotIn.
• Chunk the list of productIds into smaller sublists of 10 or fewer items.
• Query Firestore incrementally with exclusions.

Expected Behavior

All product IDs specified in productIds should be excluded from the results across all chunks.

Actual Behavior
The exclusions in the current chunk do not respect the exclusions from previous chunks, leading to incorrect results.
Relevant Code Snippet

Future<List<ProductModel>> getProducts(String categoryId, List<ProductModel> products) async {
  final productIds = products.map((product) => product.productId).toList();
  List<ProductModel> result = [];

  if (productIds.isEmpty) {
    // Handle empty case
    return await homeProvider.vendersCollection
        .doc(homeProvider.Vender_Id ?? '')
        .collection('products')
        .where('category_id', isEqualTo: categoryId)
        .get()
        .then((snapshot) => snapshot.docs.map((doc) => ProductModel.fromJson(doc.data())).toList());
  }

  final chunks = List.generate(
    (productIds.length / 10).ceil(),
    (index) => productIds.skip(index * 10).take(10).toList(),
  );

  for (final chunk in chunks) {
    final querySnapshot = await homeProvider.vendersCollection
        .doc(homeProvider.Vender_Id ?? '')
        .collection('products')
        .where('product_available_bool', isEqualTo: true)
        .where('category_id', isEqualTo: categoryId)
        .where(FieldPath.documentId, whereNotIn: chunk)
        .get();

    result.addAll(querySnapshot.docs.map((doc) => ProductModel.fromJson(doc.data())).toList());
  }

  return result;
}

Firebase Core version

^3.4.0

Flutter Version

3.24

Relevant Log Output

Flutter dependencies

name: my ex app
description: "A new Flutter project."
publish_to: 'none'
version: 1.0.0+1

environment:
sdk: '>=3.3.4 <4.0.0'
dependencies:
flutter:
sdk: flutter

cupertino_icons: ^1.0.6
firebase_core: ^3.4.0
cloud_firestore: ^5.0.1
firebase_auth: ^5.1.0
google_sign_in: ^6.2.1
shared_preferences: ^2.2.3
flutter_screenutil: ^5.9.3
fluttertoast: ^8.2.6
provider: ^6.0.5
google_fonts: ^5.1.0
url_launcher: ^6.3.0
intl: ^0.19.0
pull_to_refresh: ^2.0.0
pin_code_fields: ^8.0.1
connectivity_plus: ^2.3.0
http: ^1.0.0
geolocator: ^13.0.2
geocoding: ^3.0.0
carousel_slider: ^5.0.0
hive: ^2.2.3
hive_flutter: ^1.1.0

dev_dependencies:
flutter_test:
sdk: flutter
flutter_lints: ^3.0.0
build_runner: ^2.4.10
hive_generator: ^2.0.1

flutter:

uses-material-design: true

To add assets to your application, add an assets section, like this:

assets:
- assets/
- assets/home-icon.png
- assets/Tracking.png
- assets/offer-poster.jpg
- assets/order-history.png
- assets/user-icon.png
- assets/grocerys-texture.png
- assets\products\yippe.png
- assets/products/oreo.png
- assets/products/parle-g.png
- assets\products\biscuit (1).png
- assets\illistrations\no vender.png
- assets\illistrations\No internet.png
- assets\icons\google-maps.png

Additional context and comments

the product ids is big list

[SelaseKay]

Hi @Zubairpv, what platform are you experiencing this on?

[MichaelVerdon]

I believe we were able to reproduce it, when running the example we get a resulting list containing duplicates of the non-excluded items. For example:
all products = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]

exclude = [1,2,3,4,5,6,7,8,9,10,11,12]

results = [1,2,3,4,5,6,7,8,9,10,11,12,13,13,14,14,15,15,16,16]
(Values that we do not wish to exclude from firestore are duplicated in the list but it keeps the exlucded ids).

It appears whereNotIn query does its job but after each iteration you need to either remove the ids from the excluded id list or alter the list after to make a new list from duplicated values. Does this make sense? Let me know

[MichaelVerdon]

I managed to come up with a solution here, this was the only way I was able to make it work.:

for (final chunk in chunks) {
      debugPrint("Chunk >>> ${chunk}");
      final querySnapshot = await _firestore
          .collection('test_13486')
          .doc('vendor_id')
          .collection('products')
          .where('product_available_bool', isEqualTo: true)
          .where('category_id', isEqualTo: categoryId)
          .where(FieldPath.documentId, whereNotIn: chunk)
          .get();

      result.addAll(querySnapshot.docs
      .map((doc) => ProductModel.fromJson(doc.data())).toList());
    }

    // Count occurences in the list
    var result_map = Map();
      result.forEach((element) {
        if(!result_map.containsKey(element.productId)) {
        result_map[element.productId] = 1;
        } else {
        print("another");
        result_map[element.productId] += 1;
        }
      });

    // Create a new entry if element wasn’t seen before and increase counter when it is seen again.
    List<ProductModel> results_ = [];
    result_map.forEach((k,v) {
      if(v > 1){
        for(var item in result){
          if(item.productId == k){
            results_.add(item);
            break;
          }
        }
      }});

    // result1 = [11,12,13,14, 15,16..18]
    // result2 = [1...10, 13...18]

    return results_;