Crawler on update: Checksum remains unchanged even if HTML document has been modified

[stejacob]

Hi,

We are running a demo site on a Microsoft IIS web server and using the latest version of Norconex Crawler.

We've configured both the documentChecksummer and metadataChecksummer, but we’re noticing that the checksum value remains the same even after modifying the HTML file on the server.

We've tried using both the "Last Modified" field and the MD5 checksum on specific fields, but the document continues to be rejected because the checksum generated remains unchanged, even when the HTML document has been modified.

....
Line  8680: 14:12:28.256 [es-node2.deimscloud.mil.ca#3] INFO  REJECTED_UNMODIFIED - http://es-node2.deimscloud.mil.ca/about.html - MD5DocumentChecksummer - Checksum=4529f56e11c85023cd3b815ffd1c2b1e|

....
	Line 10840: 14:30:39.085 [es-node2.deimscloud.mil.ca#3] INFO  REJECTED_UNMODIFIED - http://es-node2.deimscloud.mil.ca/about.html - MD5DocumentChecksummer - Checksum=4529f56e11c85023cd3b815ffd1c2b1e|

Any insights or suggestions would be greatly appreciated. Thanks.

[essiembre]

Can you confirm something changed in the fields you use to create the MD5 checksum? A change anywhere else will not change the checksum.

If that is not the case, can you share your configuration?

[essiembre]

Also, the checksum is created AFTER the document is imported. That means you need to ensure the fields you use to create the checksum are still present in the document after it was imported.

[stejacob]

Morning Pascal,

Our configuration is:

....

<documentChecksummer class="MD5DocumentChecksummer"
                     combineFieldsAndContent="true"
                     keep="true"
                     toField="checksum">
  <fieldMatcher ignoreCase="false"
                ignoreDiacritic="false"
                method="CSV"
                partial="false"
                replaceAll="false">
        title,body_content,description</fieldMatcher>
</documentChecksummer>

.....

<metadataChecksummer class="com.norconex.collector.core.checksum.impl.GenericMetadataChecksummer" keep="true" targetField="metachecksum">
<sourceFieldsRegex>title|description</sourceFieldsRegex>
</metadataChecksummer>

Thanks Pascal.

[essiembre]

Are you limiting the fields present in your documents? Ensure the Importer does not eliminate the fields you represent in your checksummer. You can either share your full config, or I suggest you use the DebugTagger as the last item in your importer section (post-import). This will print all fields remaining in your document after it has been imported. You can then confirm if those fields are there for the Checksummer and if their values have changed.

[stejacob]

Hi Pascal,

Thank you very much for your response. The DebugTagger have been very useful.

We're currently using only the Title field for the checksum. I'll attach a sample config file for your reference.

During an incremental crawl, it processes correctly the first time. However, if we make additional changes and re-crawl, it consistently flags the entries as unmodified and rejects them. This has been a consistent result in our testing. Thank you

demo_site_config.txt

Regards,

Stephen Jacob

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[essiembre]

It looks like this one fell through the cracks. Do you still have the checksum problem?