NVIDIA NeMo Curator 0.6.0

What's changed

• Synthetic Data Generation for Text Retrieval
  • LLM-based Filters
    • Easiness
    • Answerability
  • Q&A Retrieval Generation Pipeline
• Parallel Dataset Curation for Machine Translation
  • Load/Write Bitext Files
  • Heuristic filtering (Histogram, Length Ratio)
  • Classifier filtering (Comet, Cometoid)