New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fasText not work with Chinese webpage #459

Open

yangjingyi opened this issue Dec 31, 2024 · 1 comment

Assignees

Labels

bug

yangjingyi commented Dec 31, 2024

Hi,

I found that jusText do not work with Chinese html data. Because languages = justext.get_stoplists() cannot get stopwords for Chinese.
https://github.com/NVIDIA/NeMo-Curator/blob/main/nemo_curator/download/commoncrawl.py

Could you please add some package or codes to make it work for Chinese data?

Thanks!

yangjingyi added the bug label

sarahyurick self-assigned this

Collaborator

sarahyurick commented Jan 22, 2025

Hi, I can look into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment