Does not work on front-end in Next.js projects

[Johnrobmiller]

Even after adding fs via yarn add fs, I still get this error. ):
Screenshot:

[Clueed]

Yes, this is a node module. fs and path are not available in the browser.

You can get around that fairly easily. vocab.bpe needs be saved as a json.

const bpe_file = fs.readFileSync(path.join(__dirname, './vocab.bpe'), 'utf-8');
fs.writeFileSync('./vocab_bpe.json', JSON.stringify(bpe_file))

Then the two jsons can be loaded directly.

//const fs = require('fs')
//const path = require('path');

//const encoder = JSON.parse(fs.readFileSync(path.join(__dirname, './encoder.json')));
//const bpe_file = fs.readFileSync(path.join(__dirname, './vocab.bpe'), 'utf-8');

const encoder = require('./encoder.json')
const bpe_file = require('./vocab_bpe.json')

Modifying the module itself didn't work for me. I get errors around process not being defined in node_module/path even through it's no longer using path and doesn't have any dependencies that do. Coping the code with the modification above into react with import { encode } from './gpt-3-encoder-react/Encoder' works perfectly fine. Tested on react 18.2.0.

Zip with both jsons and modified code.

---

There are actually two pull requests that address this issue: #33 and #30. I think this issue can be closed and one of the two should be merged.

[nem035]

If interested, a temporary fix would be to use https://www.npmjs.com/package/@nem035/gpt-3-encoder until the maintainers merge #33

npm i @nem035/gpt-3-encoder

[syonfox]

YOu can probably use the browser version from my 1.4 release. This removed all node dependencies by including data as js files. THis can be used directly or in the browserified version.

I have just pushed a new update it should be getting pretty stable. let me know if it works for you

https://syonfox.github.io/GPT-3-Encoder/browser.html

[niieani]

You can also use my fork gpt-tokenizer. See the playground here that works fully in the browser.

[ruckc]

@syonfox - getting this in a frontend svelte project with the latest release - 1.1.4.

p.js:16 ReferenceError: __dirname is not defined
    at node_modules/.pnpm/[email protected]/node_modules/gpt-3-encoder/Encoder.js (Encoder.js:5:54)
    at __require (chunk-ENY474MH.js?v=4b935346:5:50)
    at node_modules/.pnpm/[email protected]/node_modules/gpt-3-encoder/index.js (index.js:1:28)
    at __require (chunk-ENY474MH.js?v=4b935346:5:50)
    at index.js:6:1

[syonfox]

is that not built in node if your in the browser environment you will need node_modules/gptoken/browser.js copy to relevant location in build

just include this as a regular file and use let c = gptoken.count(str)

also npm install gptoken