tokenizers/implementations/bpe.py
6 distinct file contents recorded under this name ยท 60 PyPI records
Python source file Reference
A .py file contains Python source code in plain text. The Python interpreter reads the file and runs the code directly, without a separate compile step. Developers write scripts, modules, and full programs in this format. A text editor can open and change the file. Python is one of the most used programming languages, thus .py files are common on many systems.
A file extension is only a part of the file name. It does not give a guarantee about the format or the content of a file. But on Windows, the extension selects the program that opens the file.
Hashes for this file
2b59a832dea7a20b1eed2b42021330712e0f7129c8f5fa3cc179925853943e06
2528 bytes
- tokenizers
0.2.0
tokenizers-0.2.0-cp35-cp35m-win_amd64.whltokenizers-0.2.0-cp36-cp36m-win_amd64.whltokenizers-0.2.0-cp37-cp37m-win_amd64.whltokenizers-0.2.0-cp38-cp38-win_amd64.whl - tokenizers
0.2.1
tokenizers-0.2.1-cp35-cp35m-win_amd64.whltokenizers-0.2.1-cp36-cp36m-win_amd64.whltokenizers-0.2.1-cp37-cp37m-win_amd64.whltokenizers-0.2.1-cp38-cp38-win_amd64.whl
8d83244e3e0500377a36a31db9864b485df5cfc9b0df5baed4c0612419dd4421
2462 bytes
- tokenizers
0.2.0
tokenizers-0.2.0-cp35-cp35m-macosx_10_13_x86_64.whltokenizers-0.2.0-cp35-cp35m-manylinux1_x86_64.whltokenizers-0.2.0-cp36-cp36m-macosx_10_13_x86_64.whltokenizers-0.2.0-cp36-cp36m-manylinux1_x86_64.whltokenizers-0.2.0-cp37-cp37m-macosx_10_13_x86_64.whltokenizers-0.2.0-cp37-cp37m-manylinux1_x86_64.whltokenizers-0.2.0-cp38-cp38-macosx_10_13_x86_64.whltokenizers-0.2.0-cp38-cp38-manylinux1_x86_64.whl - tokenizers
0.2.1
tokenizers-0.2.1-cp35-cp35m-macosx_10_13_x86_64.whltokenizers-0.2.1-cp35-cp35m-manylinux1_x86_64.whltokenizers-0.2.1-cp36-cp36m-macosx_10_13_x86_64.whltokenizers-0.2.1-cp36-cp36m-manylinux1_x86_64.whltokenizers-0.2.1-cp37-cp37m-macosx_10_13_x86_64.whltokenizers-0.2.1-cp37-cp37m-manylinux1_x86_64.whltokenizers-0.2.1-cp38-cp38-macosx_10_13_x86_64.whltokenizers-0.2.1-cp38-cp38-manylinux1_x86_64.whl
a981817f600badeeeb2cc33e3c62a2c17a34374d9190b50353fa8afa745fdc81
2523 bytes
- tokenizers
0.1.0
tokenizers-0.1.0-cp35-cp35m-win_amd64.whltokenizers-0.1.0-cp36-cp36m-win_amd64.whltokenizers-0.1.0-cp37-cp37m-win_amd64.whltokenizers-0.1.0-cp38-cp38-win_amd64.whl - tokenizers
0.1.1
tokenizers-0.1.1-cp35-cp35m-win_amd64.whltokenizers-0.1.1-cp36-cp36m-win_amd64.whltokenizers-0.1.1-cp37-cp37m-win_amd64.whltokenizers-0.1.1-cp38-cp38-win_amd64.whl
6a29e7d0247e2c2bbbb3a8cd0c80b180e64c0b3745e4c253293dcfc3c811adf6
2457 bytes
- tokenizers
0.1.0
tokenizers-0.1.0-cp35-cp35m-macosx_10_13_x86_64.whltokenizers-0.1.0-cp35-cp35m-manylinux1_x86_64.whltokenizers-0.1.0-cp36-cp36m-macosx_10_13_x86_64.whltokenizers-0.1.0-cp36-cp36m-manylinux1_x86_64.whltokenizers-0.1.0-cp37-cp37m-macosx_10_13_x86_64.whltokenizers-0.1.0-cp37-cp37m-manylinux1_x86_64.whltokenizers-0.1.0-cp38-cp38-macosx_10_13_x86_64.whltokenizers-0.1.0-cp38-cp38-manylinux1_x86_64.whl - tokenizers
0.1.1
tokenizers-0.1.1-cp35-cp35m-macosx_10_13_x86_64.whltokenizers-0.1.1-cp35-cp35m-manylinux1_x86_64.whltokenizers-0.1.1-cp36-cp36m-macosx_10_13_x86_64.whltokenizers-0.1.1-cp36-cp36m-manylinux1_x86_64.whltokenizers-0.1.1-cp37-cp37m-macosx_10_13_x86_64.whltokenizers-0.1.1-cp37-cp37m-manylinux1_x86_64.whltokenizers-0.1.1-cp38-cp38-macosx_10_13_x86_64.whltokenizers-0.1.1-cp38-cp38-manylinux1_x86_64.whl
7d75eab9ef96de5c39cc086e59d0e7cfa5e221929f7bc5648d59d3853a5495c2
2439 bytes
- tokenizers
0.0.13
tokenizers-0.0.13-cp35-cp35m-win_amd64.whltokenizers-0.0.13-cp36-cp36m-win_amd64.whltokenizers-0.0.13-cp37-cp37m-win_amd64.whltokenizers-0.0.13-cp38-cp38-win_amd64.whl
ede120ed27c215473c77dcbd5a32bdd40c96bcf77fba5406ca308e64a6fb8c0d
2375 bytes
- tokenizers
0.0.13
tokenizers-0.0.13-cp35-cp35m-macosx_10_13_x86_64.whltokenizers-0.0.13-cp35-cp35m-manylinux1_x86_64.whltokenizers-0.0.13-cp36-cp36m-macosx_10_13_x86_64.whltokenizers-0.0.13-cp36-cp36m-manylinux1_x86_64.whltokenizers-0.0.13-cp37-cp37m-macosx_10_13_x86_64.whltokenizers-0.0.13-cp37-cp37m-manylinux1_x86_64.whltokenizers-0.0.13-cp38-cp38-macosx_10_13_x86_64.whltokenizers-0.0.13-cp38-cp38-manylinux1_x86_64.whl