PyPI project
tokenizers
100+ indexed releases ยท newest sdist 0.23.1 with 50 indexed members
Files in 0.23.1 tokenizers-0.23.1.tar.gz
- bindings/python/benches/test_tiktoken.py
- bindings/python/conftest.py
- bindings/python/examples/custom_components.py
- bindings/python/examples/example.py
- bindings/python/examples/train_bert_wordpiece.py
- bindings/python/examples/train_bytelevel_bpe.py
- bindings/python/examples/train_with_datasets.py
- bindings/python/scripts/convert.py
- bindings/python/scripts/sentencepiece_extractor.py
- bindings/python/scripts/spm_parity_check.py
- bindings/python/tests/__init__.py
- bindings/python/tests/bindings/__init__.py
- bindings/python/tests/bindings/test_decoders.py
- bindings/python/tests/bindings/test_encoding.py
- bindings/python/tests/bindings/test_models.py
- bindings/python/tests/bindings/test_normalizers.py
- bindings/python/tests/bindings/test_pre_tokenizers.py
- bindings/python/tests/bindings/test_processors.py
- bindings/python/tests/bindings/test_tokenizer.py
- bindings/python/tests/bindings/test_trainers.py
- bindings/python/tests/documentation/__init__.py
- bindings/python/tests/documentation/test_pipeline.py
- bindings/python/tests/documentation/test_quicktour.py
- bindings/python/tests/documentation/test_tutorial_train_from_iterators.py
- bindings/python/tests/implementations/__init__.py
- bindings/python/tests/implementations/test_base_tokenizer.py
- bindings/python/tests/implementations/test_bert_wordpiece.py
- bindings/python/tests/implementations/test_byte_level_bpe.py
- bindings/python/tests/implementations/test_char_bpe.py
- bindings/python/tests/implementations/test_sentencepiece.py
- bindings/python/tests/test_benchmarks.py
- bindings/python/tests/test_freethreaded.py
- bindings/python/tests/test_serialization.py
- bindings/python/tests/utils.py
- py_src/tokenizers/__init__.py
- py_src/tokenizers/decoders/__init__.py
- py_src/tokenizers/implementations/__init__.py
- py_src/tokenizers/implementations/base_tokenizer.py
- py_src/tokenizers/implementations/bert_wordpiece.py
- py_src/tokenizers/implementations/byte_level_bpe.py
- py_src/tokenizers/implementations/char_level_bpe.py
- py_src/tokenizers/implementations/sentencepiece_bpe.py
- py_src/tokenizers/implementations/sentencepiece_unigram.py
- py_src/tokenizers/models/__init__.py
- py_src/tokenizers/normalizers/__init__.py
- py_src/tokenizers/pre_tokenizers/__init__.py
- py_src/tokenizers/processors/__init__.py
- py_src/tokenizers/tools/__init__.py
- py_src/tokenizers/tools/visualizer.py
- py_src/tokenizers/trainers/__init__.py
Indexed releases
- 0.23.1
tokenizers-0.23.1.tar.gz50 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-win_amd64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-musllinux_1_2_x86_64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-musllinux_1_2_armv7l.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_31_riscv64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-macosx_10_12_x86_64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-win_arm64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-win32.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-musllinux_1_2_i686.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-musllinux_1_2_aarch64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-macosx_11_0_arm64.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0.tar.gz49 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-win_amd64.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-musllinux_1_2_x86_64.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-musllinux_1_2_armv7l.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-manylinux_2_31_riscv64.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.23.0rc0
tokenizers-0.23.0rc0-cp310-abi3-macosx_10_12_x86_64.whl16 members - 0.22.2
tokenizers-0.22.2-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.2
tokenizers-0.22.2-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.22.2
tokenizers-0.22.2-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.2
tokenizers-0.22.2-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-win_arm64.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-win32.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.22.2
tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-win_arm64.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-musllinux_1_2_i686.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-musllinux_1_2_aarch64.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.22.2rc0
tokenizers-0.22.2rc0-cp39-abi3-macosx_10_12_x86_64.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-win_amd64.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-musllinux_1_2_x86_64.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-musllinux_1_2_armv7l.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.22.1
tokenizers-0.22.1-cp39-abi3-macosx_11_0_arm64.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0.tar.gz65 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-win32.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-musllinux_1_2_i686.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-musllinux_1_2_aarch64.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.22.1rc0
tokenizers-0.22.1rc0-cp39-abi3-macosx_10_12_x86_64.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-win_amd64.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-musllinux_1_2_x86_64.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-musllinux_1_2_armv7l.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-win_arm64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-win32.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-musllinux_1_2_i686.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-musllinux_1_2_aarch64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.23.1
tokenizers-0.23.1-cp310-abi3-macosx_11_0_arm64.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0.tar.gz50 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-win_amd64.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.22.0
tokenizers-0.22.0-cp39-abi3-macosx_11_0_arm64.whl16 members - 0.22.0rc0
tokenizers-0.22.0rc0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.22.0rc0
tokenizers-0.22.0rc0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-musllinux_1_2_x86_64.whl16 members - 0.21.4
tokenizers-0.21.4.tar.gz65 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-win32.whl16 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-musllinux_1_2_i686.whl16 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-musllinux_1_2_aarch64.whl16 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-musllinux_1_2_armv7l.whl16 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.21.4
tokenizers-0.21.4-cp39-abi3-macosx_10_12_x86_64.whl16 members - 0.21.2
tokenizers-0.21.2-cp39-abi3-win_amd64.whl16 members - 0.21.2
tokenizers-0.21.2-cp39-abi3-musllinux_1_2_x86_64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_31_riscv64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl16 members - 0.23.1rc0
tokenizers-0.23.1rc0-cp310-abi3-macosx_10_12_x86_64.whl16 members
and more releases past the first 100