Add Hooks for PyInstaller #469
Description
This is directly related to #43.
Currently, packaging and running the following minimal example with PyInstaller results in a ValueError.
from tiktoken import get_encoding
get_encoding("o200k_base")This is caused by a hidden import for tiktoken_ext.openai_public not being detected by PyInstaller.
The current workaround for users is to include --hidden-import tiktoken_ext.openai_public in the PyInstaller command.
A proper fix for this would be for tiktoken to include a PyInstaller hook to make sure relevant files are collected. Documentation for adding hooks can be found here: https://pyinstaller.org/en/stable/hooks.html It is quite possible to specify hidden imports in the hook.