o i)@sddlZddlZddlZddlmZmZddlZddlmZm Z ddl m Z m Z m Z ddlmZmZddlmZe r?ddlZeeZGdd d eZdS) N)DictOptional) CONFIG_NAMEPYTORCH_WEIGHTS_NAME)cached_download hf_hub_urlis_torch_available)HfApiHfFolder) Repositoryc@seZdZddZ  d"dedeedefdd Zd d Z e      d#deededeededede deedeedefddZ e     d$deedeedeedeedeededefd d!ZdS)% ModelHubMixincOsdS)a Mix this class with your torch-model class for ease process of saving & loading from huggingface-hub Example:: >>> from huggingface_hub import ModelHubMixin >>> class MyModel(nn.Module, ModelHubMixin): ... def __init__(self, **kwargs): ... super().__init__() ... self.config = kwargs.pop("config", None) ... self.layer = ... ... def forward(self, ...) ... return ... >>> model = MyModel() >>> model.save_pretrained("mymodel", push_to_hub=False) # Saving model weights in the directory >>> model.push_to_hub("mymodel", "model-1") # Pushing model-weights to hf-hub >>> # Downloading weights from hf-hub & model will be initialized from those weights >>> model = MyModel.from_pretrained("username/mymodel@main") N)selfargskwargsrrT/var/www/edux/Edux_v2/venv/lib/python3.10/site-packages/huggingface_hub/hub_mixin.py__init__szModelHubMixin.__init__NFsave_directoryconfig push_to_hubcKstj|ddt|tr.tj|t}t|d}t ||Wdn1s)wYtj|t }| ||rE|j |fi|SdS)a Saving weights in local directory. Parameters: save_directory (:obj:`str`): Specify directory in which you want to save weights. config (:obj:`dict`, `optional`): specify config (must be dict) incase you want to save it. push_to_hub (:obj:`bool`, `optional`, defaults to :obj:`False`): Set it to `True` in case you want to push your weights to huggingface_hub model_id (:obj:`str`, `optional`, defaults to :obj:`save_directory`): Repo name in huggingface_hub. If not specified, repo name will be same as `save_directory` kwargs (:obj:`Dict`, `optional`): kwargs will be passed to `push_to_hub` T)exist_okwN) osmakedirs isinstancedictpathjoinropenjsondumpr_save_pretrainedr)rrrrrrfrrrsave_pretrained.s   zModelHubMixin.save_pretrainedcCs(t|dr|jn|}t||dS)zr Overwrite this method in case you don't want to save complete model, rather some specific layers moduleN)hasattrr%torchsave state_dict)rr model_to_saverrrr"TszModelHubMixin._save_pretrainedTcpupretrained_model_name_or_pathstrict map_locationforce_downloadresume_downloadproxiesuse_auth_token cache_dirlocal_files_onlyc  Ksv|} t|}d} t| ddkr| d\} } | tvr.tt| vr.tj| t} n'zt | t| d}t |||||| |d} Wnt j j yTtdd} Ynw| tvrgtdtj| t}nt | t| d}t |||||| |d}| durt| dd d  }t|}Wdn1swY| d |i|di| }tj||d }|j||d ||S)a Instantiate a pretrained pytorch model from a pre-trained model configuration from huggingface-hub. The model is set in evaluation mode by default using ``model.eval()`` (Dropout modules are deactivated). To train the model, you should first set it back in training mode with ``model.train()``. Parameters: pretrained_model_name_or_path (:obj:`str` or :obj:`os.PathLike`, `optional`): Can be either: - A string, the `model id` of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like ``bert-base-uncased``, or namespaced under a user or organization name, like ``dbmdz/bert-base-german-cased``. - You can add `revision` by appending `@` at the end of model_id simply like this: ``dbmdz/bert-base-german-cased@main`` Revision is the specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so ``revision`` can be any identifier allowed by git. - A path to a `directory` containing model weights saved using :func:`~transformers.PreTrainedModel.save_pretrained`, e.g., ``./my_model_directory/``. - :obj:`None` if you are both providing the configuration and state dictionary (resp. with keyword arguments ``config`` and ``state_dict``). cache_dir (:obj:`Union[str, os.PathLike]`, `optional`): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (:obj:`bool`, `optional`, defaults to :obj:`False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download (:obj:`bool`, `optional`, defaults to :obj:`False`): Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. proxies (:obj:`Dict[str, str], `optional`): A dictionary of proxy servers to use by protocol or endpoint, e.g., :obj:`{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(:obj:`bool`, `optional`, defaults to :obj:`False`): Whether or not to only look at local files (i.e., do not try to download the model). use_auth_token (:obj:`str` or `bool`, `optional`): The token to use as HTTP bearer authorization for remote files. If :obj:`True`, will use the token generated when running :obj:`transformers-cli login` (stored in :obj:`~/.huggingface`). model_kwargs (:obj:`Dict`, `optional`):: model_kwargs will be passed to the model during initialization .. note:: Passing :obj:`use_auth_token=True` is required when you want to use a private model. N@)filenamerevision)r3r/r1r0r4r2z(config.json NOT FOUND in HuggingFace Hubz$LOADING weights from local directoryrzutf-8)encodingr)r.)r-r)r'devicelensplitrlistdirrrrrrrequests exceptionsRequestExceptionloggerwarningprintrrr loadupdateload_state_dicteval)clsr,r-r.r/r0r1r2r3r4 model_kwargsmodel_idr8 config_file config_url model_file model_urlr#rmodelr)rrrfrom_pretrained[sb7      zModelHubMixin.from_pretrained add modelrKrepo_urlcommit_message organizationprivatereturncCsN|dur|}t}|durtj||||ddd}t|||d}|j|dS)a Parameters: save_directory (:obj:`Union[str, os.PathLike]`): Directory having model weights & config. model_id (:obj:`str`, `optional`, defaults to :obj:`save_directory`): Repo name in huggingface_hub. If not specified, repo name will be same as `save_directory` repo_url (:obj:`str`, `optional`): Specify this in case you want to push to existing repo in hub. organization (:obj:`str`, `optional`): Organization in which you want to push your model. private (:obj:`bool`, `optional`): private: Whether the model repo should be private (requires a paid huggingface.co account) commit_message (:obj:`str`, `optional`, defaults to :obj:`add model`): Message to commit while pushing Returns: url to commit on remote repo. NT)rUrV repo_typer) clone_fromr2)rT)r get_tokenr create_repor r)rrKrSrTrUrVtokenreporrrrs zModelHubMixin.push_to_hub)NF)Tr+FFNNNF)NNrRNN)__name__ __module__ __qualname__rstrrrboolr$r" classmethodrrQ staticmethodrrrrrr sz &  or )r loggingrtypingrrr? constantsrr file_downloadrrr hf_apir r repositoryr r' getLoggerr^rBobjectr rrrrs