o iH@sddlmZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZddlZejdd krOddlZddlZejZeed e ZGd d d eZGd ddeZd dgZdS))print_functionN) defaultdictCounter)partial)chain) text_type) Perluniprops)parallelize_preprocessgrouperzTYou should really be using Python3!!! Tick tock, tick tock, https://pythonclock.org/cseZdZdZededZededZ ededZ d$fdd Z d%d d Z  d&d dZ  d&ddZ  d&ddZ  d&ddZd'ddZd(ddZeddZddZddZd d!Zd"d#ZZS))MosesTruecaserz This is a Python port of the Moses Truecaser from https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/train-truecaser.perl https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/truecase.perl Lowercase_LetterUppercase_LetterNutf8csntt|td|j|j|j|_ td|_ hd|_ hd|_ ||_ ||_|r5|||_dSdS)a. :param load_from: :type load_from: :param is_asr: A flag to indicate that model is for ASR. ASR input has no case, make sure it is lowercase, and make sure known are cased eg. 'i' to be uppercased even if i is known. :type is_asr: bool z[{}{}{}]z(<.*(?<=>))(.*)((?=]*>).!:?[]'""'([N)superr __init__recompileformatrrTitlecase_LetterSKIP_LETTERS_REGEXXML_SPLIT_REGXSENT_ENDDELAYED_SENT_STARTencodingis_asr _load_modelmodel)self load_fromr*r) __class__N/var/www/edux/Edux_v2/venv/lib/python3.10/site-packages/sacremoses/truecase.pyr -s      zMosesTruecaser.__init__FcCsd}g}t|D]L\}}td|rq||jvrq|s#||jvr#d}q|j|s,d}qd}|s3d}n|rD|dr>d}n|dkrDd}d}|dkrT||||fq|S)z This function checks through each tokens in a sentence and returns the appropriate weight of each surface token form. T (<\S[^>]*>)Frg?) enumerater!searchr(r'r%islowerappendlower)r-tokenspossibly_use_first_token is_first_wordtruecase_weightsitokencurrent_word_weightr1r1r2learn_truecase_weightsTs4    z%MosesTruecaser.learn_truecase_weightsr4c Cshtt}t|j|d}tt||||d}|D]\} } } || | | 7<q|r/|||||S)aN :param document_iterator: The input document, each outer list is a sentence, the inner list is the list of tokens for each sentence. :type document_iterator: iter(list(str)) :param possibly_use_first_token: When True, on the basis that the first word of a sentence is always capitalized; if this option is provided then: a) if a sentence-initial token is *not* capitalized, then it is counted, and b) if a capitalized sentence-initial token is the only token of the segment, then it is counted, but with only 10% of the weight of a normal token. :type possibly_use_first_token: bool :returns: A dictionary of the best, known objects as values from `_casing_to_model()` :rtype: {'best': dict, 'known': Counter} )r; progress_bar)rrrrArr _save_model_from_casing_casing_to_model) r-document_iteratorsave_tor; processesrCcasingtrain_truecaser token_weightslowercase_token surface_tokenweightr1r1r2_trains  zMosesTruecaser._traincCs"d|_|j|||||d|_|jS)z\ Default duck-type of _train(), accepts list(list(str)) as input documents. NrB)r,rO)r- documentsrGr;rHrCr1r1r2trains zMosesTruecaser.traincCsbt||jd}dd|D}Wdn1swYd|_|j|||||d|_|jS)zj Duck-type of _train(), accepts a filename to read as a `iter(list(str))` object. r)cs|]}|VqdSNsplit.0liner1r1r2  z1MosesTruecaser.train_from_file..NrB)openr) readlinesr,rO)r-filenamerGr;rHrCfinrFr1r1r2train_from_files zMosesTruecaser.train_from_filecCs4dd|D}d|_|j|||||d|_|jS)zm Duck-type of _train(), accepts a file object to read as a `iter(list(str))` object. csrSrTrUrWr1r1r2rZr[z8MosesTruecaser.train_from_file_object..NrB)r]r,rO)r- file_objectrGr;rHrCrFr1r1r2train_from_file_objects z%MosesTruecaser.train_from_file_objectc Cs"td}t|ds J|d}g}||}t|D]m\}} td| r*|| q| dks3| dr9|| qtd| \} } |j rJ| } |j d | d} |j d  | d} |re| re| } n | rn|rk| n| } n| rr| } | | } || | |j v}| |jvrd }q|rd |S|S) a  Truecase a single sentence / line of text. :param text: A single string, i.e. sentence text. :type text: str :param use_known: Use the known case if a word is a known word but not the first word. :type use_known: bool zV Use Truecaser.train() to train a model. Or use Truecaser('modefile') to load a model.r,Tr3|z ^([^\|]+)(.*)bestNknownF )strhasattr split_xmlr5r!r6r8 startswithgroupsr*r9r,getr'r(join) r-text return_str use_knowncheck_model_messager<truecased_tokensr:r>r? other_factors best_case known_caser1r1r2truecasesB        zMosesTruecaser.truecaseTccsbt||jd}|D]}||}|rd|n|Vq WddS1s*wYdS)NrRrf)r\r)rvstriprm)r-r^ror_rYrrr1r1r2 truecase_file>s"zMosesTruecaser.truecase_filecCs(|}g}|rtd|}td|}td|}|ra|\}}td|rWt|dkrWtd|drW|d|7<td|}|rV|d|d 7<|d }n||d |}n'|rq||d |d }n|r||d |d }ntd ||d|d<|s|S) a  Python port of split_xml function in Moses' truecaser: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/truecaser.perl :param line: Input string, should be tokenized, separated by space. :type line: str z^\s*(<\S[^>]*>)(.*)$z^\s*([^\s<>]+)(.*)$z^\s*(\S+)(.*)$z^\Srz\|$z ^(\|+)(.*)$r4rfzERROR: huh? {}) rwr!r6rklengroupr8 Exceptionr#)rYr:has_xml is_non_xml xml_cognates potential_xml line_next is_factorr1r1r2riEsF           &zMosesTruecaser.split_xmlc Csli}t}|D]&}||}|dd||<|js-|ddD] \}}||d7<q q|||d}|S)zg :returns: A tuple of the (best, known) objects. :rtype: tuple(dict, Counter) rr4N)rdrerI)r most_commonr*) r-rIrdre token_lowerr:r?countr,r1r1r2rExs  zMosesTruecaser._casing_to_modelcCs||jd|dS)NrI)rDr,)r-r^r1r1r2 save_modelszMosesTruecaser.save_modelc Cst|d|jdI}|D]=}t||}g}t||D]\}\}}|dkr1d|||} nd|||} || qtd |d|dq Wd d S1sTwYd S) a" Outputs the truecaser model file in the same output format as https://github.com/moses-smt/mosesdecoder/blob/master/scripts/recaser/train-truecaser.perl :param casing: The dictionary of tokens counter from `train()`. :type casing: default(Counter) wrRrz {} ({}/{})z{} ({})rf )endfileN) r\r)sumvaluesr5rr#r8printrm) r-rIr^foutr?total_token_count tokens_countsr>r out_tokenr1r1r2rDs "z&MosesTruecaser._save_model_from_casingcCstt}t||jd1}|D]&}|}t|dD]\}}|ddd}t||||<qqWdn1s?wY| |S)z Loads pre-trained truecasing file. :returns: A dictionary of the best, known objects as values from `_casing_to_model()` :rtype: {'best': dict, 'known': Counter} rRrz/rz()N) rrr\r)rwrVr intr9rE)r-r^rIr_rYr?rr1r1r2r+s  zMosesTruecaser._load_model)NNr)F)NFr4FFF)T)__name__ __module__ __qualname____doc__rrm perlunipropscharsrrr$r rArOrQr`rbrvrx staticmethodrirErrDr+ __classcell__r1r1r/r2r !sF '1 -    H 2r cs&eZdZfddZdddZZS)MosesDetruecasercs0tt|hd|_hd|_hd|_dS)Nrr>#al-.+el-.+aasatbebyinisofontoandanyforhisitsnotoffthewasfromlastthanthiswerewillwithaftertheirwhichduringagainstbecausebetween)rrr r'r( ALWAYS_LOWERr-r/r1r2r s   zMosesDetruecaser.__init__Fcsg}d}|D](}|r|dd|ddn|}|||jvr)d}q|jvr0d}q|r<fdd|D}|rCd|S|S)z Detruecase the translated files from a model that learnt from truecased tokens. :param text: A single string, i.e. sentence text. :type text: str TNr4Fcs6g|]}|jvr |n |dd|ddqS)Nr4)rupper)rXr?rr1r2 s(z/MosesDetruecaser.detruecase..rf)rVrr8r'r(rm)r-rn is_headlinero cased_tokenssentence_startr?r1rr2 detruecases  $    zMosesDetruecaser.detruecaser)rrrr rrr1r1r/r2rs 6r) __future__rr! collectionsrr functoolsr itertoolsrsixrsacremoses.corpusrsacremoses.utilr r sys version_infoiowarningsr\warnrgrobjectr r__all__r1r1r1r2s2      T