o i!@szddlmZWneyddlmZYnwddlmZddlmZmZddlm Z m Z ddl m Z Gddde Z d d Zd d Zd dZddZdddZdddZdS)) zip_longest) izip_longest)tee)escapeunescape)Paralleldelayed)tqdmc @sZeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd Zeeeeee e e e eeg ZdS)CJKCharsz An object that enumerates the code points of the CJK characters as listed on http://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Basic_Multilingual_Plane iii.iϤi@iiiiii0iOiei)ioioipiii/ipiiiN)__name__ __module__ __qualname____doc__ Hangul_Jamo CJK_RadicalsPhags_PaHangul_SyllablesCJK_Compatibility_IdeographsCJK_Compatibility_FormsKatakana_Hangul_Halfwidth#Ideographic_Symbols_And_PunctuationTangutKana_SupplementNushuSupplementary_Ideographic_Planerangesr'r'J/var/www/edux/Edux_v2/venv/lib/python3.10/site-packages/sacremoses/util.pyr s4r cstfdddDS)u This checks for CJK character. >>> CJKChars().ranges [(4352, 4607), (11904, 42191), (43072, 43135), (44032, 55215), (63744, 64255), (65072, 65103), (65381, 65500), (94208, 101119), (110592, 110895), (110960, 111359), (131072, 196607)] >>> is_cjk(u'㏾') True >>> is_cjk(u'﹟') False :param character: The character that needs to be checked. :type character: char :return: bool cs,g|]\}}|tko|knqSr')ord).0startend characterr'r( sszis_cjk..) r r r rrrrrrrr)anyr-r'r-r(is_cjkcs  r1cCt|dddddddS)a This function transforms the input text into an "escaped" version suitable for well-formed XML formatting. Note that the default xml.sax.saxutils.escape() function don't escape some characters that Moses does so we have to manually add them to the entities dictionary. >>> input_str = ''')| & < > ' " ] [''' >>> expected_output = ''')| & < > ' " ] [''' >>> escape(input_str) == expected_output True >>> xml_escape(input_str) ')| & < > ' " ] [' :param text: The text that needs to be escaped. :type text: str :rtype: str '"|[])'"|[]entities)rtextr'r'r( xml_escapesrAcCr2)ai This function transforms the "escaped" version suitable for well-formed XML formatting into humanly-readable string. Note that the default xml.sax.saxutils.unescape() function don't unescape some characters that Moses does so we have to manually add them to the entities dictionary. >>> from xml.sax.saxutils import unescape >>> s = ')| & < > ' " ] [' >>> expected = ''')| & < > ' " ] [''' >>> xml_unescape(s) == expected True :param text: The text that needs to be unescaped. :type text: str :rtype: str r8r9r:r;r<)r3r4r5r6r7r=)rr?r'r'r( xml_unescapesrBcCs t|\}}t|dt||S)zp From https://docs.python.org/3/library/itertools.html#recipes s -> (s0,s1), (s1,s2), (s2, s3), ... N)rnextzip)iterableabr'r'r(pairwises   rHNcCst|g|}t|d|iS)ziCollect data into fixed-length chunks or blocks from https://stackoverflow.com/a/16789869/610569 fillvalue)iterr)rEnrIargsr'r'r(groupersrMFcs>|rt|n|}|dkrt|St|dfdd|DS)N)n_jobsc3s|] }t|VqdSN)r)r*linefuncr'r( sz)parallelize_preprocess..)r mapr)rSiterator processes progress_barr'rRr(parallelize_preprocesss rYrP)F) itertoolsr ImportErrorrrxml.sax.saxutilsrrjoblibrrr objectr r1rArBrHrMrYr'r'r'r(s    S#