o i6@sdZddlZddlZddlmZddlmZmZddlm Z ddZ Gdd d e Z Gd d d Z Gd d d Z GdddeZdS)z9 A reader for corpora whose documents are in MTE format. N)reduce)TaggedCorpusReaderconcat) XMLCorpusViewcCs |||SN)findall)rootpathnsr Q/var/www/edux/Edux_v2/venv/lib/python3.10/site-packages/nltk/corpus/reader/mte.pyxpath s r c@s$eZdZdZdddZdddZdS) MTECorpusViewz0 Class for lazy viewing the MTE Corpus. NcCst||||dSr)r__init__)selffileidtagspec elt_handlerr r r rszMTECorpusView.__init__c Csttddt||||S)NcS|duSrr xr r r z*MTECorpusView.read_block..)listfilterr read_block)rstreamrrr r r rs zMTECorpusView.read_blockr)NN)__name__ __module__ __qualname____doc__rrr r r r rs rc@seZdZdZdddZdZdZdZdZd Z d d Z e d d Z e ddZ e ddZe ddZe ddZe ddZe ddZe ddZe ddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0S)1 MTEFileReaderz Class for loading the content of the multext-east corpus. It parses the xml files and does some tag-filtering depending on the given method parameters. zhttps://www.tei-c.org/ns/1.0z%https://www.w3.org/XML/1998/namespace)teixmlz{https://www.tei-c.org/ns/1.0}z'{https://www.w3.org/XML/1998/namespace}zTEI/text/body/div/div/p/s/(w|c)zTEI/text/body/div/div/p/szTEI/text/body/div/div/pcCs ||_dSr)_MTEFileReader__file_path)r file_pathr r r r3s zMTEFileReader.__init__cCs|jSr)textclseltcontextr r r _word_elt6szMTEFileReader._word_eltcfddt|djDS)Ncg|]}|dqSr)r+.0wr(r r <z+MTEFileReader._sent_elt..*r r r'r r1r _sent_elt:zMTEFileReader._sent_eltcr,)Ncr-r)r6r/sr1r r r2@r3z+MTEFileReader._para_elt..r4r5r'r r1r _para_elt>r7zMTEFileReader._para_eltcCsd|jvr |jdfS|jdkr|jdkr|j|jdfS|jdkr1|jdkr1|jt|jdfStdtdd|jd}| |jdr`|jdkrU|j|jdfS|jt|jdfSdS) Nanamsd universal^-.z.*$) attribr&_MTEFileReader__tags_MTEFileReader__tagsetMTETagConvertermsd_to_universalrecompilesubmatch)r(r)r*tagsr r r _tagged_word_eltBs   zMTEFileReader._tagged_word_eltc*ttddfddt|djDS)NcSrrr rr r r r\rz0MTEFileReader._tagged_sent_elt..cr-r)rLr.r1r r r2]r3z2MTEFileReader._tagged_sent_elt..r4rrr r r'r r1r _tagged_sent_eltX zMTEFileReader._tagged_sent_eltcrM)NcSrrr rr r r rerz0MTEFileReader._tagged_para_elt..cr-r)rOr8r1r r r2fr3z2MTEFileReader._tagged_para_elt..r4rNr'r r1r _tagged_para_eltarPzMTEFileReader._tagged_para_eltcCs$d|jvr |jdfS|j|jdfS)Nlemmar<)rBr&r'r r r _lemma_word_eltjs  zMTEFileReader._lemma_word_eltcr,)Ncr-r)rSr.r1r r r2sr3z1MTEFileReader._lemma_sent_elt..r4r5r'r r1r _lemma_sent_eltqr7zMTEFileReader._lemma_sent_eltcr,)Ncr-r)rTr8r1r r r2wr3z1MTEFileReader._lemma_para_elt..r4r5r'r r1r _lemma_para_eltur7zMTEFileReader._lemma_para_eltcCt|jtjtjSr)rr$r! word_pathr+rr r r wordsy zMTEFileReader.wordscCrVr)rr$r! sent_pathr6rXr r r sents~rZzMTEFileReader.sentscCrVr)rr$r! para_pathr:rXr r r parasrZzMTEFileReader.parascCrVr)rr$r!rWrSrXr r r lemma_wordsrZzMTEFileReader.lemma_wordscC|t_|t_t|jtjtjSr)r!rDrCrr$rWrLrtagsetrKr r r tagged_words  zMTEFileReader.tagged_wordscCrVr)rr$r!r[rTrXr r r lemma_sentsrZzMTEFileReader.lemma_sentscCr`r)r!rDrCrr$r[rOrar r r tagged_sentsrdzMTEFileReader.tagged_sentscCrVr)rr$r!r]rUrXr r r lemma_parasrZzMTEFileReader.lemma_parascCr`r)r!rDrCrr$r]rQrar r r tagged_parasrdzMTEFileReader.tagged_parasN)rrrr r tag_nsxml_nsrWr[r]r classmethodr+r6r:rLrOrQrSrTrUrYr\r^r_rcrerfrgrhr r r r r!"sL          r!c @s:eZdZdZdddddddd d d d d d ZeddZdS)rEzu Class for converting msd tags to universal tags, more conversion options are currently not implemented. ADJADPADVCONJDETNOUNNUMPRTPRONVERBrAX) ASRCDNMQPVrAr@cCs4|ddks |dn|d}|tjvrd}tj|S)z This function converts the annotation from the Multex-East to the universal tagset as described in Chapter 5 of the NLTK-Book Unknown Tags will be mapped to X. Punctuation marks are not supported in MSD tags, so r#r@)rEmapping_msd_universal)tag indicatorr r r rFs  z MTETagConverter.msd_to_universalN)rrrr r staticmethodrFr r r r rEs"rEc@s|eZdZdZdddZddZddd Zdd d Zdd d ZdddZ dddZ dddZ dddZ dddZ dddZdS)MTECorpusReaderz Reader for corpora following the TEI-p5 xml scheme, such as MULTEXT-East. MULTEXT-East contains part-of-speech-tagged words with a quite precise tagging scheme. These tags can be converted to the Universal tagset Nutf8cCst||||d|_dS)a. Construct a new MTECorpusreader for a set of documents located at the given root directory. Example usage: >>> root = '/...path to corpus.../' >>> reader = MTECorpusReader(root, 'oana-*.xml', 'utf8') # doctest: +SKIP :param root: The root directory for this corpus. (default points to location in multext config file) :param fileids: A list or regexp specifying the fileids in this corpus. (default is oana-en.xml) :param encoding: The encoding of the given files (default is utf8) z 00README.txtN)rr_readme)rrfileidsencodingr r r rs zMTECorpusReader.__init__csP|durj}nt|tr|g}tfdd|}tdd|}|s&td|S)Ncs |jvSr)_fileidsrrXr r rs z+MTECorpusReader.__fileids..cSs|dvS)N)z oana-bg.xmlz oana-mk.xmlr rr r r rrz$No valid multext-east file specified)r isinstancestrrprintrrr rXr __fileidss zMTECorpusReader.__fileidsctfdd|DS)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words and punctuation symbols. :rtype: list(str) c$g|]}ttjj|qSr )r!osr join_rootrYr/frXr r r2z)MTECorpusReader.words..r_MTECorpusReader__fileidsrr rXr rYs  zMTECorpusReader.wordscr)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings :rtype: list(list(str)) crr )r!rr rrr\rrXr r r2 rz)MTECorpusReader.sents..rrr rXr r\  zMTECorpusReader.sentscr)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word string :rtype: list(list(list(str))) crr )r!rr rrr^rrXr r r2rz)MTECorpusReader.paras..rrr rXr r^rzMTECorpusReader.parascr)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words, the corresponding lemmas and punctuation symbols, encoded as tuples (word, lemma) :rtype: list(tuple(str,str)) crr )r!rr rrr_rrXr r r2&rz/MTECorpusReader.lemma_words..rrr rXr r_rzMTECorpusReader.lemma_wordsr=r<c<dksdkrtfdd|DStddS)a; :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of tagged words and punctuation symbols encoded as tuples (word, tag) :rtype: list(tuple(str, str)) r>r=c(g|]}ttjj|qSr )r!rr rrrcrrrKrbr r r29 z0MTECorpusReader.tagged_words..Unknown tagset specified.NrrrrrrbrKr rr rc,  zMTECorpusReader.tagged_wordscr)aB :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(list(tuple(str, str))) crr )r!rr rrrerrXr r r2Lrz/MTECorpusReader.lemma_sents..rrr rXr reC  zMTECorpusReader.lemma_sentscr)aH :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of sentences or utterances, each each encoded as a list of (word,tag) tuples :rtype: list(list(tuple(str, str))) r>r=crr )r!rr rrrfrrr r r2_rz0MTECorpusReader.tagged_sents..rNrrr rr rfRrzMTECorpusReader.tagged_sentscr)am :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(List(List(tuple(str, str)))) crr )r!rr rrrgrrXr r r2rrz/MTECorpusReader.lemma_paras..rrr rXr rgirzMTECorpusReader.lemma_parascr)a :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of (word,tag) tuples :rtype: list(list(list(tuple(str, str)))) r>r=crr )r!rr rrrhrrr r r2rz0MTECorpusReader.tagged_paras..rNrrr rr rhxs  zMTECorpusReader.tagged_paras)NNrr)Nr=r<)rrrr rrrYr\r^r_rcrerfrgrhr r r r rs       r)r rrG functoolsrnltk.corpus.readerrrnltk.corpus.reader.xmldocsrr rr!rErr r r r s   %