o ii.@sddlmZddlmZddlZddlZddlZddlm Z ddl Z ddl m Z ddl mZddlmZddlZddlZddlZeZejejdd eeZejejed Ze ed  Z!e"e!Z#Wdn1sqwYed d e#ddZ$ze j%&dWne'ye (dYnwze j%&dWne'ye (dYnwddZ)e)Z*ddZ+ddZ,ddZ-d:ddZ.ddZ/d d!Z0d"d#Z1d$d%Z2d&d'Z3d(d)Z4d;d+d,Z5d)build)parse_durationN)fuzz) PorterStemmer) stopwords) HttpErrorz4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformatzApiConfig.jsonryoutubev3API_KEY) developerKeyztokenizers/punktpunktzcorpora/stopwordsrcCsLtjtjtd}t| }t|WdS1swYdS)NzQuotaConfig.json)ospathjoindirname__file__openjsonload)quota_config_pathquota_config_filer*/var/www/edux/Edux_v2/service/VideoCall.pyload_quota_config's $rcCs dt_dS)Nr) thread_local quota_usagerrrrinit_thread_local.s rcCstjt|d7_dS)Ncost)rr quota_config) api_methodrrr update_quota1sr#cCs|}d|d}|S)z Modifies the search criteria by adding appropriate keywords for YouTube API searches. Args: search_criteria (str): The original search criteria. Returns: str: Modified search criteria with additional keywords. ")stripsearch_criteriarrrmodify_search_criteria4s r(cs^tttdfdd}||}||}|d|r#d|nd}t||}|S)aL Calculates a fuzzy match score based on the presence of search criteria keywords in the video title and tags. :param search_criteria: The search query string. :param video_title: The title of the video. :param video_tags: A list of tags associated with the video. :return: Match score as a percentage. englishcs*|}fdd|D}d|S)Ncsg|] }|vr|qSr)stem.0wordstemmer stop_wordsrr RzHcalculate_fuzzy_match_score..preprocess_text.. )lowersplitr)textwords stemmed_wordsr.rrpreprocess_textPs  z4calculate_fuzzy_match_score..preprocess_textr3)rsetrr7rrtoken_set_ratio)r' video_title video_tagsr9search_criteria_stemmedvideo_title_stemmed video_info match_scorerr.rcalculate_fuzzy_match_scoreDs  rCcCs>tjd|d}td|dgr|dgdSiS)z7 Fetches channel information from YouTube API. zsnippet,statisticspartid channels.listitemsr)r channelslistexecuter#get) channel_idchannel_responserrrfetch_channel_info`s rOc Csid|d|ddddddddddd dd dd dd dd ddddddddddd||ddddgdddd gS)z, Creates header row for Excel file. Search Criteria Search Type%Matchr:Full Playlist NameSplit Playlist1Split Playlist2Split Playlist3Split Playlist4Split Playlist5 Full Title Split Title1 Split Title2 Split Title3 Split Title4 Split Title5 Video Link Published At) Channel TitleChannel Total VideosLengthViewsLikesCommentsTags DescriptionVideo LanguageLicenser)r' search_type channel_title total_videosrrrcreate_header_rowks^     rnc Cs|d}|d}|d}|d} |d} |d} | dd} |dd} | d gd t| | d gd t| |d }|d }|d }| d}|dd}|dd}|dd}|dd }|dg}| dd }|dd }t|}t|d}t|d}d|}|durt|}|didd}|didd}id|d|d|d | d!d"| d#d$| d%d&| d'd(| dd)| d*| d!d+| d#d,| d%d-| d'd.| dd/|d0|d1||||d2|d3|||||||d4 S)5z* Processes a single video detail. rFsnippet statisticscontentDetailsstatustitle|r: publishedAt channelId channelTitleduration viewCount0 likeCount commentCount descriptiontagslicensedefaultAudioLanguage<z https://www.youtube.com/watch?v=NUnknown Channel videoCountUnknownrPrQrSrTrrUrVrWrXrYrZr[r\r]r^r_r`ra:02d) z Channel IDrbrcrdrerfrgrhrirj)r5extendlenrLrint total_secondsrO) video_detail group_namerMrmr'rkvideo_idrorpcontent_detailsrrr=title_segmentsplaylist_segmentsvideo_published_atvideo_channel_idvideo_channel_title video_lengthvideo_view_countvideo_like_countvideo_comment_countrrrlanguagerzminutesseconds video_linkchannel_info_cachedrlrrrprocess_single_videos               rc sjr tdSfdd|dkrKdjvrIdfddd<jdd d jd d j}|| d | | d|S|dkrTd jvrdjvrt | d fddd<i| D]\}}ddt|D|<qsjfddddd<fdd| D}dfddd<d d}d |d<d |d<dd} j| ddd<jgdgdd jd d ddg} fd d!| D} | rj| d"j}d|vrTd|vrTd|vrTd|vrTd |vrT|| d | | d|| d | | d|| d | | d|| d | | d|S)#a Processes the DataFrame based on the search type. Args: df (pandas.DataFrame): The DataFrame containing video details. search_type (str): Type of search ('Videos' or 'Playlists'). grouped_video_details (dict): Dictionary of video details grouped by playlist. search_criteria (str): The criteria used for searching videos. Returns: pandas.DataFrame: The processed DataFrame. z6No videos passed filtering - returning empty DataFramec sz=t|tr |s WdS|rsWdSddD}|s(WdStfdd|D}|t|d}|WSty\}ztd|dt|WYd}~dSd}~ww) NrcSs(g|]}t|dkr|qS)r)rr%r4)r,wrrrr1s(zOprocess_based_on_search_type..calculate_match_score..c3s|] }|vrdVqdS)rNrr+ text_lowerrr szNprocess_based_on_search_type..calculate_match_score..dz#Error calculating match score for 'z': ) isinstancestrr%r4r5sumr Exceptionprint)r6 search_wordsmatchesmatch_percentageer&rrcalculate_match_scores" z;process_based_on_search_type..calculate_match_scoreVideosrYc|SNrxrrrz.process_based_on_search_type..rRF)by ascendingT)droprS Playlistscs |Sr)indexr)original_playlist_orderrrrs Original Playlist OrdercSsi|] \}}|dd|qS)rorsr)r,ivideorrr r2z0process_based_on_search_type..cs|d|dS)NrSrYrrow) video_ordersrrrsr)axisOriginal Video Ordercsi|]}||qSrr)r,playlistrrrrscrrrrrrrrrzIndividual Video %MatchAverage Video %MatchPlaylist %MatchcSs,|ddkr |dS|dd|ddS)Nrrg?rrrrrrcalculate_combined_score!s z>process_based_on_search_type..calculate_combined_score)rRrr)FTTcsg|] }|jvr|qSrcolumns)r,col)dfrrr1/sz0process_based_on_search_type..r)emptyloggerwarningrapply sort_values reset_indextolistinsertrpoprJkeysrH enumerategroupbymeanto_dictmapr) rrkgrouped_video_detailsr'rrvideosplaylist_scoresavg_video_scoresr temp_columnsexisting_temp_columnsr)rrrr'rrprocess_based_on_search_typesX    -     2rcCstd}tj|dd}|j|ddd|jd}|jd}|D]\}}|d}|j|d|||d q ||S) z3 Writes processed DataFrame to Excel file. video_details.xlsx xlsxwriterengineFSheet1r sheet_namer_r)string) pd ExcelWriterto_excelsheetsrget_lociterrows write_urlclose)rexcel_file_pathwriter worksheetvideo_link_col_idxidxrrrrrwrite_excel_file=s  rc Csg}d}d}d}|dur$t|}|didd}|didd}|sGt||||}t|}tjd d d } |j| d d d| |Sd} d} |D]\} } | D]}| d7} |r|di}|dd}|dd}|d| }t | |}|dkrt d|d|qU| d7} z|t|| ||||WqUty}z|jjdkrdt|vrWYd}~qUd}~wty}zt d|WYd}~qUd}~wwqO|r|rt d| d| d|dn t d| dt|}t||||}t||S)aA Processes video details from a dictionary of video IDs grouped by playlist. Args: grouped_video_details (dict): A dictionary where keys are playlist titles and values are lists of video details dictionaries. channel_id (str): The ID of the YouTube channel. search_criteria (str): The criteria used for searching videos. search_type (str): Type of search ('Videos' or 'Playlists'). Returns: pandas.DataFrame: A DataFrame containing video details. Raises: HttpError: If there's an error in the API request. Nr:rorsrrprrrrrFrrrrrr3rzFiltered out video: 'z' - fuzzy score  quotaExceededz$Unexpected error processing video - zSmart keyword filtering: kept z out of z videos for: ''z)No keyword filtering applied - processed z videos)rOrLrnr DataFramerrrrHr4rr<rdebugappendrrresprrrr exceptionr%inforr)rrMr'rkvideo_details_listrrlrmrroriginal_count kept_countr video_detailsrrorsrr6scoreherrrrprocess_video_detailsOsd       r 2c Cstd|d|g}z|rjtdd}|dus!t||kritjd|r1td|t|ndt|dd||||d }t d | d g}|sMn| d d |D| d}|s_n |dus!t||ks!nXtdd}|dus{t||krtjd|rtd|t|ndt|dd|||d}t d | d g}|sn| dd |D| d}|sn |dus{t||ks{tdt|d|siWSd|dur|n|d|iWSt y} z t d| d} ~ ww)u Updated video discovery: - If channel_id provided → fetch from channel uploads playlist (reliable) - Else → fallback to search API (original behavior) Returns SAME format: {"Video Without Playlist": [ids]} z$Starting video discovery | Channel: z | Criteria: zHUsing Search API with channelId filter for channel-specific video searchNror r relevance) rE maxResultsqtypeorderrx videoLicense videoDuration pageToken search.listrHcSg|]}|ddqSrFvideoIdrr,itemrrrr1z-get_video_ids_only_videos.. nextPageTokenz%No channel provided, using search API)rErrrrrrrcSrrrrrrrr1rz Discovered z video IDszVideo Without PlaylistzError during video discovery: )rrrr searchrJminr(rKr#rLrrr) r' max_resultsrMrr all_video_idsnext_page_tokensearch_responserHrrrrget_video_ids_only_videossz        (r$r c Cstd|dd}ddd}d}g}d}i} td|d  td |||||} | d }| d }|D];} | d} | d} td| || |}|| | <|t|7}tdt|d| d||durv||krvnq;|r|dur||krtd|d | Sq#)a Retrieves video IDs from playlists based on the given search criteria and parameters. Args: search_criteria (str): The search query string. max_videos_total (int, optional): Maximum total number of videos to fetch. Defaults to 100. max_videos_per_playlist (int, optional): Maximum number of videos to fetch per playlist. Defaults to 10. channel_id (str, optional): Optional parameter to restrict searches to a specific channel. Defaults to None. Returns: dict: A dictionary where keys are playlist titles and values are lists of video IDs. Raises: HttpError: If there's an error in the API request. z4Starting playlist video ID retrieval with criteria: cSstd|td|}g}d} tjd|||d}tddd |d gD}| ||d }|r@t ||krAnqtd t |d ||S)a% Fetches video IDs from a specific playlist. Args: playlist_id (str): The ID of the playlist. max_videos_per_playlist (int): Maximum number of videos to fetch per playlist. Returns: list: List of video IDs in the playlist. zFetching videos from playlist: r NTro)rEr playlistIdrplaylistItems.listcSsg|] }|dddqS)ro resourceIdrrrrrrr1.zUget_video_ids_playlist_videos..fetch_videos_from_playlist..rHrFetched  videos from playlist: ) rrrr playlistItemsrJrKr#rLrr) playlist_idmax_videos_per_playlistmax_results_per_requestr! page_tokenplaylist_items_responsevideo_ids_in_playlistrrrfetch_videos_from_playlists,   zAget_video_ids_playlist_videos..fetch_videos_from_playlistNc Sstd|d|d||dur=tjddt|dd||d }td |d g}d d |D}||ddSt jd|d|d}td|d g}dd |D}||ddS)a Fetches playlists based on the given criteria. Args: next_page_token (str, optional): Token for pagination. Defaults to None. channel_id (str, optional): Channel ID to restrict searches. Defaults to None. search_criteria (str, optional): Search query string. Defaults to None. Returns: dict: Dictionary containing playlists and next page token. z"Fetching playlists with criteria: z, channel_id: z, page_token: Nror rr)rErrrrrxrrrHcSs&g|]}|dd|dddqS)rFr&rorsrFrsrrrrrr1Xs&zJget_video_ids_playlist_videos..fetch_playlists..r) playlistsr")rErxrrz playlist.listcSs"g|] }|d|dddqS)rFrorsr4rrrrrr1fs") rrr rrJr(rKr#rLr5)r"rMr'r#r5custom_playlistsplaylist_responserrrfetch_playlists;s:   z6get_video_ids_playlist_videos..fetch_playlistsrz5Starting main loop to fetch videos (max_videos_total=)TzFetching playlists with token: r"r5rFrszProcessing playlist: r*r+z. Total fetched: zCompleted fetching 'z' playlist videos)NNN)rrrr)r'max_videos_totalr.rMr3r8total_videos_fetched all_playlistsr"playlist_videosplaylistandtokenrr-playlist_titler2rrrget_video_ids_playlist_videoss: *.    r@c s~tdgt}|D]}|D]}||vr#|||qtd|dqqfddtdtdD}tdt|d g}|D];}d |}t d |d dd t j d|d}td||dgt dt|dgdqLi} |D]\} fdd|D| | <t dt| | d| qtdt|dt| d| S)a Processes video details from a dictionary of video IDs grouped by playlist. Args: videos (dict): A dictionary where keys are playlist titles and values are lists of video IDs. Returns: dict: A dictionary where keys are playlist titles and values are lists of video detail dictionaries. Raises: HttpError: If there's an error in the API request. z!Starting video details processingz Video id 'z' already processedcsg|] }||dqS)r r)r,r)r!rrr1r)z%get_video_details..rr zCreated z chunks of video IDs,zFetching details for chunk: Nz...z(snippet,statistics,contentDetails,statusrDz videos.listrHr*z video detailscsg|] }|dvr|qS)rFr)r,detail) video_idsrrr1r)zGrouped z videos for playlist: z Processed z video details across z playlists)rrr;valuesraddrrangerrrr rrJrKr#rrLrH) rprocessed_video_idsr=rall_video_ids_chunksrchunk chunk_idsvideo_details_responserr?r)r!rCrget_video_detailss>        rLc CsRtd|d|dtjd|d}td|dg}|s-td|iS|d dd d }td |g}d } |rL|t |nd}t d|}t jd|||d} td| dgD] } | | ddqjtdt |d|rt ||krtd|n | d}|snqDtdt |d|iS)ac Retrieves video IDs from a YouTube channel via its uploads playlist. No search criteria or filters are applied. Args: channel_id (str): The YouTube channel ID. max_results (int, optional): Maximum number of videos to fetch. If None, fetches all videos. Returns: dict: {"All Channel Videos": [video_id, ...]} zFetching videos for channel: z# via uploads playlist (max_results=r9rqrDrGrHz4Channel not found or missing contentDetails for ID: rrelatedPlaylistsuploadszFound uploads playlist ID: NTr )rEr&rrr'rr*z videos so far...zReached max_results cap of rz#Total videos fetched from channel: zAll Channel Videos) rrr rIrJrKr#rLrrrr,r) rMr rNrHuploads_playlist_idr!r" remaining page_sizer7rrrrget_all_channel_videossP    rRcCstd|tjdd|dd}tdd|vr9t|ddkr9|ddd d }td ||Std |d S)a Retrieves the channel ID for a given channel name. Args: channel_name (str): The name of the YouTube channel. Returns: str or None: The channel ID if found, None otherwise. Raises: HttpError: If there's an error in the API request. zSearching for channel: rorvchannel)rErrrrrHrrFrxzFound channel ID: z!No channels found matching name: N) rrr rrJrKr#rr) channel_namer#rMrrrget_channel_id_from_name s rUc Csztd|d|d|d|d|d|d|ttd|r)t|}|d kr;td t|||||}n0|d krLtd t||||}n|d krc|sVtdd}tdt||}ntd|dtdt|}tdt ||||t j fWSty} z t d| d} ~ wt y} zt d| | jjdkrdt| vrt dd} ~ wty} z t d| d} ~ ww)a Fetches video metadata based on the given search criteria and parameters. Args: search_type (str): Type of search. Can be 'Videos', 'Playlists', or 'AllChannelVideos'. search_criteria (str): The search query string. max_results (int, optional): Maximum number of results to return. Defaults to 1000. max_videos_per_playlist (int, optional): Maximum number of videos to fetch per playlist. Defaults to 50. channel_id (str, optional): Optional parameter to restrict searches to a specific channel. Defaults to None. videoLicense (str, optional): Filter videos by license. Defaults to None. videoDuration (str, optional): Filter videos by duration. Defaults to None. Returns: tuple: A tuple containing the processed video details and the quota usage. Raises: ValueError: If the search_type is invalid or if channel_id is None for 'AllChannelVideos' search type. HttpError: If there's an error in the API request. zsf            $Ie X W ?> 'R