[PYTHON] A story I was addicted to trying to get a video url with tweepy

Introduction

The latest images and videos of actress Mayu Matsuoka are automatically sent every day. While creating a LINE Bot, I was addicted to trying to get the image / video URL from tweepy, so I will share it.

Looking back on it now, I spent a lot of wasted time, but if I write an article for the time being, that wasted time will be rewarded a little! !! I will write it.

Original code

search_tweets.py

import os
import tweepy
from datetime import datetime, date, timedelta
from dateutil.relativedelta import relativedelta

consumer_key = os.getenv('TWITTER_CONSUMER_KEY')
consumer_secret = os.getenv('TWITTER_CONSUMER_SECRET')
access_token = os.getenv('TWITTER_ACCESS_TOKEN')
access_token_secret = os.getenv('TWITTER_ACCESS_TOKEN_SECRET')
bearer_token = os.getenv('TWITTER_ACCESS_TOKEN_SECRET')


#Search for tweets related to Mayu Matsuoka on twitter and get the URL
def search_tweets():
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)

    yesterday = datetime.strftime(datetime.today() - relativedelta(days=1), f"%Y-%m-%d")
    #Omit retweets, 10 likes or more, yesterday-today, 1 or more retweets, twitter search on the condition that there are images and videos
    q = f'#Mayu Matsuoka OR Mayu Matsuoka filter:media exclude:retweets min_faves:10 since:{yesterday} min_retweets:1'

    #The problematic part
    tweets = tweepy.Cursor(api.search, q=q).items(20)

    contents = []
    for tweet in tweets:
        print(tweet.text)
        print(tweet.extended_entities)
        try:
            media = tweet.extended_entities['media']
            for m in media:
                print(m)
                preview = m['media_url_https']
                if m['type'] == 'video':
                    origin = [variant['url'] for variant in m['video_info']['variants'] if variant['content_type'] == 'video/mp4'][0]
                else:
                    origin = m['media_url_https']

                #Formatted for sending on LINE Bot
                content = {'preview': preview, 'origin': origin, 'type': m['type']}
                contents.append(content)

            print('--------------------------------------------')
        except:
            print('Error')
            print('--------------------------------------------')
    return contents


if __name__ == "__main__":
    search_tweets()

** With this code, I should have been able to get the URL of both tweets with images and videos. .. .. .. .. ** **

I can't get the video URL as I expected

I can search for tweets, but I notice that some videos URLs can be obtained and some cannot.

There may be an error in the try. .. ..

When I printed each tweet, I found some tweets without extended_entities.

for tweet in tweets:
        print(tweet.text)
        #Cause part
        # extened_Since there are no entities, an error occurs here except:I was going to.
        print(tweet.extended_entities)
        try:
            media = tweet.extended_entities['media']

When I looked it up, there were many articles.

Get tweets over 140 characters https://qiita.com/hitsumabushi845/items/f7fd87106381fc65fc86

** It seems that extended_entities disappears when the tweet and video URL exceeds 140 characters. ** **

It took me a long time to notice this. Or rather, when I looked it up, it was resolved in an instant.

Solution

Adding tweet_mode ='extended' and include_entities = True to the api.search parameters will fix it.

tweets = tweepy.Cursor(api.search, q=q).items(20)

tweets = tweepy.Cursor(api.search, q=q, 
    #Here, extend the omitted part
    tweet_mode='extended', #Get all omitted tweets
    include_entities=True).items(20) #Get all omitted links

I was able to get the URL of the video, but I'm throwing an error. ..

AttributeError: 'Status' object has no attribute 'text'

Apparently, when tweet_mode is set to extended, the key name of the tweet text changes from text to full_text.

for tweet in tweets:
        # text →　full_text
        # print(tweet.text)
        print(tweet.full_text)
        print(tweet.extended_entities)
        try:
            media = tweet.extended_entities['media']

Modified code

search_tweets.py

import os
import tweepy
from datetime import datetime, date, timedelta
from dateutil.relativedelta import relativedelta

consumer_key = os.getenv('TWITTER_CONSUMER_KEY')
consumer_secret = os.getenv('TWITTER_CONSUMER_SECRET')
access_token = os.getenv('TWITTER_ACCESS_TOKEN')
access_token_secret = os.getenv('TWITTER_ACCESS_TOKEN_SECRET')
bearer_token = os.getenv('TWITTER_ACCESS_TOKEN_SECRET')


#Search for tweets related to Mayu Matsuoka on twitter and get the URL
def search_tweets():
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)

    yesterday = datetime.strftime(datetime.today() - relativedelta(days=1), f"%Y-%m-%d")
    #Omit retweets, 10 likes or more, yesterday-today, 1 or more retweets, twitter search on the condition that there are images and videos
    q = f'#Mayu Matsuoka OR Mayu Matsuoka filter:media exclude:retweets min_faves:10 since:{yesterday} min_retweets:1'

    #The problematic part
    tweets = tweepy.Cursor(api.search, q=q).items(20)

    contents = []
    for tweet in tweets:
        print(tweet.full_text)
        print(tweet.extended_entities)
        try:
            media = tweet.extended_entities['media']
            for m in media:
                print(m)
                preview = m['media_url_https']
                if m['type'] == 'video':
                    origin = [variant['url'] for variant in m['video_info']
                              ['variants'] if variant['content_type'] == 'video/mp4'][0]
                else:
                    origin = m['media_url_https']

                #Formatted for sending on LINE Bot
                content = {'preview': preview, 'origin': origin, 'type': m['type']}
                contents.append(content)

            print('--------------------------------------------')
        except:
            print('Error')
            print('--------------------------------------------')
    return contents


if __name__ == "__main__":
    search_tweets()

in conclusion

For me, who is a fan of Mayu Matsuoka (commonly known as Mayura) and has run out of time to check the Twitter timeline every day, this LINE Bot is only for me, but if you like, I'm reading this article. If you are a fan of Mayu Matsuoka, please register as a friend with the QR code below! !! !!

Thank you! !!

Also, if there are any mistakes in this article, or if you think you should do more, please comment more and more! !!

Reference article

[Python] Search and get Twitter tweets with tweepy https://vatchlog.com/tweepy-search/ Get tweets over 140 characters https://qiita.com/hitsumabushi845/items/f7fd87106381fc65fc86 Get Video URL with Python + tweepy https://thinkami.hatenablog.com/entry/2017/11/02/062226