[PYTHON] Do something regularly from heroku to Shotgun

What is this?

This is a continuation of this article

[Python + heroku] From the state without Python to displaying something on heroku (Part 1) http://qiita.com/it_ks/items/afd1bdb792d41d0e1145 (Part 2) http://qiita.com/it_ks/items/ca6c7f6e8fc89e49e46d

I've been touching heroku lately to use the ** Shotgun API ** to target shotguns.

Supplement 1: Background of use

GUNCY'S Co., Ltd.'s Road to Stingray project I participated as a Shotgun setup staff. Here, I think it would be good if I could write the knowledge I gained about Shotugn (in the future). I haven't been able to go too deep, so I feel that the scope is limited to the one that has been officially published (Japanese version) ...

I think this is a case of a relatively small to medium-sized project and multiple locations (remote work). (Shotgun is also used in productions of 100 to 1000 people, but I wonder if it will be a different story from that case)

Supplement 2: Tag

** "Shotgun" tag ** on qiita It was already used in the topic about the ruby library of the same name. So, I think it would be better for each person to live separately with the tag "Shotgun Softrware". If you read this article and would like to post an article about Shotgun, If you agree with the intention, we would appreciate it if you could tag it in the same way.

Shotgun

https://shotgunsoftware.com/ (Domestic agency) https://www.borndigital.co.jp/software/4070.html

While it is possible to handle various situations, It is a gem that has a side that is quite jaja horse without setup (* It is an individual impression)

API It is published on github.

https://github.com/shotgunsoftware/python-api

wiki

https://github.com/shotgunsoftware/python-api/wiki

Pipeline Toolkit It is a general term for a group of kits prepared to wrap the above API, link with various external tools, and realize the extension of Shotgun (web). https://support.shotgunsoftware.com/entries/94042238-Toolkit-Home-Page I will not deal with it this time.

Like the API, it is published on github, and it is mainly the one with "** tk-**" in the head. tk is an abbreviation for toolkit. https://github.com/shotgunsoftware

In the ftrack featured in this article (↓) http://qiita.com/it_ks/items/374a320bc8282c7f65a8 It can be said to be equivalent to "ftrack connect".

heroku schedule processing

Official document:

Scheduled Jobs with Custom Clock Processes in Python with APScheduler https://devcenter.heroku.com/articles/clock-processes-python

Just qiita has an article on heroku and Python

Get regular exchange rates on Heroku and upload logs to Amazon S3 http://qiita.com/gumob@github/items/ca5d76186f94e592a5f0

The periodic processing itself uses the "** APScheduler **" module. If you search with AP Schuler, the one that appears at the top is here.

A little advanced job scheduling with APScheduler http://qiita.com/yushin/items/a026626dbb291dd43dd8

Preparation for regular processing

First, let's set up regular processing on heroku. We will proceed according to this.

https://devcenter.heroku.com/articles/clock-processes-python

Introduced AP Schuler

After going down to the app and activating vertical env (← this is the content up to the last time),

pip install apscheduler

python

(sgenv) >pip install apscheduler
Collecting apscheduler
  Downloading APScheduler-3.0.5-py2.py3-none-any.whl (49kB)
    100% |################################| 53kB 2.4MB/s
Requirement already satisfied (use --upgrade to upgrade): six in c:\users\path\to\app\sgenv\lib\site-packages (from apscheduler)
Collecting futures (from apscheduler)
  Downloading futures-3.0.4-py2-none-any.whl
Collecting pytz (from apscheduler)
  Downloading pytz-2015.7-py2.py3-none-any.whl (476kB)
    100% |################################| 479kB 853kB/s
Collecting tzlocal (from apscheduler)
  Downloading tzlocal-1.2.tar.gz
Building wheels for collected packages: tzlocal
  Running setup.py bdist_wheel for tzlocal ... done
  Stored in directory: C:\Users\{user}\AppData\Local\pip\Cache\wheels\39\8d\3b\21db6b23bc7483a2e1cf391865427e8fbd09022
3d9db2cfab3
Successfully built tzlocal
Installing collected packages: futures, pytz, tzlocal, apscheduler
Successfully installed apscheduler-3.0.5 futures-3.0.4 pytz-2015.7 tzlocal-1.2

Also, In requirements.txt APScheduler==3.0.0 I will add.

clock.py Prepare a Python script that you want to process regularly. It is named "clock.py" according to the previous document.

clock.py

from apscheduler.schedulers.blocking import BlockingScheduler

sched = BlockingScheduler()

@sched.scheduled_job('interval', minutes=3)
def timed_job():
    print('This job is run every three minutes.')

@sched.scheduled_job('cron', day_of_week='mon-fri', hour=17)
def scheduled_job():
    print('This job is run every weekday at 5pm.')

sched.start()

First of all, the content of the script is the same as the document.

--Output log once every 3 minutes --Output log every day at 17:00

There are two jobs.

Added to Procfile

Add the following contents to ** Procfile **.

clock: python clock.py

clock: is the setting name, and python ~ ~ is the command you want to execute. Just write it in the Procfile and it will appear on the heroku dashboard on the web, and you can turn it on and off respectively. With free heroku, you can only have one running, so if you turn one off, you can turn it on. (The details of turning on / off will be described later)

Deploy

git add . andgit commit -m "hogehoge"and git push heroku master. In add and commit, various things will flow by the amount of APScheduler added earlier.

scale

If it's a document heroku ps:scale clock=1 It says, but with this, you can change the "clock" added to the Procfile to "1 (= valid. ON)". However, since a process called "web" is already running, it is invalid to execute it in this state.

python

>heroku ps:scale clock=1
Scaling dynos... failed
 !    Please verify your account in order to change resources (please enter a credit card) For more information, see htt
ps://devcenter.heroku.com/categories/billing Verify now at https://heroku.com/verify

Log in to heroku with your browser, turn off the already turned on, and then turn on "clock". You can do it from a command instead of a browser. I wanted to use the dashboard.

Verification

Now the process called "clock" has started and the contents of clock.py are executed. I would like to see the log. When looking at heroku's log ↓

heroku logs

python

>heroku logs
2016-02-02T13:36:08.456761+00:00 app[web.1]:     res = instance.__dict__[self.name] = self.func(instance)
2016-02-02T13:36:08.456761+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/functional.py", line 33, in __get__
2016-02-02T13:36:08.456762+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 417, in url_patterns
2016-02-02T13:36:08.456763+00:00 app[web.1]:     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
2016-02-02T13:36:08.456765+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/urlresolvers.py", line 410, in urlconf_module
(Omitted)
2016-02-08T12:23:23.304081+00:00 heroku[api]: Scale to web=0 by 〜〜@gmail.com
2016-02-08T12:23:24.758985+00:00 heroku[api]: Scale to clock=1 by 〜〜@gmail.com
2016-02-08T12:23:29.007235+00:00 heroku[clock.1]: Starting process with command `python clock.py`
2016-02-08T12:23:29.571304+00:00 heroku[clock.1]: State changed from starting to up
2016-02-08T12:26:30.559161+00:00 app[clock.1]: This job is run every three minutes.

It started to move safely

Use Shotgun API

Once it's working, add the content using the Shotgun API to clock.py.

API preparation

Pip according to Installing on github. https://github.com/shotgunsoftware/python-api#installing

pip install git+git://github.com/shotgunsoftware/python-api.git

If you want to fix the version, it is here, so pip install git+git://github.com/shotgunsoftware/[email protected] I wrote this in requirements.txt. By the way, v3.0.25 at that time

Register script users with Shotgun

"Users" in Shotgun are divided into "** Human User " and " Script User **". The former is the so-called user, and the latter is a script such as Pipeline Toolkit. From the point of view of shotgun, the person who accesses both people and scripts is the user. Let's register a ScriptUser to access from heroku.

After logging in to Shotgun, open "Scripts" from the menu on the upper right and click "+ Scripts".

"** Script name " to be registered here, " Application key **" obtained after registration, And the URL of the licensed ** Shotgun server ** ("https: // {studio-name} .shotgunstudio.com") I will need it next time, so make a note of it.

Also, since it makes me feel excited, let's add an appropriate icon to ScriptUser.

Environment variable registration on heroku

Even from the command (ry, but from the dashboard. You can set environment variables in "Config Variables" of "Settings" on the far right.

Read environment variables

The environment variables you filled in earlier are in the Python script ʻOs.environ.get ('{variable name}') ` You can use it by writing like. I added it to clock.py like this ↓

python

import os

SG_KEY = os.environ.get('SG_KEY')
SG_SCRIPT = os.environ.get('SG_SCRIPT')
SG_SERVER = os.environ.get('SG_SERVER')

Be careful as you may forget to write import os as you get used to it. ←

Get shotgun object

Shotgun() - Usage Example @Reference: Methods https://github.com/shotgunsoftware/python-api/wiki/Reference%3A-Methods#usage-example

↑ Import and receive the instance in this way

python

import os
from shotgun_api3 import Shotgun

### get sg object ###
SG_KEY = os.environ.get('SG_KEY')
SG_SCRIPT = os.environ.get('SG_SCRIPT')
SG_SERVER = os.environ.get('SG_SERVER')

sg = Shotgun(SG_SERVER, SG_SCRIPT, SG_KEY)

Search Shotgun and print results every 3 minutes

This time, I'll search for a project named ** "TEST" and output it to the log.

This time there is only one result returned, so The method to use is find_one.

sg.find_one("Project",[['name','is','TEST']],[])

If it hits, it will be returned as ** dictionary type ** with data type (Entity Type) and ID on Shotgun like this. {'type': 'Project', 'id': 69}

• This time, the ID of "TEST" was 69.

The script created by incorporating this looks like this ↓

clock.py

import os
from shotgun_api3 import Shotgun
from apscheduler.schedulers.blocking import BlockingScheduler

### get sg object ###
SG_SERVER = os.environ.get('SG_SERVER')
SG_SCRIPT = os.environ.get('SG_SCRIPT')
SG_KEY = os.environ.get('SG_KEY')
sg = Shotgun(SG_SERVER, SG_SCRIPT, SG_KEY)

sched = BlockingScheduler()

@sched.scheduled_job('interval', minutes=3)
def timed_job():
    testPrj = sg.find_one("Project",[['name','is','TEST']],[])
    print('every 3min search...',testPrj)

sched.start()

Verification

push to heroku logs.

python

2016-02-08T12:42:19.165168+00:00 heroku[clock.1]: State changed from crashed to starting
2016-02-08T12:42:23.381828+00:00 heroku[clock.1]: Starting process with command `python clock.py`
2016-02-08T12:42:24.077203+00:00 heroku[clock.1]: State changed from starting to up
2016-02-08T12:45:27.542367+00:00 app[clock.1]: ('every 3min search...', {'type': 'Project', 'id': 69})
2016-02-08T12:48:27.586542+00:00 app[clock.1]: ('every 3min search...', {'type': 'Project', 'id': 69})
2016-02-08T12:51:27.559255+00:00 app[clock.1]: ('every 3min search...', {'type': 'Project', 'id': 69})
(Omitted)

The return value that was searched and hit is printed every 3 minutes. I have reached my goal of doing something on a regular basis.

Summary

What i did

--Add process to Procfile, enable it on dashboard --Prepare a script using Shotgun API

However, the free version of heroku always needs to sleep for more than 6 hours a day, so you can't spin it all the time.

the next deployment

--Use Shotgun API processes other than find (create, update, etc.) --Use Shotgun API that is not a regular process --Linkage between Shotgun API and API of another service

Etc. are possible.

20200207 Addendum

Speaking of which, it seems that a web hook is attached. I'm looking forward to it.

https://developer.shotgunsoftware.com/ja/3d448f5e/

Remarks

find_one https://github.com/shotgunsoftware/python-api/wiki/Reference%3A-Methods#find_one

How to write a filter used for search Reference: Filter Syntax https://github.com/shotgunsoftware/python-api/wiki/Reference%3A-Filter-Syntax