[PYTHON] I tried to uncover our darkness with Chatwork API

The application of the article is available on github

I've put together a story about creating an application that looks up silent retirees from Chatwork that the company doesn't announce. (* We are not responsible for any troubles with the company to which the user belongs due to operation)

Slack, scraping, Excel documents should be able to be diverted by arranging the data acquisition method

Origin

――We have more than 200 employees, but I was very uncomfortable because there was no such thing as an announcement of retirees because some of the employees I knew had quit before I knew it. ――You are an engineer only if you solve problems with technology! ――When I noticed my previous job, people had quit, so there should be people who have the same problems → The desire for approval to be satisfied ―― ~~ I have no plans for consecutive holidays ~~

Functional goals

--Monitor daily enrollment changes based on your Chatwork contacts --Post the monitoring results to My Channel -Allow to see summary reports for a certain period such as yearly or monthly --Be careful when handling personal information and API keys

Technology used

--Host environment: Windows10 + Ubuntu20.4 on WSL2 (If you can use Docker, the host OS does not matter)

• Docker
• Python(Django)

Outcome

I was able to display new employees / retirees within the period and grasp daily changes in internal members.

↓ Report screen

↓ Chatwork notification result screen

Work process

Environment

--If you can use the docker-compose command, you can find other necessary items in Project published on github.

--Please prepare your own Chatwork API token (+ room_id of notification destination if you use batch notification) required for application operation. How to get API token (official)

Docker environment construction

Dockerfile

FROM python:3
ENV PYTHONUNBUFFERED 1
ENV PYTHONIOENCODING utf-8
RUN mkdir /script /app
WORKDIR /script
COPY requirements.txt /script/
RUN apt update && apt install -y cron vim
RUN pip install -r requirements.txt
WORKDIR /app

requirements.txt

Django>=3.0,<4.0
psycopg2-binary>=2.8
django-environ
requests
python-dateutil

docker-compose.yml

version: '3'

services:
  app:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    environment:
      - CHATWORK_API_TOKEN=${CHATWORK_API_TOKEN}
      - ROOM_ID=${ROOM_ID}
      - TZ=Asia/Tokyo
    volumes:
      - ./app:/app
      - ./export_environment.sh:/script/export_environment.sh
      - ./crontab:/script/crontab
    ports:
      - "8000:8000"

Main processing explanation

--Get a list of contact members using your account token --Clarify the joining / leaving members by comparing the list data with different acquisition times. --Since past data cannot be referenced from API, the data to be compared is stored in the DB. --Designed to return the difference between contact members when given two dates to get_diff () --show (), which has a function to display a report, displays a member change report for the past six months on a monthly basis. ――In batch processing, the same method as for reports is used to post members who have changed day by day to Chatwork as breaking news.

Report screen processing + Chatwork contact list acquisition and saving, data comparison method

app/chatwork/views.py

from django.shortcuts import render
from django.http import HttpResponse
from chatwork.models import Account
import requests
from datetime import date
from dateutil.relativedelta import relativedelta
from django.db.models import Count
import environ
env = environ.Env(DEBUG=(bool, False))

# Create your views here.

def show(request):
    diff_list = list()
    for i in range(6):
        end = (date.today() - relativedelta(months=i)).isoformat()
        start = (date.today() - relativedelta(months=(i+1))).isoformat()
        diff_list.append(get_diff(start, end))

    params = dict(d1=diff_list[0], d2=diff_list[1], d3=diff_list[2], d4=diff_list[3], d5=diff_list[4], d6=diff_list[5])
    return render(request, 'chatwork/show.html', params)

def get_diff(start, end):
    if not Account.objects.filter(date=date.today().isoformat()):
        base = 'https://api.chatwork.com/v2/'
        end_point = 'contacts'
        api_token = env('CHATWORK_API_TOKEN')
        headers = {'X-ChatWorkToken': api_token, 'Content-Type': 'application/x-www-form-urlencoded'}
        res = requests.get(base + end_point, headers=headers)
        for contact in res.json():
            data = dict(account_id=contact['account_id'], name=contact['name'][:2], department=contact['department'], date=date.today().isoformat())
            Account.objects.update_or_create(**data)
    query = Account.objects.filter(date__gte=start, date__lte=end).values('date').annotate(Count('date'))
    if len(query) < 2:
        return dict(period='no comparable data found during ' + start + ' ~ ' + end, added=list(), dropped=list())
    latest = query.order_by('-date')[0]['date'].isoformat()
    oldest = query.order_by('date')[0]['date'].isoformat()
    period = oldest + '~' + latest

    data_latest = Account.objects.filter(date=latest) or list()
    data_oldest = Account.objects.filter(date=oldest) or list()

    ids_latest = data_latest.values_list('account_id', flat=True) if data_latest else list()
    ids_oldest = data_oldest.values_list('account_id', flat=True) if data_oldest else list()

    added = Account.objects.filter(date=latest).filter(account_id__in=ids_latest).exclude(account_id__in=ids_oldest)
    dropped = Account.objects.filter(date=oldest).filter(account_id__in=ids_oldest).exclude(account_id__in=ids_latest)

    return dict(period=period, added=added, dropped=dropped)

Batch processing (send the difference between the current day and the previous day data to your own My Chat)

app/chatwork/management/commands/contact_daily.py

from django.core.management.base import BaseCommand

from chatwork.models import Account
from chatwork.views import get_diff

from datetime import date
from dateutil.relativedelta import relativedelta
import environ
import requests
env = environ.Env(DEBUG=(bool, False))

class Command(BaseCommand):
    def handle(self, *args, **options):
        today = date.today().isoformat()
        yesterday = (date.today() - relativedelta(days=1)).isoformat()
        data = get_diff(yesterday, today)
        report_title = data['period']
        report_added = 'added: ' + '(' + str(len(data['added'])) + ')' + ' / '.join(list(d.name for d in data['added']))
        report_dropped = 'dropped: ' + '(' + str(len(data['dropped'])) + ')' + ' / '.join(list(d.name for d in data['dropped']))
        report = """
{report_title}
{report_added}
{report_dropped}
        """.format(report_title=report_title, report_added=report_added, report_dropped=report_dropped).strip()

        base = 'https://api.chatwork.com/v2/'
        room_id = env('ROOM_ID')
        end_point = 'rooms/' + room_id + '/messages'
        api_token = env('CHATWORK_API_TOKEN')
        headers = {'X-ChatWorkToken': api_token, 'Content-Type': 'application/x-www-form-urlencoded'}
        payload = dict(body=report, self_unread=1)
        res = requests.post(base + end_point, headers=headers, params=payload)

Other

--Unit test is simple but implemented --Since leakage of personal information is not fashionable, the specification is such that only the first two characters of the name obtained from the API are saved in the DB. --Since it is absolutely NG to leak the API token of the company account, we decided to pass it as an environment variable. --Batch processing is executed every day, so it is via cron, but since the environment variables are independent of the user, it was necessary to take measures against it. --There were many inconveniences such as DB acquisition using Django's Models and handling of iterators. Recognize that Laravel's QueryBuilder and Collection classes are apt, including the fact that they are easy to understand even in the code of other companies.

Finally

It's a rough app that I made on holidays, but it's very encouraging to hear your opinions, so please leave a comment.

--Detailed commentary addition request --Guidance on improvement plans --Other technical questions, etc.