[PYTHON] Create a private DMP with zero initial cost and zero development with BigQuery

Private DMPs are popular these days, but most of them are treasure data (YBI). Treasure data is very useful (especially td-js-sdk), but it's a bit expensive.

On the other hand, BigQuery is attractive for its low price and query execution speed, but it is very inconvenient to input and output data compared to treasure data.

So, I made an application with GAE (Python) that realizes ease of use like treasure data on BigQuery. If you use your own source code, you can build it with zero development.

Source code: https://github.com/mats116/ElasticBigQuery

• You are free to make any repairs, but if possible, we would appreciate it if you could give us an Issue or Pull Request.
• By the way, it's a completely individual project, so it has nothing to do with the company I belong to.
• ʻElasticbigquery.appspot.com` runs without charge, so you can hit it freely, but please do so within the bounds of common sense.

To be able to

Get logs from td-js-sdk

--It works just like treasure data just by changing the endpoint as shown below. --writeKey is static at the moment, but we plan to make it possible to issue and manage permissions from the screen. --Only JSONP, which has become the default since v1.4.0, is supported. --BigQuery is not schemaless, but it automatically identifies it in the API and automatically generates datasets and tables. --The generated table name has a date (UTC) at the end, such as table_id + YYYYMMDD, making it a daily table. --As a unique function, the following parameters are acquired in the URL-decoded state. - td_path - td_referrer - td_url

pageviews.js

<script type="text/javascript">
  var td = new Treasure({
    host: 'elasticbigquery.appspot.com',
    pathname: '/dmp/v1/event/',
    writeKey: 'thie_is_static_setting_yet',
    database: '<dataset_id>'
  });

  td.trackPageview('<table_id>');
</script>

Get logs from web beacons

--Returns a transparent GIF with the endpoint below. - //elasticbigquery.appspot.com/dmp/v1/beacon/<dataset_id>/measurement --You can get the GET parameter. --There are some parameters such as referrer that are acquired by default.

Issuing a cookie ID

--Issue bqid under the GAE domain xxx-xxx.appspot.com. --Please change the domain and id name according to the purpose. --You can check it at the following endpoint. - http://elasticbigquery.appspot.com/dmp/v1/bqid/get --If you add callback = hoge to the GET parameter, you can use it like JSONP.

What I want to be able to do from now on

Account control

--I want to be able to grant permissions for BigQuery datasets to the account signed up with Oauth (Google) from the UI.

control of writeKey

--I want to be able to issue a writeKey from the UI. (Currently static) ――It's hard to refer to Cloud Datastore every time, so maybe memocache?

Export function

--I want to be able to export query results to S3 and Google Cloud Storage. ――I wonder if it can be integrated with Apps Script in a nice way

Impressions

--Completely freeride to td-js-sdk. .. ――When you fail to put it in BigQuery, it is loaded in the Task queue, so it is made quite firmly. ――Because it is a personal project, we are looking for a company that can do GAE load verification. ――I'm not good at making UI, so please help me.

* GAE construction method

For your reference. If you are new to GAE, do your best.

What to prepare

--Google account (@gmail address is also OK) -PC with Google App Engine SDK

Creating a project

-Create a new project in the Google Developers Console. --App Engine location is ** us-central ** is the closest to Japan --By default ** BigQuery API ** should be enabled, but just in case

Get source code

--Since it is published on GitHub, please clone it appropriately. - https://github.com/mats116/ElasticBigQuery

setting change

--Open ʻapp.yaml` and rename the project --In terms of source code, here

Deploy

--Deploy from Google AppEngine Launcher.