I checked the Python package pre-installed in Google Cloud Dataflow

Google Cloud Dataflow doesn't get much attention, but it's quite convenient because you can easily switch the execution environment between local and remote. Moreover, if you think that you can only use the standard library, you can install it from list of pip or [install your own](http://qiita. com / orfeon / items / 78ff952052c4bde4bcd3) is also possible. Then, I tried to find out which library was pre-installed because it did not come out even if I searched Document briefly.

Preparation

First of all, setting options, this area is @ orfeon's round plagia ...

Option settings

import apache_beam as beam
import apache_beam.transforms.window as window

options = beam.utils.pipeline_options.PipelineOptions()

google_cloud_options = options.view_as(beam.utils.pipeline_options.GoogleCloudOptions)
google_cloud_options.project = '{PROJECTID}'
google_cloud_options.job_name = 'test'
google_cloud_options.staging_location = 'gs://{BUCKET_NAME}/binaries'
google_cloud_options.temp_location = 'gs://{BUCKET_NAME}/temp'

worker_options = options.view_as(beam.utils.pipeline_options.WorkerOptions)
worker_options.max_num_workers = 1

# options.view_as(beam.utils.pipeline_options.StandardOptions).runner = 'DirectRunner'
options.view_as(beam.utils.pipeline_options.StandardOptions).runner = 'DataflowRunner'

p = beam.Pipeline(options=options)

Run pip freeze to log the Python package list.

Package list output part

def inspect_df(dat):
    import subprocess
    import logging
    process = subprocess.Popen('pip freeze', shell=True,
                               stdout=subprocess.PIPE, 
                               stderr=subprocess.PIPE)
    for line in process.stdout:
        logging.info(line)

Run on Dataflow. You may not need hello world ...

Pipeline execution

(p | 'init' >> beam.Create(['hello', 'world'])
   | 'inspect' >> beam.Map(inspect_df))

p.run()

When the execution of the pipeline is completed, the package list will be output to the log, so check it in the Cloud Console.

Log check

In Dataflow Document, you can check the log from the Job details screen of Dataflow, but as of March 4, 2017, Stackdriver -> Moving to Logging.

Package List

It is a list of packages spit out in the above log. ** As of March 4, 2017 **

Is it because tf.transform has arrived? In Cloud ML, ~~ TensorFlow ver is 0.12 ~~ ** (EDIT: The latest Ver is [here](https://cloud.google.com/ml-engine/docs/concepts/runtime-version-" You can check it with list)) ** It is 1.0.0 in Dataflow. scikit-learn seems a bit old.

Although staging is a little slow, Dataflow, which can easily switch between local and remote from Jupyter Notebook and perform fully managed instance startup and startup without permission, seems to be a powerful tool for applications such as data analysis and machine learning. is.