[PYTHON] Try to get the road surface condition using big data of road surface management

Currently, as a "demonstration experiment in the sophistication of road surface management," big data that collects data acquired by inexpensive sensors attached to public transportation vehicles such as buses and taxis and aggregates them is open to the public.

** Road surface management big data open data contest ** http://micrms.force.com/apis

This time, we made it possible to check the road surface condition on the map using this data.

** Road surface condition Viewer ** http://needtec.sakura.ne.jp/road_mgt/road_mgt.html

GitHub https://github.com/mima3/road_mgt

Demonstration experiment API for advanced road surface management

The API specifications can be downloaded from the following. http://micrms.force.com/developer

The data can be roughly divided into the following three. -Road surface measurement data Data measured by a machine called i-DRIMS. Acceleration, angular velocity, position data ・ Road surface condition data Road surface condition data analyzed from road surface measurement data ・ Road surface specification data Road management information set in a short section for each local government

Normally, you will use road surface condition data and road surface specification data.

How to get road surface condition data

To get the road surface condition, GET to the following URL.

http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface

The result is the following XML.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:rm="http://roadmgt.herokuapp.com/vocab/rm#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
  <rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282">
    <rm:iri>2.65</rm:iri>
    <rm:step>0.0</rm:step>
    <rm:patching_num>0</rm:patching_num>
    <rm:municipality_id>1</rm:municipality_id>
    <rm:too_slow_fast_flag>0</rm:too_slow_fast_flag>
    <rm:subsidence_and_puddle>0.0</rm:subsidence_and_puddle>
    <rm:section_id>-1</rm:section_id>
    <rm:speed>37.55711977</rm:speed>
    <rdf:type>http://roadmgt.herokuapp.com/vocab/rm#road-surface</rdf:type>
    <geo:alt>0.0</geo:alt>
    <rm:rms>21.14314346</rm:rms>
    <geo:long>140.1434769</geo:long>
    <rm:rutting_amount>3.75</rm:rutting_amount>
    <rm:pothole_num>0</rm:pothole_num>
    <rm:speed_fluctuation_flag>0</rm:speed_fluctuation_flag>
    <rdfs:label>road-surface_282</rdfs:label>
    <rm:cracking_rate>5.3</rm:cracking_rate>
    <geo:lat>35.61753776</geo:lat>
    <rm:analysis_timestamp>2014-12-30 17:00:02.0</rm:analysis_timestamp>
    <rm:distance>220</rm:distance>
  </rdf:Description>
  <rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_294">
    <rm:speed_fluctuation_flag>1</rm:speed_fluctuation_flag>
    <rm:patching_num>0</rm:patching_num>
    <rm:iri>2.5</rm:iri>
    <rm:step>0.0</rm:step>
    <rm:too_slow_fast_flag>0</rm:too_slow_fast_flag>
    <rm:subsidence_and_puddle>0.0</rm:subsidence_and_puddle>
    <rm:section_id>-1</rm:section_id>
    <rm:municipality_id>1</rm:municipality_id>
    <rm:distance>340</rm:distance>
    <geo:alt>0.0</geo:alt>
    <rm:cracking_rate>7.5</rm:cracking_rate>
    <geo:long>140.1464737</geo:long>
    <rdf:type>http://roadmgt.herokuapp.com/vocab/rm#road-surface</rdf:type>
    <rm:rms>26.23188158</rm:rms>
    <geo:lat>35.61759593</geo:lat>
    <rm:pothole_num>0</rm:pothole_num>
    <rm:speed>42.15859739</rm:speed>
    <rm:analysis_timestamp>2014-12-30 17:00:02.0</rm:analysis_timestamp>
    <rm:rutting_amount>5.3</rm:rutting_amount>
    <rdfs:label>road-surface_294</rdfs:label>
  </rdf:Description>
Abbreviation
</rdf:RDF>

How to get all the data of the road surface condition

The maximum number of acquisitions when executing the API is 300 by default. To get the subsequent data, you need to use offset as follows.

(1) First get the beginning http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&limit=300&offset=0

(2) If the data can be acquired normally, change the offset and execute it. http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&limit=300&offset=300

(3) Repeat this until the following response is obtained.

404 notFound

Road surface condition parameters

The road surface condition can be filtered by specifying parameters. The parameters that can be specified are as follows.

Execution example with longitude and latitude specified:

http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&lat_start=34.66802946&lon_start=135.362425&lat_end=35.67436512&lon_end=140.2823427&offset=300&limit=300

Caution If you specify the parameters introduced here and parameters other than offset and limit, an error will occur. This means that code like the one below will result in an error.

$.ajax({
  type : 'GET',
  url : 'http://roadmgt.herokuapp.com/api/v1/datapoints',
  cache: false //Avoiding the cache will result in an error.
  data : {
    data_type : 'road-surface',
    lat_start  : lat_start,
    lon_start : lon_start,
    lat_end  : lat_end,
    lon_end  : lon_end,
    offset : offset,
    limit : limit
  },
  success : function (data) {
  },
  'error' :  function(xhr, textStatus, error) {
  }
});

This involves the risk that the content will not change even if the data is updated in the IE system.

In case of cahce = false, by specifying a unique temporary value for the parameter, it is recognized as another request and avoids the use of cache, but this is not available.

http://stackoverflow.com/questions/4303829/how-to-prevent-a-jquery-ajax-request-from-caching-in-internet-explorer

Direct target specification

You can get the target directly by specifying the URI described in rdf: about.

Execution example: http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282

Acquisition result:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:rm="http://roadmgt.herokuapp.com/vocab/rm#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282">
<rm:iri>2.65</rm:iri>
<rm:step>0.0</rm:step>
<rm:patching_num>0</rm:patching_num>
<rm:municipality_id>1</rm:municipality_id>
<rm:too_slow_fast_flag>0</rm:too_slow_fast_flag>
<rm:subsidence_and_puddle>0.0</rm:subsidence_and_puddle>
<rm:section_id>-1</rm:section_id>
<rm:speed>37.55711977</rm:speed>
<rdf:type>http://roadmgt.herokuapp.com/vocab/rm#road-surface</rdf:type>
<geo:alt>0.0</geo:alt>
<rm:rms>21.14314346</rm:rms>
<geo:long>140.1434769</geo:long>
<rm:rutting_amount>3.75</rm:rutting_amount>
<rm:pothole_num>0</rm:pothole_num>
<rm:speed_fluctuation_flag>0</rm:speed_fluctuation_flag>
<rdfs:label>road-surface_282</rdfs:label>
<rm:cracking_rate>5.3</rm:cracking_rate>
<geo:lat>35.61753776</geo:lat>
<rm:analysis_timestamp>2014-12-30 17:00:02.0</rm:analysis_timestamp>
<rm:distance>220</rm:distance>
</rdf:Description>
</rdf:RDF>

By specifying the property name after the target name, you can get only the property. At this time, it is necessary to change ":" to "_". For example, if you want to get "rm: step", specify "rm_step".

Execution example: http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282/rm_step

Acquisition result

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:rm="http://roadmgt.herokuapp.com/vocab/rm#">
<rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282">
<rm:step>0.0</rm:step>
</rdf:Description>
</rdf:RDF>

Description of the response

The main properties are explained below.

How to get road surface data

It is basically the same as the road surface condition. GET to the following URL.

http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-master&lat_start=34.66802946&lon_start=135.362425&lat_end=35.67436512&lon_end=140.2823427&offset=300&limit=300

Currently, only the following red line data exists.

As you can see by expanding it, it seems that data exists for each lane.

Used in programming language

Python Here, we will describe how to get the road surface condition in Python.

road_mgt.py

# -*- coding: utf-8 -*-
import urllib
import urllib2
from lxml import etree
import csv
from collections import defaultdict


def get_road_surface(offset, limit):
    """
Road surface condition data
    """
    url = ('http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&offset=%d&limit=%d' % (offset, limit))

    req = urllib2.Request(url)
    opener = urllib2.build_opener()
    conn = opener.open(req)
    cont = conn.read()
    parser = etree.XMLParser(recover=True)
    root = etree.fromstring(cont, parser)
    namespaces = {
        'geo' : 'http://www.w3.org/2003/01/geo/wgs84_pos#',
        'rdf' : 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', 
        'rdfs' : 'http://www.w3.org/2000/01/rdf-schema#', 
        'rm' : 'http://roadmgt.herokuapp.com/vocab/rm#'
    }

    row = []
    datas = {}
    value_tags = root.xpath('//rdf:Description', namespaces=namespaces)
    value_tags = root.xpath('//rdf:Description', namespaces=namespaces)
    for value_tag in value_tags:
        label = value_tag.find('rdfs:label', namespaces).text
        step = value_tag.find('rm:step', namespaces).text
        alt = value_tag.find('geo:alt', namespaces).text
        analysis_timestamp = value_tag.find('rm:analysis_timestamp', namespaces).text
        rutting_amount = value_tag.find('rm:rutting_amount', namespaces).text
        municipality_id = value_tag.find('rm:municipality_id', namespaces).text
        speed_fluctuation_flag = value_tag.find('rm:speed_fluctuation_flag', namespaces).text
        section_id = value_tag.find('rm:section_id', namespaces).text
        distance = value_tag.find('rm:distance', namespaces).text
        long = value_tag.find('geo:long', namespaces).text
        iri = value_tag.find('rm:iri', namespaces).text
        cracking_rate = value_tag.find('rm:cracking_rate', namespaces).text
        pothole_num = value_tag.find('rm:pothole_num', namespaces).text
        subsidence_and_puddle = value_tag.find('rm:subsidence_and_puddle', namespaces).text
        speed = value_tag.find('rm:speed', namespaces).text
        rms = value_tag.find('rm:rms', namespaces).text
        lat = value_tag.find('geo:lat', namespaces).text
        too_slow_fast_flag = value_tag.find('rm:too_slow_fast_flag', namespaces).text
        patching_num = value_tag.find('rm:patching_num', namespaces).text

        row.append({
            'label' : label,
            'step' : step,
            'alt' : alt,
            'analysis_timestamp' : analysis_timestamp,
            'rutting_amount' : rutting_amount,
            'municipality_id' : municipality_id,
            'speed_fluctuation_flag' : speed_fluctuation_flag,
            'section_id' : section_id,
            'distance' : distance,
            'long' : long,
            'iri' : iri,
            'cracking_rate' : cracking_rate,
            'pothole_num' : pothole_num,
            'subsidence_and_puddle' : subsidence_and_puddle,
            'speed' : speed,
            'rms' : rms,
            'lat' : lat,
            'too_slow_fast_flag' : too_slow_fast_flag,
            'patching_num' :patching_num
        })
    return row

def get_meas_locale(offset, limit):
    """
Position data at the time of road surface measurement
    """
    url = ('http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=meas-locale&offset=%d&limit=%d' % (offset, limit))

    req = urllib2.Request(url)
    opener = urllib2.build_opener()
    conn = opener.open(req)
    cont = conn.read()
    parser = etree.XMLParser(recover=True)
    root = etree.fromstring(cont, parser)

    namespaces = {
        'geo' : 'http://www.w3.org/2003/01/geo/wgs84_pos#',
        'rdf' : 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', 
        'rdfs' : 'http://www.w3.org/2000/01/rdf-schema#', 
        'rm' : 'http://roadmgt.herokuapp.com/vocab/rm#'
    }

    row = []
    datas = {}
    value_tags = root.xpath('//rdf:Description', namespaces=namespaces)
    for value_tag in value_tags:
        label = value_tag.find('rdfs:label', namespaces).text
        gpstimestamp = value_tag.find('rm:gpstimestamp', namespaces).text
        course = value_tag.find('rm:course', namespaces).text
        measurement_start_timestamp = value_tag.find('rm:measurement_start_timestamp', namespaces).text
        mounting_direction = value_tag.find('rm:mounting_direction', namespaces).text
        car_model = value_tag.find('rm:car_model', namespaces).text
        long = value_tag.find('geo:long', namespaces).text
        lat = value_tag.find('geo:lat', namespaces).text
        alt = value_tag.find('geo:alt', namespaces).text
        model_number = value_tag.find('rm:model_number', namespaces).text
        car_number = value_tag.find('rm:car_number', namespaces).text
        speed = value_tag.find('rm:speed', namespaces).text
        vertical_accuracy = value_tag.find('rm:vertical_accuracy', namespaces).text
        measurement_timestamp = value_tag.find('rm:measurement_timestamp', namespaces).text
        horizontal_accuracy = value_tag.find('rm:horizontal_accuracy', namespaces).text
        row.append({
            'label' : label,
            'gpstimestamp' : gpstimestamp,
            'course' : course,
            'measurement_start_timestamp' : measurement_start_timestamp,
            'mounting_direction' : mounting_direction,
            'car_model' : car_model,
            'long' : long,
            'lat' : lat,
            'alt' : alt,
            'model_number' : model_number,
            'car_number' : car_number,
            'speed' : speed,
            'vertical_accuracy' : vertical_accuracy,
            'measurement_timestamp' : measurement_timestamp,
            'horizontal_accuracy' : horizontal_accuracy
        })
    return row

def get_road_surface_all():
    ret = []
    limit = 300
    offset = 0
    try:
        while True:
            print ('get_road_surface_all %d' % offset)
            r = get_road_surface(offset, limit)
            ret.extend(r)
            offset += limit
    except urllib2.HTTPError, ex:
        if ex.code == 404:
            return ret
        else:
            raise

You can get all road conditions by using get_road_surface_all (). As a program, I just execute the API with urllib and parse the response with lxml.

JavaScript Here, we will describe how to get the road surface condition with JavaScript.

  /**
   *Execute API for road surface information
   */
  function getDataByRange(data_type, lat_start, lon_start, lat_end, lon_end, callback) {
    var offset = 0;
    var limit = 300;
    var obj = {};

    function loadRoadOffset(lat_start, lon_start, lat_end, lon_end, offset, limit, obj, cb) {

      console.log(offset, limit);
      $.ajax({
        type : 'GET',
        url : 'http://roadmgt.herokuapp.com/api/v1/datapoints',
        cache: true , //,An error will occur if cache is set to false. Perhaps instead of ignoring weird parameters, they're flipping
        data : {
          data_type : data_type,
          lat_start  : lat_start,
          lon_start : lon_start,
          lat_end  : lat_end,
          lon_end  : lon_end,
          offset : offset,
          limit : limit
        },
        success : function (data) {
          console.log(data);
          var root = data.documentElement;
          var attrs = root.attributes;
          var records = root.childNodes;
          for(var i=0; i<records.length; i++){
            if(records[i].nodeName.match(/rdf:Description/i)){
              var s = records[i].attributes["rdf:about"].value;
              var props = records[i].childNodes;
              for(var j=0; j<props.length; j++){
                if(props[j].nodeType == 1){
                  var p = props[j].nodeName;
                  var o = props[j].textContent;
                  if (!obj[s]) {
                    obj[s] = {};
                  }
                  if (obj[s][p]) {
                    if (!Array.isArray(obj[s][p])) {
                      var tmp = arys[s][p];
                      obj[s][p] = [];
                      obj[s][p].push(tmp);
                    }
                    obj[s][p].push(o);
                  } else {
                    obj[s][p] = o;
                  }
                }
              }
            }
          }
          loadRoadOffset(lat_start, lon_start, lat_end, lon_end, offset + limit, limit, obj, cb);
        },
        'error' :  function(xhr, textStatus, error) {
          console.log(xhr);
          var err = xhr.responseText;
          if (err == '404 notFound') {
            cb(null , obj);
          } else {
            cb(err , null);
          }
        }
      });
    }
    loadRoadOffset(lat_start, lon_start, lat_end, lon_end, offset, limit, obj, function(err, res) {
      callback(err, res);
    });
  }

This function runs as follows:

getDataByRange('road-master', surfaceRange.lat_min, surfaceRange.long_min, surfaceRange.lat_max, surfaceRange.long_max, function(err, data) {
      console.log('loadRoad', err, data);
});

This process gets all the road specification data in the specified range.

Summary

You can get the road surface condition by executing the API of "Demonstration experiment in advanced road surface management".

This makes it possible to visualize the deterioration status of roads.