Let's write python code that parses go code and generates go code

This article is the 14th day article of WACUL Advent Calendar 2016. The writer has been working at WACUL since September of this year, writing mainly go code and sometimes python on the backend.

This article is a continuation of Previous article (The previous article is uselessly long so you don't have to read it forcibly).

Last time, I wrote python code that generates go code from API response example by imitating JSON-to-Go. .. This time, I would like to generate the go code using the result of analyzing the go code.

Introduction

Automatic generation of conversion process

For example, using go-swagger will generate a struct definition for the response. This is a struct that is slightly different from the struct stored in the internal database, but has a similar structure.

When creating the actual web API, it may be necessary to convert the struct that is persisted in the database to the struct for this response. This is rather boring and tedious. I want to automatically generate the code for this part.

Overview

To summarize the story a little more, what I want is a conversion code from model (src) to response object (dst). To obtain this, analyze the struct definition of the conversion source and conversion destination and extract the type information. In this area, go is made with ast package etc. After extracting the type information with the created tool, it is output as JSON. It is an attempt to generate a conversion code from src to dst using this JSON.

sample: github api

I would like to introduce an example of executing code generation using a textual example.

Originally, it is a story to generate a process to convert the expression of the persistence layer to the expression of response, but since it is troublesome, it is a little arbitrary, but the conversion process code is automatically generated in the following form. I would like to try it.

  1. Generate a struct from the response example of a nice API (Use the code created in the previous article)
  2. Generate a struct from yaml of a nice swagger (using go-swagger)

I would like to automatically generate the process of converting from src to dst, assuming that the go code of the two structs generated by these is the struct of the response expression of model and api for persistence, respectively.

The automatic generation itself is created by the following procedure.

  1. Extract information from the struct definition of the source go ʻextract :: go [src] => JSON [src] `
  2. Extract information from the struct definition of the destination go ʻextract :: go [dst] => JSON [dst] `
  3. Generate go code from the extracted information convert :: JSON [src], json [dst] => go [convert]

Using these two, the code of the conversion process from the struct of the conversion source (src) to the struct of the conversion destination (dst) is generated.

Generate go code (src)

I will take the response example of api to get the authenticated user from the following part of the document of github. There is no particular reason to choose this API. That's all because I somehow noticed it.

Save it as github-get-authenticated-user.json.

{
  "login": "octocat",
  "id": 1,
  "avatar_url": "https://github.com/images/error/octocat_happy.gif",
  "gravatar_id": "",
  "url": "https://api.github.com/users/octocat",
  "html_url": "https://github.com/octocat",
  "followers_url": "https://api.github.com/users/octocat/followers",
  "following_url": "https://api.github.com/users/octocat/following{/other_user}",
  "gists_url": "https://api.github.com/users/octocat/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/octocat/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/octocat/subscriptions",
  "organizations_url": "https://api.github.com/users/octocat/orgs",
  "repos_url": "https://api.github.com/users/octocat/repos",
  "events_url": "https://api.github.com/users/octocat/events{/privacy}",
  "received_events_url": "https://api.github.com/users/octocat/received_events",
  "type": "User",
  "site_admin": false,
  "name": "monalisa octocat",
  "company": "GitHub",
  "blog": "https://github.com/blog",
  "location": "San Francisco",
  "email": "[email protected]",
  "hireable": false,
  "bio": "There once was...",
  "public_repos": 2,
  "public_gists": 1,
  "followers": 20,
  "following": 0,
  "created_at": "2008-01-14T04:33:35Z",
  "updated_at": "2008-01-14T04:33:35Z",
  "total_private_repos": 100,
  "owned_private_repos": 100,
  "private_gists": 81,
  "disk_usage": 10000,
  "collaborators": 8,
  "plan": {
    "name": "Medium",
    "space": 400,
    "private_repos": 20,
    "collaborators": 0
  }
}

Generate the struct definition from here. Use Script created in the previous article.

$ swagger_src
mkdir -p src/github
python src/jsontogo/jsontogo.py json/github-get-authenticated-user.json --package github --name AuthenticatedUser | gofmt > src/github/authenticated_user.go

You can get the following struct definition. (Save it as src / github / authenticated_user.go)

package github

import (
	"time"
)

/* structure
AuthenticatedUser
	Plan
*/
// AuthenticatedUser : auto generated JSON container
type AuthenticatedUser struct {
	AvatarURL         string    `json:"avatar_url" example:"https://github.com/images/error/octocat_happy.gif"`
	Bio               string    `json:"bio" example:"There once was..."`
	Blog              string    `json:"blog" example:"https://github.com/blog"`
	Collaborators     int       `json:"collaborators" example:"8"`
	Company           string    `json:"company" example:"GitHub"`
	CreatedAt         time.Time `json:"created_at" example:"2008-01-14T04:33:35Z"`
	DiskUsage         int       `json:"disk_usage" example:"10000"`
	Email             string    `json:"email" example:"[email protected]"`
	EventsURL         string    `json:"events_url" example:"https://api.github.com/users/octocat/events{/privacy}"`
	Followers         int       `json:"followers" example:"20"`
	FollowersURL      string    `json:"followers_url" example:"https://api.github.com/users/octocat/followers"`
	Following         int       `json:"following" example:"0"`
	FollowingURL      string    `json:"following_url" example:"https://api.github.com/users/octocat/following{/other_user}"`
	GistsURL          string    `json:"gists_url" example:"https://api.github.com/users/octocat/gists{/gist_id}"`
	GravatarID        string    `json:"gravatar_id" example:""`
	HTMLURL           string    `json:"html_url" example:"https://github.com/octocat"`
	Hireable          bool      `json:"hireable" example:"False"`
	ID                int       `json:"id" example:"1"`
	Location          string    `json:"location" example:"San Francisco"`
	Login             string    `json:"login" example:"octocat"`
	Name              string    `json:"name" example:"monalisa octocat"`
	OrganizationsURL  string    `json:"organizations_url" example:"https://api.github.com/users/octocat/orgs"`
	OwnedPrivateRepos int       `json:"owned_private_repos" example:"100"`
	Plan              Plan      `json:"plan"`
	PrivateGists      int       `json:"private_gists" example:"81"`
	PublicGists       int       `json:"public_gists" example:"1"`
	PublicRepos       int       `json:"public_repos" example:"2"`
	ReceivedEventsURL string    `json:"received_events_url" example:"https://api.github.com/users/octocat/received_events"`
	ReposURL          string    `json:"repos_url" example:"https://api.github.com/users/octocat/repos"`
	SiteAdmin         bool      `json:"site_admin" example:"False"`
	StarredURL        string    `json:"starred_url" example:"https://api.github.com/users/octocat/starred{/owner}{/repo}"`
	SubscriptionsURL  string    `json:"subscriptions_url" example:"https://api.github.com/users/octocat/subscriptions"`
	TotalPrivateRepos int       `json:"total_private_repos" example:"100"`
	Type              string    `json:"type" example:"User"`
	URL               string    `json:"url" example:"https://api.github.com/users/octocat"`
	UpdatedAt         time.Time `json:"updated_at" example:"2008-01-14T04:33:35Z"`
}

// Plan : auto generated JSON container
type Plan struct {
	Collaborators int    `json:"collaborators" example:"0"`
	Name          string `json:"name" example:"Medium"`
	PrivateRepos  int    `json:"private_repos" example:"20"`
	Space         int    `json:"space" example:"400"`
}

Now you have the struct definition for the source (src).

Generate go code (dst)

Next, let's get the struct definition of the conversion destination (dst). Get the go struct definition using go-swagger from the swagger definition.

api.gru

There is a service called api.gru. This is a service that shares the API specs of web services out there. Many are provided in OpenAPI / Swagger 2.0 format. By the way, if you look at swagger.yaml that you can get here using ReDoc etc., you can understand what swagger is like.

api.gru

Let's try automatic generation using the API of github as an example. Download the swagger definition yaml from the following URL.

$ make swagger_fetch
mkdir -p yaml
wget https://api.apis.guru/v2/specs/github.com/v3/swagger.yaml -O yaml/github-swagger.yaml
gsed -i "s/type: *'null'/type: object/g; s/'+1':/'plus1':/g; s/'-1':/'minus1':/" yaml/github-swagger.yaml

You can get yaml like the link. The sed after wget is a minor fix to address the current go-swagger issue and the registered swagger.yaml issue.

Extract type information from struct definition

こちらは普通にgo-swaggerを実行するだけです。以下の様にして実行します。今回はstructの変換だけなのでmodelのみを生成します。本筋とは関係ないのですがなぜか"github.com/go-openapi/swag"のimportが出力されないという問題があったためgoimportsで追加しています。

$ make swagger_gen
rm -rf dst/swagger/gen
mkdir -p dst/swagger/gen
swagger generate model -f yaml/github-swagger.yaml --target dst/swagger/gen --model-package def
goimports -w dst/swagger/gen/def/*.go

A lot of code is generated, but this time only User.go cares.

package def

// This file was generated by the swagger tool.
// Editing this file might prove futile when you re-run the swagger generate command

import (
	strfmt "github.com/go-openapi/strfmt"
	"github.com/go-openapi/swag"

	"github.com/go-openapi/errors"
)

// User user
// swagger:model user
type User struct {

	// avatar url
	AvatarURL string `json:"avatar_url,omitempty"`

	// bio
	Bio string `json:"bio,omitempty"`

	// blog
	Blog string `json:"blog,omitempty"`

	// collaborators
	Collaborators int64 `json:"collaborators,omitempty"`

	// company
	Company string `json:"company,omitempty"`

	// ISO 8601 format: YYYY-MM-DDTHH:MM:SSZ
	CreatedAt string `json:"created_at,omitempty"`

	// disk usage
	DiskUsage int64 `json:"disk_usage,omitempty"`

	// email
	Email string `json:"email,omitempty"`

	// followers
	Followers int64 `json:"followers,omitempty"`

	// following
	Following int64 `json:"following,omitempty"`

	// gravatar id
	GravatarID string `json:"gravatar_id,omitempty"`

	// hireable
	Hireable bool `json:"hireable,omitempty"`

	// html url
	HTMLURL string `json:"html_url,omitempty"`

	// id
	ID int64 `json:"id,omitempty"`

	// location
	Location string `json:"location,omitempty"`

	// login
	Login string `json:"login,omitempty"`

	// name
	Name string `json:"name,omitempty"`

	// owned private repos
	OwnedPrivateRepos int64 `json:"owned_private_repos,omitempty"`

	// plan
	Plan *UserPlan `json:"plan,omitempty"`

	// private gists
	PrivateGists int64 `json:"private_gists,omitempty"`

	// public gists
	PublicGists int64 `json:"public_gists,omitempty"`

	// public repos
	PublicRepos int64 `json:"public_repos,omitempty"`

	// total private repos
	TotalPrivateRepos int64 `json:"total_private_repos,omitempty"`

	// type
	Type string `json:"type,omitempty"`

	// url
	URL string `json:"url,omitempty"`
}

// Validate validates this user
func (m *User) Validate(formats strfmt.Registry) error {
	var res []error

	if err := m.validatePlan(formats); err != nil {
		// prop
		res = append(res, err)
	}

	if len(res) > 0 {
		return errors.CompositeValidationError(res...)
	}
	return nil
}

func (m *User) validatePlan(formats strfmt.Registry) error {

	if swag.IsZero(m.Plan) { // not required
		return nil
	}

	if m.Plan != nil {

		if err := m.Plan.Validate(formats); err != nil {
			return err
		}
	}

	return nil
}

// UserPlan user plan
// swagger:model UserPlan
type UserPlan struct {

	// collaborators
	Collaborators int64 `json:"collaborators,omitempty"`

	// name
	Name string `json:"name,omitempty"`

	// private repos
	PrivateRepos int64 `json:"private_repos,omitempty"`

	// space
	Space int64 `json:"space,omitempty"`
}

// Validate validates this user plan
func (m *UserPlan) Validate(formats strfmt.Registry) error {
	var res []error

	if len(res) > 0 {
		return errors.CompositeValidationError(res...)
	}
	return nil
}

Extracting type information from struct

At least you have to extract the type information from the go code in the struct definition. Let's do our best in this area. How much if you have a spirit with ast. I made a command called go-structjson. Don't expect too much code quality inside as it's a mess.

From the code below

package alias

// Person :
type Person struct {
	Name string
}

type P *Person
type PS []Person
type PS2 []P
type PSP *[]P

The following JSON is output.

{
  "module": {
    "alias": {
      "file": {
        "GOPATH/src/github.com/podhmo/go-structjson/examples/alias/person.go": {
          "alias": {
            "P": {
              "candidates": null,
              "name": "P",
              "original": {
                "kind": "pointer",
                "value": {
                  "kind": "primitive",
                  "value": "Person"
                }
              }
            },
            "PS": {
              "candidates": null,
              "name": "PS",
              "original": {
                "kind": "array",
                "value": {
                  "kind": "primitive",
                  "value": "Person"
                }
              }
            },
            "PS2": {
              "candidates": null,
              "name": "PS2",
              "original": {
                "kind": "array",
                "value": {
                  "kind": "primitive",
                  "value": "P"
                }
              }
            },
            "PSP": {
              "candidates": null,
              "name": "PSP",
              "original": {
                "kind": "pointer",
                "value": {
                  "kind": "array",
                  "value": {
                    "kind": "primitive",
                    "value": "P"
                  }
                }
              }
            }
          },
          "name": "GOPATH/src/github.com/podhmo/go-structjson/examples/alias/person.go",
          "struct": {
            "Person": {
              "fields": {
                "Name": {
                  "embed": false,
                  "name": "Name",
                  "tags": {},
                  "type": {
                    "kind": "primitive",
                    "value": "string"
                  }
                }
              },
              "name": "Person"
            }
          }
        }
      },
      "fullname": "github.com/podhmo/go-structjson/examples/alias",
      "name": "alias"
    }
  }
}

There was a story that the name newtype is more appropriate for the ones handled by the name alias. Here, type MyInt int etc. are called type aliases. sorry.

Output JSON file for each src and dst.

$ make swagger_extract
mkdir -p dst/swagger/convert
mkdir -p json/extracted
go-structjson -target src/github > json/extracted/src.json
go-structjson -target dst/swagger/gen/def > json/extracted/dst.json

Generating code for conversion from JSON of extracted type information

Well, I was able to get the code of the conversion source and conversion destination struct. Next is the automatic generation of conversion processing. I am making a library called goconvert for this time. The fact that it is being made appears in the place where only to do is written in the readme. I will use this for the time being. Actually, the code for converting src to dst using convert.py created using this library. To generate.

$make swagger_convert
mkdir -p dst/swagger/convert
python src/convert.py --logger=DEBUG --src json/extracted/src.json --dst json/extracted/dst.json --override json/extracted/convert.json > dst/swagger/convert/autogen.go || rm dst/swagger/convert/autogen.go
INFO:goconvert.builders:start register function AuthenticatedUserToUser: ['github.com/podhmo/advent2016/src/github.AuthenticatedUser'] -> ['github.com/podhmo/advent2016/dst/swagger/gen/def.User']
DEBUG:goconvert.builders:resolve: avatarurl ('string',) -> avatarurl ('string',)
DEBUG:goconvert.minicode:gencode: ('string',) -> ('string',)
...
NotImplementedError: mapping not found ('time.Time',) -> ('string',)

Oh. It doesn't work.

I can't seem to generate the code for string from time.Time. Looking inside, it seemed that I was having trouble handling Created At.

src (generated from api response)

type AuthenticatedUser struct {
	AvatarURL         string    `json:"avatar_url" example:"https://github.com/images/error/octocat_happy.gif"`
...
	CreatedAt         time.Time `json:"created_at" example:"2008-01-14T04:33:35Z"`

dst (generated from swagger.yaml)

// User user
// swagger:model user
type User struct {

	// avatar url
	AvatarURL string `json:"avatar_url,omitempty"`

...
	// ISO 8601 format: YYYY-MM-DDTHH:MM:SSZ
	CreatedAt string `json:"created_at,omitempty"`

Well, this is, in a sense, natural and unavoidable. No matter how much you trace the go type information, you don't know how to convert time.Time to string. Only the developer knows. Write the following time.Time to string conversion process as primitive.go in the same package as the output destination of the conversion process.

package convert

import "time"

// TimeToString :
func TimeToString(t time.Time) string {
	return t.Format(time.RFC3339)
}

Then redo the JSON output. This time, go-funcjson is also added (By passing the JSON generated by go-funcjson, the handwritten conversion processing function will be used as it is at the time of automatic generation).

$ make swagger_extract
mkdir -p dst/swagger/convert
mkdir -p json/extracted
go-structjson -target src/github > json/extracted/src.json
go-structjson -target dst/swagger/gen/def > json/extracted/dst.json
go-funcjson -target dst/swagger/convert > json/extracted/convert.json || echo '{}' > json/extracted/convert.json
...
-- write: convert --
gofmt -w dst/swagger/convert/autogen.go

This time it was successful. The following code is generated (dst / swagger / convert / autogen.go).

package convert

import (
	def "github.com/podhmo/advent2016/dst/swagger/gen/def"
	github "github.com/podhmo/advent2016/src/github"
)

// AuthenticatedUserToUser :
func AuthenticatedUserToUser(from *github.AuthenticatedUser) *def.User {
	to := &def.User{}
	to.AvatarURL = from.AvatarURL
	to.Bio = from.Bio
	to.Blog = from.Blog
	tmp1 := (int64)(from.Collaborators)
	to.Collaborators = tmp1
	to.Company = from.Company
	to.CreatedAt = TimeToString(from.CreatedAt)
	tmp2 := (int64)(from.DiskUsage)
	to.DiskUsage = tmp2
	to.Email = from.Email
	tmp3 := (int64)(from.Followers)
	to.Followers = tmp3
	tmp4 := (int64)(from.Following)
	to.Following = tmp4
	to.GravatarID = from.GravatarID
	to.Hireable = from.Hireable
	to.HTMLURL = from.HTMLURL
	tmp5 := (int64)(from.ID)
	to.ID = tmp5
	to.Location = from.Location
	to.Login = from.Login
	to.Name = from.Name
	tmp6 := (int64)(from.OwnedPrivateRepos)
	to.OwnedPrivateRepos = tmp6
	to.Plan = PlanToUserPlan(&(from.Plan))
	tmp7 := (int64)(from.PrivateGists)
	to.PrivateGists = tmp7
	tmp8 := (int64)(from.PublicGists)
	to.PublicGists = tmp8
	tmp9 := (int64)(from.PublicRepos)
	to.PublicRepos = tmp9
	tmp10 := (int64)(from.TotalPrivateRepos)
	to.TotalPrivateRepos = tmp10
	to.Type = from.Type
	to.URL = from.URL
	return to
}

// PlanToUserPlan :
func PlanToUserPlan(from *github.Plan) *def.UserPlan {
	to := &def.UserPlan{}
	tmp11 := (int64)(from.Collaborators)
	to.Collaborators = tmp11
	to.Name = from.Name
	tmp12 := (int64)(from.PrivateRepos)
	to.PrivateRepos = tmp12
	tmp13 := (int64)(from.Space)
	to.Space = tmp13
	return to
}

The TimeToString () added in primitive.go is also used in the CreatedAt part. Will it compile?

$ (cd dst/swagger/convert/; go install)

It looks okay.

Actually execute the conversion process

Let's actually use the generated conversion processing code. Execute the following code.

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"os"

	"github.com/davecgh/go-spew/spew"
	"github.com/podhmo/advent2016/dst/swagger/convert"
	"github.com/podhmo/advent2016/src/github"
)

func main() {
	decoder := json.NewDecoder(os.Stdin)
	var from github.AuthenticatedUser
	if err := decoder.Decode(&from); err != nil {
		log.Fatal(err)
	}

	{
		fmt.Println("------------------------------")
		fmt.Println("source:")
		fmt.Println("------------------------------")
		spew.Dump(from)
	}
	{
		to := convert.AuthenticatedUserToUser(&from)
		fmt.Println("------------------------------")
		fmt.Println("destination:")
		fmt.Println("------------------------------")
		spew.Dump(to)
	}
}

If the conversion processing code is generated successfully, the compilation will pass. You should be able to do it. Since it is troublesome, let's pass the response of the originally downloaded github api to the input.

$ make swagger_run1
cat json/github-get-authenticated-user.json | go run dst/swagger/main/spew/*.go
------------------------------
source:
------------------------------
(github.AuthenticatedUser) {
 AvatarURL: (string) (len=49) "https://github.com/images/error/octocat_happy.gif",
 Bio: (string) (len=17) "There once was...",
 Blog: (string) (len=23) "https://github.com/blog",
 Collaborators: (int) 8,
 Company: (string) (len=6) "GitHub",
 CreatedAt: (time.Time) 2008-01-14 04:33:35 +0000 UTC,
 DiskUsage: (int) 10000,
 Email: (string) (len=18) "[email protected]",
 EventsURL: (string) (len=53) "https://api.github.com/users/octocat/events{/privacy}",
 Followers: (int) 20,
 FollowersURL: (string) (len=46) "https://api.github.com/users/octocat/followers",
 Following: (int) 0,
 FollowingURL: (string) (len=59) "https://api.github.com/users/octocat/following{/other_user}",
 GistsURL: (string) (len=52) "https://api.github.com/users/octocat/gists{/gist_id}",
 GravatarID: (string) "",
 HTMLURL: (string) (len=26) "https://github.com/octocat",
 Hireable: (bool) false,
 ID: (int) 1,
 Location: (string) (len=13) "San Francisco",
 Login: (string) (len=7) "octocat",
 Name: (string) (len=16) "monalisa octocat",
 OrganizationsURL: (string) (len=41) "https://api.github.com/users/octocat/orgs",
 OwnedPrivateRepos: (int) 100,
 Plan: (github.Plan) {
  Collaborators: (int) 0,
  Name: (string) (len=6) "Medium",
  PrivateRepos: (int) 20,
  Space: (int) 400
 },
 PrivateGists: (int) 81,
 PublicGists: (int) 1,
 PublicRepos: (int) 2,
 ReceivedEventsURL: (string) (len=52) "https://api.github.com/users/octocat/received_events",
 ReposURL: (string) (len=42) "https://api.github.com/users/octocat/repos",
 SiteAdmin: (bool) false,
 StarredURL: (string) (len=59) "https://api.github.com/users/octocat/starred{/owner}{/repo}",
 SubscriptionsURL: (string) (len=50) "https://api.github.com/users/octocat/subscriptions",
 TotalPrivateRepos: (int) 100,
 Type: (string) (len=4) "User",
 URL: (string) (len=36) "https://api.github.com/users/octocat",
 UpdatedAt: (time.Time) 2008-01-14 04:33:35 +0000 UTC
}
------------------------------
destination:
------------------------------
(*def.User)(0xc4200ae140)({
 AvatarURL: (string) (len=49) "https://github.com/images/error/octocat_happy.gif",
 Bio: (string) (len=17) "There once was...",
 Blog: (string) (len=23) "https://github.com/blog",
 Collaborators: (int64) 8,
 Company: (string) (len=6) "GitHub",
 CreatedAt: (string) (len=20) "2008-01-14T04:33:35Z",
 DiskUsage: (int64) 10000,
 Email: (string) (len=18) "[email protected]",
 Followers: (int64) 20,
 Following: (int64) 0,
 GravatarID: (string) "",
 Hireable: (bool) false,
 HTMLURL: (string) (len=26) "https://github.com/octocat",
 ID: (int64) 1,
 Location: (string) (len=13) "San Francisco",
 Login: (string) (len=7) "octocat",
 Name: (string) (len=16) "monalisa octocat",
 OwnedPrivateRepos: (int64) 100,
 Plan: (*def.UserPlan)(0xc4204da1b0)({
  Collaborators: (int64) 0,
  Name: (string) (len=6) "Medium",
  PrivateRepos: (int64) 20,
  Space: (int64) 400
 }),
 PrivateGists: (int64) 81,
 PublicGists: (int64) 1,
 PublicRepos: (int64) 2,
 TotalPrivateRepos: (int64) 100,
 Type: (string) (len=4) "User",
 URL: (string) (len=36) "https://api.github.com/users/octocat"
})

It seems to be working. (By the way, if the structure is exactly the same, it may be easier to convert the value via JSON with json.Unmarshal after json.Marshal)

Makefile such as commands used for conversion is here.

Further talk when the existing type is updated

Actually, the src of the code used in the code conversion so far is the same as the previous article, but it is not completely the same. Last code I was commenting out the following part.

        # elif "://" in val:
        #     return "github.com/go-openapi/strfmt.Uri"

Uncomment this and try to generate it again.

$ make swagger_src
...
$ make swagger_extract
...
$ make swagger_convert
...
KeyError: 'Uri'

I have failed. Actually, this is because the library github.com/go-openapi/strfmt that I have been using since I wrote the previous article has been updated. did. Specifically, it seems that strfmt.Uri has changed to strfmt.URI.

Let's update it again and then generate it again.

$ go get -u github.com/go-openapi/strfmt
$ gsed -i 's/strfmt.Uri/strfmt.URI/' src/jsontogo/jsontogo.py
$ make swagger_src
$ make swagger_extract
$ make swagger_convert

This time it seems to have succeeded. Of course you can also perform conversions.

$ make swagger_run1
cat json/github-get-authenticated-user.json | go run dst/swagger/main/spew/*.go
------------------------------
source:
------------------------------
(github.AuthenticatedUser) {
 AvatarURL: (strfmt.URI) (len=49) https://github.com/images/error/octocat_happy.gif,
 Bio: (string) (len=17) "There once was...",
 Blog: (strfmt.URI) (len=23) https://github.com/blog,
 Collaborators: (int) 8,
 Company: (string) (len=6) "GitHub",
 CreatedAt: (time.Time) 2008-01-14 04:33:35 +0000 UTC,
 DiskUsage: (int) 10000,
 Email: (string) (len=18) "[email protected]",
 EventsURL: (strfmt.URI) (len=53) https://api.github.com/users/octocat/events{/privacy},
 Followers: (int) 20,
 FollowersURL: (strfmt.URI) (len=46) https://api.github.com/users/octocat/followers,
 Following: (int) 0,
 FollowingURL: (strfmt.URI) (len=59) https://api.github.com/users/octocat/following{/other_user},
 GistsURL: (strfmt.URI) (len=52) https://api.github.com/users/octocat/gists{/gist_id},
 GravatarID: (string) "",
 HTMLURL: (strfmt.URI) (len=26) https://github.com/octocat,
 Hireable: (bool) false,
 ID: (int) 1,
 Location: (string) (len=13) "San Francisco",
 Login: (string) (len=7) "octocat",
 Name: (string) (len=16) "monalisa octocat",
 OrganizationsURL: (strfmt.URI) (len=41) https://api.github.com/users/octocat/orgs,
 OwnedPrivateRepos: (int) 100,
 Plan: (github.Plan) {
  Collaborators: (int) 0,
  Name: (string) (len=6) "Medium",
  PrivateRepos: (int) 20,
  Space: (int) 400
 },
 PrivateGists: (int) 81,
 PublicGists: (int) 1,
 PublicRepos: (int) 2,
 ReceivedEventsURL: (strfmt.URI) (len=52) https://api.github.com/users/octocat/received_events,
 ReposURL: (strfmt.URI) (len=42) https://api.github.com/users/octocat/repos,
 SiteAdmin: (bool) false,
 StarredURL: (strfmt.URI) (len=59) https://api.github.com/users/octocat/starred{/owner}{/repo},
 SubscriptionsURL: (strfmt.URI) (len=50) https://api.github.com/users/octocat/subscriptions,
 TotalPrivateRepos: (int) 100,
 Type: (string) (len=4) "User",
 URL: (strfmt.URI) (len=36) https://api.github.com/users/octocat,
 UpdatedAt: (time.Time) 2008-01-14 04:33:35 +0000 UTC
}
------------------------------
destination:
------------------------------
(*def.User)(0xc42015e280)({
 AvatarURL: (string) (len=49) "https://github.com/images/error/octocat_happy.gif",
 Bio: (string) (len=17) "There once was...",
 Blog: (string) (len=23) "https://github.com/blog",
 Collaborators: (int64) 8,
 Company: (string) (len=6) "GitHub",
 CreatedAt: (string) (len=20) "2008-01-14T04:33:35Z",
 DiskUsage: (int64) 10000,
 Email: (string) (len=18) "[email protected]",
 Followers: (int64) 20,
 Following: (int64) 0,
 GravatarID: (string) "",
 Hireable: (bool) false,
 HTMLURL: (string) (len=26) "https://github.com/octocat",
 ID: (int64) 1,
 Location: (string) (len=13) "San Francisco",
 Login: (string) (len=7) "octocat",
 Name: (string) (len=16) "monalisa octocat",
 OwnedPrivateRepos: (int64) 100,
 Plan: (*def.UserPlan)(0xc420330420)({
  Collaborators: (int64) 0,
  Name: (string) (len=6) "Medium",
  PrivateRepos: (int64) 20,
  Space: (int64) 400
 }),
 PrivateGists: (int64) 81,
 PublicGists: (int64) 1,
 PublicRepos: (int64) 2,
 TotalPrivateRepos: (int64) 100,
 Type: (string) (len=4) "User",
 URL: (string) (len=36) "https://api.github.com/users/octocat"
})

in conclusion

This time I parsed the code of go and tried to generate the code of conversion from one struct to another. By the way, it was troublesome, so I stopped writing detailed internal stories. I used python for code generation, but it may be more beautiful and convenient to complete with go. There are some topics that I can talk about if I have the energy to talk about the analysis part of pointers, the handling of aliases (newtype), the code corresponding to slices, what to do if conversion correspondence is not found, so I may write it if I feel like it. Maybe.

bonus

The generated code has no momentum, so I'll add an extra. I will try to make it a pointer by playing with some type definitions of src.

diff --git a/src/github/authenticated_user.go b/src/github/authenticated_user.go
index 8893144..7e04142 100644
--- a/src/github/authenticated_user.go
+++ b/src/github/authenticated_user.go
@@ -11,8 +11,8 @@ AuthenticatedUser
 */
 // AuthenticatedUser : auto generated JSON container
 type AuthenticatedUser struct {
-	AvatarURL         strfmt.URI `json:"avatar_url" example:"https://github.com/images/error/octocat_happy.gif"`
-	Bio               string     `json:"bio" example:"There once was..."`
+	AvatarURL         *******strfmt.URI `json:"avatar_url" example:"https://github.com/images/error/octocat_happy.gif"`
+	Bio               *********string     `json:"bio" example:"There once was..."`
 	Blog              strfmt.URI `json:"blog" example:"https://github.com/blog"`
 	Collaborators     int        `json:"collaborators" example:"8"`
 	Company           string     `json:"company" example:"GitHub"`
@@ -35,10 +35,10 @@ type AuthenticatedUser struct {
 	OrganizationsURL  strfmt.URI `json:"organizations_url" example:"https://api.github.com/users/octocat/orgs"`
 	OwnedPrivateRepos int        `json:"owned_private_repos" example:"100"`
 	Plan              Plan       `json:"plan"`
-	PrivateGists      int        `json:"private_gists" example:"81"`
-	PublicGists       int        `json:"public_gists" example:"1"`
-	PublicRepos       int        `json:"public_repos" example:"2"`
-	ReceivedEventsURL strfmt.URI `json:"received_events_url" example:"https://api.github.com/users/octocat/received_events"`
+	PrivateGists      *int        `json:"private_gists" example:"81"`
+	PublicGists       **int        `json:"public_gists" example:"1"`
+	PublicRepos       ***int        `json:"public_repos" example:"2"`
+	ReceivedEventsURL ****strfmt.URI `json:"received_events_url" example:"https://api.github.com/users/octocat/received_events"`
 	ReposURL          strfmt.URI `json:"repos_url" example:"https://api.github.com/users/octocat/repos"`
 	SiteAdmin         bool       `json:"site_admin" example:"False"`
 	StarredURL        strfmt.URI `json:"starred_url" example:"https://api.github.com/users/octocat/starred{/owner}{/repo}"`

The generated code looks like this: It seems to work properly. Come to think of it, there were no slices in this example.

package convert

import (
	def "github.com/podhmo/advent2016/dst/swagger/gen/def"
	github "github.com/podhmo/advent2016/src/github"
)

// AuthenticatedUserToUser :
func AuthenticatedUserToUser(from *github.AuthenticatedUser) *def.User {
	to := &def.User{}
	if from.AvatarURL != nil {
		tmp1 := *(from.AvatarURL)
		if tmp1 != nil {
			tmp2 := *(tmp1)
			if tmp2 != nil {
				tmp3 := *(tmp2)
				if tmp3 != nil {
					tmp4 := *(tmp3)
					if tmp4 != nil {
						tmp5 := *(tmp4)
						if tmp5 != nil {
							tmp6 := *(tmp5)
							if tmp6 != nil {
								tmp7 := *(tmp6)
								tmp8 := (string)(tmp7)
								to.AvatarURL = tmp8
							}
						}
					}
				}
			}
		}
	}
	if from.Bio != nil {
		tmp9 := *(from.Bio)
		if tmp9 != nil {
			tmp10 := *(tmp9)
			if tmp10 != nil {
				tmp11 := *(tmp10)
				if tmp11 != nil {
					tmp12 := *(tmp11)
					if tmp12 != nil {
						tmp13 := *(tmp12)
						if tmp13 != nil {
							tmp14 := *(tmp13)
							if tmp14 != nil {
								tmp15 := *(tmp14)
								if tmp15 != nil {
									tmp16 := *(tmp15)
									if tmp16 != nil {
										tmp17 := *(tmp16)
										to.Bio = tmp17
									}
								}
							}
						}
					}
				}
			}
		}
	}
	tmp18 := (string)(from.Blog)
	to.Blog = tmp18
	tmp19 := (int64)(from.Collaborators)
	to.Collaborators = tmp19
	to.Company = from.Company
	to.CreatedAt = TimeToString(from.CreatedAt)
	tmp20 := (int64)(from.DiskUsage)
	to.DiskUsage = tmp20
	to.Email = from.Email
	tmp21 := (int64)(from.Followers)
	to.Followers = tmp21
	tmp22 := (int64)(from.Following)
	to.Following = tmp22
	to.GravatarID = from.GravatarID
	to.Hireable = from.Hireable
	tmp23 := (string)(from.HTMLURL)
	to.HTMLURL = tmp23
	tmp24 := (int64)(from.ID)
	to.ID = tmp24
	to.Location = from.Location
	to.Login = from.Login
	to.Name = from.Name
	tmp25 := (int64)(from.OwnedPrivateRepos)
	to.OwnedPrivateRepos = tmp25
	to.Plan = PlanToUserPlan(&(from.Plan))
	if from.PrivateGists != nil {
		tmp26 := *(from.PrivateGists)
		tmp27 := (int64)(tmp26)
		to.PrivateGists = tmp27
	}
	if from.PublicGists != nil {
		tmp28 := *(from.PublicGists)
		if tmp28 != nil {
			tmp29 := *(tmp28)
			tmp30 := (int64)(tmp29)
			to.PublicGists = tmp30
		}
	}
	if from.PublicRepos != nil {
		tmp31 := *(from.PublicRepos)
		if tmp31 != nil {
			tmp32 := *(tmp31)
			if tmp32 != nil {
				tmp33 := *(tmp32)
				tmp34 := (int64)(tmp33)
				to.PublicRepos = tmp34
			}
		}
	}
	tmp35 := (int64)(from.TotalPrivateRepos)
	to.TotalPrivateRepos = tmp35
	to.Type = from.Type
	tmp36 := (string)(from.URL)
	to.URL = tmp36
	return to
}

// PlanToUserPlan :
func PlanToUserPlan(from *github.Plan) *def.UserPlan {
	to := &def.UserPlan{}
	tmp37 := (int64)(from.Collaborators)
	to.Collaborators = tmp37
	to.Name = from.Name
	tmp38 := (int64)(from.PrivateRepos)
	to.PrivateRepos = tmp38
	tmp39 := (int64)(from.Space)
	to.Space = tmp39
	return to
}

<!-## Picking up fallen leaves !-- !-- misc !-- ! ---- What if I want to handwrite a part of the conversion function? ! ---- What is pointer support? ! ---- What is the interface support? ! --- What is the correspondence of slices? ! ---- What is the correspondence of type alias (new type)? ! ---- What is embed support? ! ---- What is the correspondence of non-convertible fields? ! --- Supports notational fluctuations (Id, ID, ...) ! --- Does it support functions with multiple arguments? ! --- What happens if a new handwritten function requires an additional argument? ! ---- Does it work if you want to use a function you defined in the middle of omitting the cast? ->

Recommended Posts

Let's write python code that parses go code and generates go code
Try to write python code to generate go code --Try porting JSON-to-Go and so on
Let's write a Python program and run it
Let's write python with cinema4d.
Create code that outputs "A and pretending B" in python
How to write a metaclass that supports both python2 and python3
That Python code has no classes ...
[Python] Let's master all and any
[PEP8] Take over the Python source code and write it neatly
[Python] Let's write briefly about comprehensions
[Code] Module and Python version output
This and that of python properties
Write selenium test code in python
Write Python code that applies type information at runtime using pydantic
[Python] I wrote a simple code that automatically generates AA (ASCII art)
Let's throw away JavaScript and write a web front end in Python!
Let's create a customer database that automatically issues a QR code in Python
Code reading of faker, a library that generates test data in Python
Installation of Visual studio code and installation of python
Write test-driven FizzBuzz code using Python doctest.
Let's try gRPC with Go and Docker
Write O_SYNC file in C and Python
Read and write JSON files in Python
Compress python data and write to sqlite
A nice nimporter that connects nim and python
Implemented memoization recursion and exploration in Python and Go
Python code that removes contiguous spaces into one
Compare xml parsing speeds with Python and Go
[Python3] Read and write with datetime isoformat with json
Links and memos of Python character code strings
Let's control EV3 motors and sensors with Python
Y / n processing in bash, python and Go
List of Python code to move and remember
Let's make an app that can search similar images with Python and Flask Part1
Let's make an app that can search similar images with Python and Flask Part2
mong --I tried porting the code that randomly generates Docker container names to Python -
[Note] How to write QR code and description in the same image with python
A note that runs an external program in Python and parses the resulting line