This article is the 24th day article of CyberAgent PTA Advent Calendar 2020.

My name is @ shunta-furukawa and I am an advertising system engineer at AbemaTV Business Development Headquarters. Today is Christmas Eve. Merry Christmas Eve! !!

Introduction

By the way, as mentioned in About AJA SSP and its technology, the Go language is often used in systems created by CyberAgent.

I think Go is a popular language because it has simple language specifications compared to other languages, so it is easy to handle while providing high performance. On the other hand, I think it is a language with a lot of description without abstracted clever notation.

――When you realize that you are writing, it's already night ... ――I want to speed up the implementation somehow ...

In such a case, it was streamlined to some extent by implementing it in a metaprogramming manner using go generate, Mustache, and YAML, so I would like to introduce a method.

A simple Sample Code has been prepared for this article, so please check it as well.

Metaprogramming in Go language

Writing a lot of Go language makes it vulnerable to change

As we proceed with the implementation in Go language, we often write code with a similar structure. After writing a certain amount of code, if there is a change in the specifications on the way, it is said that the affected parts are scattered Even though it is the same change, the amount of change is also large and it is genial.

In a similar example, "Go code generation by muscle" was helpful, so I would like to introduce it. When I tried to make changes to go-slack, I heard the following story.

When I was sending PR to github.com/nlopes/slack, I had to manually change 20 files to fix one mechanism.

As you can see, there are many cases where the amount of code changes is large despite one change.

Generate code from abstracted data to make it resistant to change

In order to be strong against such changes, it is effective to describe a code that is well abstracted and a conversion rule from the abstracted one to the actual code. By doing this, you can change only the abstracted definition and spread the change throughout the code with a small amount of change. This technique is his so-called ** metaprogramming **.

Metaprogramming-wikipedia

Metaprogramming is a type of programming technique that involves programming with high-level logic that produces logic with a pattern, and defining that high-level logic, rather than coding the logic directly. ..

In the previous example of Go code generation by muscle, by generating code from the file endpoint.json, even if there is a change, this json file can be modified. .. It can be said that we are doing metaprogramming in a different language.

Go language go generate

The Go language provides a mechanism called go generate that generates go code in advance.

When go generate is executed, when go generate [PATH] is executed, it extracts the file with the comment // go: generate ~ from the go file in the path passed as an argument. , It is a mechanism that executes the command written after this comment.

If you implement the auto-generate part and write this comment, just execute go generate ./... to complete the code generation.

This time, we will call go from this go generate to generate code. For more information on go generate, please refer to the following articles.

Reference of go generate

-Generate Go files by processing source (Official) -Go generate complete introduction --go generate best practices

Code generation using YAML and Mustache

From now on, I'm going to create a code generator that can be run with go generate using YAML and Mustache while actually showing the code.

Actual code is here.

Premise (what you want to achieve)

Since it's Christmas, we have prepared a sample code with the following concept.

--Various gifts (Gift) prepared by Santa are in the Santa bag (sack). --There are children (kids) waiting for Santa's gifts. --Each child has their own gift requirements. —— Outputs which child wants which gift.

(Please forgive me that it is not a simple sample because it took some scale to feel the benefits of code generation.)

Click here for the actual code

`main.go`


package main

import (
	"fmt"
	"strings"

	"../app/gift"
	"../app/kid"
)

func main() {

	//In the bag of Santa Claus
	sack := make([]gift.Gift, 0)

	//Fill the Santa Claus bag
	sack = append(sack, gift.NewSportsCar())

	//The kids
	kids := make([]kid.Kid, 0)

	//Fill the Santa Claus bag
	kids = append(kids, kid.NewTaro())

	//Show presents for children
	giftlist := make([]string, 0)
	for _, gift := range sack {
		giftlist = append(giftlist, gift.Display())
	}

	fmt.Printf("=======================================\n===【:*･ ﾟ ☆ † Merry X'mas †.¡.:*･ ゜]=== \n=======================================\n\n Yoita present: \n  - %s\n\n", strings.Join(giftlist, "\n  - "))

	for _, kid := range kids {
		fmt.Printf("%s\n what you want: \n  %s\n toys you can get: \n  %s \n",
			kid.Display(),
			kid.Wishlist(),
			kid.CanGet(sack),
		)
	}

	fmt.Printf("\n! !! !! Merry Christmas! !! !!\n")

}

--NewSportsCar returns SportsCar and SportsCar implements the Gift interface. --Similarly, NewTaro returns Taro and Taro implements the Kid interface.

Here is an example output:

=======================================
===【:*･ ﾟ ☆ † Merry X'mas †.¡.:*･ ゜]=== 
=======================================

A gift: 
  -Sports car [vehicle|Grime|For men]

☆ ★ ☆ ★ Taro-kun(4) ★☆★☆
What you want: 
I want a red vehicle
Toys you can get: 
Sports car [vehicle|Grime|For men]

!! !! !! Merry Christmas! !! !!

This time, I generated the implementation of SportsCar and Taro based on the data described in YAML. By changing YAML, we aim to behave as intended even if we increase the variation.

Below is the target code you want to generate.

SportsCar code

`sportscar.go`


package gift

import "fmt"

// SportsCar represents SportCar.
type SportsCar struct {
	Name     string
	Category string
	Color    string
	Gender   string
}

// NewSportsCar returns new SportCar.
func NewSportsCar() SportsCar {
	return SportsCar{
		Name:     "sports car",
		Category: "Vehicle",
		Color:    "Grime",
		Gender:   "boy",
	}
}

// Display returns spec of SportCar.
func (g SportsCar) Display() string {
	return fmt.Sprintf(`%s【%s|%s|%For s]`,
		g.Name,
		g.Category,
		g.Color,
		g.Gender,
	)
}

// GetName returns its name.
func (g SportsCar) GetName() string {
	return g.Name
}

// GetCategory returns its category.
func (g SportsCar) GetCategory() string {
	return g.Category
}

// GetGender returns its gender.
func (g SportsCar) GetGender() string {
	return g.Gender
}

// GetColor returns its color.
func (g SportsCar) GetColor() string {
	return g.Color
}

Taro code

`taro.go`


package kid

import (
	"strings"

	"../gift"
)

// Taro represents Taro.
type Taro struct {
	Name   string
	Gender string
	Age    int
}

// NewTaro returns instance of Kids Impl as Taro
func NewTaro() Taro {
	return Taro{
		Name:   "Taro",
		Gender: "boy",
		Age:    4,
	}
}

// Display returns name of the kid.
func (k Taro) Display() string {
	return "☆ ★ ☆ ★ Taro-kun(4) ★☆★☆"
}

// Wishlist returns the kid's wishlist.
func (k Taro) Wishlist() string {
	return "I want a red vehicle"
}

// CanGet returns gift the kid can get.
func (k Taro) CanGet(sack []gift.Gift) string {
	gds := make([]string, 0)

	for _, gift := range sack {
		if gift.GetColor() == "Grime" && gift.GetCategory() == "Vehicle" {
			gds = append(gds, gift.Display())
		}
	}

	if len(gds) == 0 {
		return "I couldn't find what I wanted..."
	}
	return strings.Join(gds, "\n  ")
}

YAML preparation

In order to generate the above code, consider the parts that can be shared and the information that is different for each individual (Gift/Kid). Structure the parts that differ from individual to individual and define them in YAML.

See here for YAML itself:

For example, in the case of SportsCar, I picked up the following parts.

`gifts.yml`


gifts: 
  - name: SportsCar
    jname:sports car
    category:Vehicle
    color:Grime
    gender:boy

Mustache preparation

Mustache is a type of template language. This time we will use it to generate the Go language, but there are many implementations in other languages as well.

Reference: -Mustache Official

Mustache Template Engine for Go

** {{}} ** This double curly braces is characteristic and seems to be called Mustache because it resembles a mustache.

Now, in Mustache, write a template to populate the YAML data.

`gift.mustache`


package gift

import "fmt"

// {{Name}} represents SportCar.
type {{Name}} struct {
	Name     string
	Category string
	Color    string
	Gender   string
}

// New{{Name}} returns new SportCar.
func New{{Name}}() {{Name}} {
	return {{Name}}{
		Name:     "{{JName}}",
		Category: "{{Category}}",
		Color:    "{{Color}}",
		Gender:   "{{Gender}}",
	}
}

// Display returns spec of SportCar.
func (g {{Name}}) Display() string {
	return fmt.Sprintf(`%s【%s|%s|%For s]`,
		g.Name,
		g.Category,
		g.Color,
		g.Gender,
	)
}

// GetName returns its name.
func (g {{Name}}) GetName() string {
	return g.Name
}

// GetCategory returns its category.
func (g {{Name}}) GetCategory() string {
	return g.Category
}

// GetGender returns its gender.
func (g {{Name}}) GetGender() string {
	return g.Gender
}

// GetColor returns its color.
func (g {{Name}}) GetColor() string {
	return g.Color
}

When working with Mustache in Go language, you need to be able to access Go structures. The contents of {{}} must be written in uppercase.

Write a go program to generate code

Once you have YAML and Mustache ready, create a go file that will eventually generate the code.

The general flow is as follows.

Unmarshal YAML into a Go struct
Pour into Mustache
Output the result of pouring as a file

is.

Below is a part of the actual code

1. Unmarshal YAML into a Go struct

Predefine the struct that matches the structure of yaml.

`gift.go`


package model

type (
	// GiftContainer wrap interfaces
	GiftContainer struct {
		Gifts []Gift `yaml:"gifts"`
	}

	// Gift represents kid
	Gift struct {
		Name     string `yaml:"name"`
		JName    string `yaml:"jname"`
		Category string `yaml:"category"`
		Color    string `yaml:"color"`
		Gender   string `yaml:"gender"`
	}
)

Then specify the yaml file and unmarshal to this struct

`main.go`


package main

import (...)
... 

//go:generate go run main.go
func main() {

	//Kid generation...abridgement

	//Gift generation
	giftBuf, err := ioutil.ReadFile(giftsInputPath)
	if !errors.Is(err, nil) {
		panic(err)
	}
	giftContainer := model.GiftContainer{}
	giftContainer.Gifts = make([]model.Gift, 0)
	err = yaml.Unmarshal(giftBuf, &giftContainer)
	if !errors.Is(err, nil) {
		panic(err)
	}
	generateGifts(giftContainer.Gifts)

        // ...abridgement
}

2. Pour into Mustache-> 3. Output the pour result as a file

`main.go`


func generateGifts(gifts []model.Gift) {
	//Loading template
	giftTemplate, err := mustache.ParseFile(giftTemplatePath)
	if !errors.Is(err, nil) {
		panic(err)
	}
	//Export from Gifts
	for _, p := range gifts {
		//When the number of templates increases, it can be expanded by increasing the elements here.
		for _, r := range []Renderer{
			{
				Tmpl: giftTemplate,
				Path: giftOutputPath,
			},
		} {
			outputFile(r, p.Name, p)
		}
	}
}

func outputFile(r Renderer, name string, data interface{}) {
	output, err := r.Tmpl.Render(data)
	if !errors.Is(err, nil) {
		panic(err)
	}

	outputBytes, err := format.Source([]byte(output))
	if !errors.Is(err, nil) {
		panic(err)
	}
	// outputBytes := []byte(output)

	//Create a directory and ignore it if it exists.
	_ = os.MkdirAll(r.Path, 0755)

	// []Overwriting byte to file.
	filename := r.Path + strings.ToLower(name) + r.Postfix + ".go"
	err = ioutil.WriteFile(filename, outputBytes, 0755)
	if err != nil {
		panic(err)
	}

	fmt.Printf("mustache: generate %s\n", filename)

}

At this point, you're ready for metaprogramming.

(The implementation of Kids will be long, so I will omit it. Please check the repository.)

Actual code generation

At this point, make sure that the code you wrote at the beginning is spit out by the code generator.

Actually, the following comment line has been added to main.go.

//go:generate go run main.go

By writing this, when you execute the generate command, main.go will be executed. The code will be generated.

go generate ./...

With this, if there is no difference, it is complete.

Increase and extend YAML

Since it's a big deal, by increasing YAML Try to make it lively by increasing the number of presents and children.

Try rewriting the YAML to generate and run the code

`gifts.yaml`


gifts: 
  - name: SportsCar
    jname:sports car
    category:Vehicle
    color:Grime
    gender:boy
  - name: GabageCollector
    jname:Garbage
    category:Vehicle
    color:Blue
    gender:boy
  - name: Sword
    jname:Tsurugi
    category:Weapon
    color:Blue
    gender:boy
  - name: Gun
    jname:Handgun
    category:Weapon
    color:Kuro
    gender:boy
  - name: TeddyBear
    jname:Plush bear
    category:Doll
    color:Brown
    gender:girl
  - name: BabyDoll
    jname:Akachan doll
    category:Doll
    color:Pink
    gender:girl
  - name: PrincessDoll
    jname:Doll
    category:Doll
    color:Pink
    gender:girl
  - name: CatBulletTrain
    jname:Cat's Shinkansen
    category:Vehicle
    color:Pink
    gender:girl
  - name: HeroToy 
    jname:Yusha's figure
    category:Doll
    color:Kuro
    gender:boy

`kids.yml`


kids: 
  - name: Taro
    jname:Taro
    gender:boy
    age: 4 
    preferences: 
      - attribute: Color 
        value:Grime
      - attribute: Category 
        value:Vehicle
  - name: Jiro
    jname:Jiro
    gender:boy
    age: 5 
    preferences: 
      - attribute: Color 
        value:Kuro
    genderBiased: true
  - name: Yuta
    jname:Yuta
    gender:boy
    age: 7
    preferences: 
      - attribute: Name
        value:Handgun
  - name: Yuuko
    jname:Yuko
    gender:girl
    age: 10 
    preferences: 
      - attribute: Color 
        value:Pink
      - attribute: Gender
        value:girl
  - name: Hinako
    jname:Hinako
    gender:girl
    age: 8
    preferences: 
      - attribute: Category 
        value:Doll

In this state, do the following:

go generate ./... 
go run cmd/main.go

Then you can see that many presents and many children are increasing.

=======================================
===【:*･ ﾟ ☆ † Merry X'mas †.¡.:*･ ゜]=== 
=======================================

A gift: 
  -Sports car [vehicle|Grime|For men]
  -Garbage|Blue|For men]
  -Sword [Buki]|Blue|For men]
  -Weapon [Buki]|Kuro|For men]
  -Plush bear [doll]|Brown|For girls]
  -Akachan doll [doll]|Pink|For girls]
  -Doll [doll]|Pink|For girls]
  -Cat's Shinkansen [Vehicle|Pink|For girls]
  -Yusha's figure [Ningyo|Kuro|For men]

☆ ★ ☆ ★ Taro-kun(4) ★☆★☆
What you want: 
I want a red vehicle
Toys you can get: 
Sports car [vehicle|Grime|For men]

☆ ★ ☆ ★ Jiro-kun(5) ★☆★☆
What you want: 
I want something from Kuro
But I don't want it for men.
Toys you can get: 
Weapon [Buki]|Kuro|For men]
Yusha's figure [Ningyo|Kuro|For men]

☆ ★ ☆ ★ Yuta-kun(7) ★☆★☆
What you want: 
I want a kenju
Toys you can get: 
Weapon [Buki]|Kuro|For men]

☆ ★ ☆ ★ Yuko-chan(10) ★☆★☆
What you want: 
I want a pink one for girls
Toys you can get: 
Akachan doll [doll]|Pink|For girls]
Doll [doll]|Pink|For girls]
Cat's Shinkansen [Vehicle|Pink|For girls]

☆ ★ ☆ ★ Hinako-chan(8) ★☆★☆
What you want: 
I want a doll
Toys you can get: 
Plush bear [doll]|Brown|For girls]
Akachan doll [doll]|Pink|For girls]
Doll [doll]|Pink|For girls]
Yusha's figure [Ningyo|Kuro|For men]


!! !! !! Merry Christmas! !! !!

at the end

So far, I've used Mustache and YAML to implement the go language like metaprogramming. If you take advantage of this technique, for example, even if there are some changes, you can change the YAML and it will be immediately reflected in the whole.

I hope you can make good use of it and lead a comfortable program life!

Tomorrow is the last day. This is an article by yamaguchi_naoto! Stay tuned: D

Then **! !! !! Merry Christmas! !! !! ** **

Metaprogram go with YAML + Mustache + go-generate

Introduction

Metaprogramming in Go language

Writing a lot of Go language makes it vulnerable to change

Generate code from abstracted data to make it resistant to change

Go language go generate

Reference of go generate

Code generation using YAML and Mustache

Premise (what you want to achieve)

main.go

SportsCar code

sportscar.go

Taro code

taro.go

YAML preparation

gifts.yml

Mustache preparation

gift.mustache

Write a go program to generate code

1. Unmarshal YAML into a Go struct

gift.go

main.go

2. Pour into Mustache-> 3. Output the pour result as a file

main.go

Actual code generation

Increase and extend YAML

gifts.yaml

kids.yml

at the end

`main.go`

`sportscar.go`

`taro.go`

`gifts.yml`

`gift.mustache`

`gift.go`

`main.go`

`main.go`

`gifts.yaml`

`kids.yml`