[GO] Registered contents for each screen operation are linked to another system in chronological order via Firestore.

Assumed reader

――I know about GCP. -I have used Firestore (or Datastore) .. I don't know what it is. --Golang can be written as it is.

theme

Assuming that there is a very ordinary Web application that registers, updates, and refers to some information, I want to save the contents operated on the screen in the DB of my system and at the same time, keep the order of operations and reflect it in another system. Suppose there is a requirement. Another system provides an API for reflecting data for each function, so you can hit it, but the problem is that each API is heavy (= in short, the time from receiving a request to responding is 5 seconds or more. It takes). Therefore, simply thinking, it is not possible to hit the API synchronously as it is in the flow of operations on the screen. (If you do that, you will end up with a system that waits about 7 to 8 seconds each time the user registers or updates some information.)

So, the part that hits the API is asynchronous. Then, the part of "'Keep the order of operation`'" becomes suspicious. What should I do now? Since it will be asynchronous for the time being, do you want to put the content you want to reflect in the message queue?

• By the way, it is assumed that the system uses Google Cloud. There are many options, but for the time being, let's assume that the emphasis is on using managed services, and consider the following.

• Cloud Tasks
• Cloud Pub/Sub

However, if you try to use the above, it conflicts with this requirement in the following points.

--The execution order is not guaranteed. * --There is a possibility of duplicate execution.

https://cloud.google.com/tasks/docs/common-pitfalls?hl=ja

• For Cloud Pub / Sub, although it is a Beta version, a function called "Specify message order" has been added. For your reference. https://qiita.com/sky0621/items/3df3ae65b859b8196e39 However, this is also an order specification when messages are sent from Subscription. At least if you have a Push type Subscription, after pushing message 1 (= requesting to the specified endpoint), it will not wait for the response and push message 2. ("Push message 1"-> "Push message 2"-> ... without asking questions)

Also, regarding the "possibility of duplicate execution", it may be okay to update the same data twice for certain data updates, but for new data creation, yes. There is no. Two identical data will be created. In other words, even if the message is duplicated, a mechanism to eliminate it must be prepared separately. Maybe in general, give a unique ID at the origin of the message and use it as a "Memorystore" (https://cloud.google.com/memorystore?hl=ja) to see if it has been processed. ) Etc. (Insert an implementation that considers this message processed (that is, duplicated) and deletes it if it has already been registered.)

It is difficult to guarantee the execution order, but it is necessary to deal with duplicate execution, and there are too many elements that must be considered and implemented for what you want to do. So, I think more simply, "Isn't it just a matter of keeping the screen operation history in the database and processing it in the order of storage (hit the API of another system)?" However, I also want to consider scalability. So, I thought about adopting Datastore, but since it says "Firestore is the next generation Datastore.", Let's try using Firestore.

The big picture of the system

--"Web application" (this time, the actual Web API server) is Cloud Run --"RDB" (actual write logic is omitted in this source) is Cloud SQL --"NoSQL Store" is Firestore --"Synchronization service" (this time, the implementation that just puts the API to sleep for a few seconds in a simulated manner) is GKE

Web application overview

I can't always find a good salt plum for this kind of theme. This time, information such as "school", "grade", "class", "teacher", and "student" can be registered. (It's just a pseudo thing.)

• By the way, I am not a student.

end point

--/ add-school ・ ・ ・ Registration of" school " --/ add-grade ・ ・ ・ Registration of "grade" --/ add-class ・ ・ ・ Registration of "class" --/ add-teacher ・ ・ ・ Registration of "teacher" --/ add-student ・ ・ ・ Registration of" student "

Premise

--Go development environment has been built locally. --GCP contract completed. --Cloud SDK has been set up locally. --The key JSON file path (of the service account with all the required permissions) has been set in the local environment variable GOOGLE_APPLICATION_CREDENTIALS.

Development environment

OS - Linux(Ubuntu)

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"

#Backend

#Language --Golang

$ go version
go version go1.15.2 linux/amd64

IDE - Goland

GoLand 2020.2.3
Build #GO-202.7319.61, built on September 16, 2020

All sources this time

Web application

https://github.com/sky0621/go-publisher-fs/tree/v0.1.1

Sync service

https://github.com/sky0621/go-subscriber-fs/tree/v0.1.1

Source excerpt commentary

Web application

main.go

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"time"

	"cloud.google.com/go/firestore"
	"github.com/google/uuid"
	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

func main() {
	project := os.Getenv("PUB_PROJECT")

	e := echo.New()
	e.Use(middleware.Logger())
	e.Use(middleware.Recover())

	e.GET("/add-school", handler(project, "add-school"))
	e.GET("/add-grade", handler(project, "add-grade"))
	e.GET("/add-class", handler(project, "add-class"))
	e.GET("/add-teacher", handler(project, "add-teacher"))
	e.GET("/add-student", handler(project, "add-student"))

	e.Logger.Fatal(e.Start(":8080"))
}

func handler(project, path string) func(c echo.Context) error {
	return func(c echo.Context) error {
		ctx := c.Request().Context()

		client, err := firestore.NewClient(ctx, project)
		if err != nil {
			log.Fatal(err)
		}
		defer client.Close()

		order := fmt.Sprintf("%s:%s", path, createUUID())

		_, err = client.Collection("operation").Doc(order).
			Set(ctx, map[string]interface{}{
				"order":    order,
				"sequence": time.Now().UnixNano(),
			}, firestore.MergeAll)
		if err != nil {
			log.Fatal(err)
		}
		return c.String(http.StatusOK, order)
	}
}

func createUUID() string {
	u, err := uuid.NewRandom()
	if err != nil {
		log.Fatal(err)
	}
	return u.String()
}

Sync service

main.go

package main

import (
	"context"
	"log"
	"os"
	"strings"
	"time"

	"cloud.google.com/go/firestore"
)

func main() {
	ctx := context.Background()

	client, err := firestore.NewClient(ctx, os.Getenv("PUB_PROJECT"))
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	operationIter := client.Collection("operation").
		Where("sequence", ">", 0).OrderBy("sequence", firestore.Asc).Snapshots(ctx)
	defer operationIter.Stop()

	for {
		operation, err := operationIter.Next()
		if err != nil {
			log.Fatalln(err)
		}

		for _, change := range operation.Changes {
			ope, err := change.Doc.Ref.Get(ctx)
			if err != nil {
				log.Fatalln(err)
			}
			d := ope.Data()
			order, ok := d["order"]
			if ok {
				ods := strings.Split(order.(string), ":")
				if len(ods) > 0 {
					od := ods[0]
					switch od {
					case "add-school":
						time.Sleep(5 * time.Second)
					case "add-grade":
						time.Sleep(4 * time.Second)
					case "add-class":
						time.Sleep(3 * time.Second)
					case "add-teacher":
						time.Sleep(2 * time.Second)
					case "add-student":
						time.Sleep(1 * time.Second)
					}
				}
			}
			log.Printf("[operation-Data] %#+v", d)
		}
	}
}

Practice

Writer to Firestore

First, check the following parts.

Hit the following 5 endpoints in order from the top, and repeat it twice,

--/ add-school ・ ・ ・ Registration of" school " --/ add-grade ・ ・ ・ Registration of "grade" --/ add-class ・ ・ ・ Registration of "class" --/ add-teacher ・ ・ ・ Registration of "teacher" --/ add-student ・ ・ ・ Registration of" student "

Certainly, there are logs that were hit in order,

Documents are also collected in the Firestore.

• By default, Firestore seems to be arranged in ascending order of document ID, so it is not always arranged in the order of hitting the endpoint.

The side that gets information from the Firestore in ascending order of sequence

This time, the following part.

The following synchronization services are deployed.

Look at the container log to see if it is being processed in chronological order.

I made two rounds in the following order, so it was as expected.

--/ add-school ・ ・ ・ Registration of" school " --/ add-grade ・ ・ ・ Registration of "grade" --/ add-class ・ ・ ・ Registration of "class" --/ add-teacher ・ ・ ・ Registration of "teacher" --/ add-student ・ ・ ・ Registration of" student "

Within the sync service, reading / add-school from the Firestore puts it to sleep for 5 seconds, but of course it's not overtaken by reading / add-grade.

Summary

Like this, if you put the operation into the Firestore in chronological order (it is necessary to have a Nano level Unix Timestamp in the field where OrderBy can be applied so that it can be maintained), it will be in another system while keeping the order. You can create a linked system. However, of course, this alone is not enough to raise it to production. For example, documents are accumulating in the Firestore (even though it is unnecessary document once linked to another system), and therefore, when the synchronization service is redeployed, it also (although it has been processed) 1 There is a problem that the document is picked up from and started to be processed. About this I think the simple solution is "Isn't it okay to delete the document every time the process is finished?", But then, "Delete the document" triggered the process to finish processing one document. It happens that the same document is started to be processed again (because it has been deleted, an error occurs in the middle, or the second process runs as it is depending on the logic). so, There are still many points to think about, but for the time being, let's go to this point.