[GO] Kubernetes Scheduler Introduction to Homebrew

Introduction

The scheduling framework, which is a framework for creating your own Kubernetes scheduler, has been put in place. However, it is still in the alpha stage, so there is not enough information to actually make and try it, and it is a confusing situation. So, in this article, I will introduce a sample of a simple scheduling framework and how to run it. Furthermore, as an application example, we will explain the Coscheduling plug-in developed by kubernetes scheduler sigs.

What is a scheduler in the first place? What is a scheduling framework? What are the specifications?

I will explain only roughly. For more information, please refer to the good commentary already available.

• Scheduling Framework # k8sjp that even cats can understand
• Scheduling Framework # k8sjp to learn again in the fall of 2020
• Official Manual

Scheduler

A k8s management component that controls which node the pod starts on. By default, one default-scheduler is running.

• default-scheduler behavior can be extended with webhooks
• You can run multiple schedulers on one k8s.
• You can use the default and your own scheduler properly.
• You can specify the scheduler to be used in yaml's schedulerName property when creating a pod.

scheduling framework There were some cases where people who were not satisfied with the default-scheduler webhook extension developed a scheduler with full scratch. However, full scratching seems excessive, as some parts behave the same as the default-scheduler. Therefore, a framework was created that allows scheduler development by describing only the behavior of the part that you want to change based on default-scheduler. That is the scheduling framework. (maybe)

scheduling framework specifications

The behavior of default schdeuler is roughly divided into the following phases

• Scheduling prioritization (Queue Sort) for pods waiting to be assigned
• Determine a node for the highest priority pod from the pods waiting to be assigned (remove one pod from the queue)
• Exclude nodes that cannot execute the target pod (Filter)
• Score which of the remaining Nodes in the Filter is optimal (Scoring)
• Determine the Node to assign to the pod based on the score and send it to the kubernetes API server (bind)

Code only the phase part you want to replace for these By compiling in combination with the default scheduler You can create a scheduler that says, "Change only the part you want to change and the others are the same as the default."

Actually, there are more detailed phases. There are small extension points such as PreFilter that run before Filter. See Kubernetes: Understanding kube-scheduler at the source code level and Official Manual for more information.

Actually make and move

The sample source can be found on github. In addition, since I touched Go for the first time this time, please point out any strange points.

Implementation details

Create extension plugin structure

sample-scheduler.go

import(
    "k8s.io/kubernetes/cmd/kube-scheduler/app"
    //(Abbreviation)
    framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1"
)

type SampleScheduler struct {
	framework.PreFilterPlugin
	framework.PostBindPlugin
}

• This time, I will put the processing in the PreFilter and PostBind parts. Define the type of structure for that.

Create extension plugin method

sample-scheduler.go

func (cs *SampleScheduler) PostBind(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) {
	klog.Infof("pod %v is binded to %v", pod.Name, nodeName)
}

• Define the method corresponding to the extension point and describe the processing content. This time just write to standard output
• I probably don't have detailed documentation yet, so I'll have to write the method while looking at type errors, completions, and interface comments.
• Only postbind is excerpted, but prefilter needs to be created as well.

Create a plugin generation function

sample-scheduler.go

func New(_ runtime.Object, _ framework.FrameworkHandle) (framework.Plugin, error) {
	return &SampleScheduler{}, nil
}

Create an entry point by embedding a plugin in the scheduler

sample-scheduler.go

func main() {
	command := app.NewSchedulerCommand(
		app.WithPlugin(Name, New),
	)

	logs.InitLogs()
	defer logs.FlushLogs()

	if err := command.Execute(); err != nil {
		os.Exit(1)
	}
}

• By passing the created plugin generation function and name to the argument of app.NewSchedulerCommand, command with the extension plugin will be generated in the default scheduler.

compile

go build sample-scheduler.go

• The generated binaries contain plugins, but by default they will not run.
• Write the setting to enable the plug-in in the startup configuration file described later and pass it at startup

Prepare the scheduler start settings

scheduler-config.yaml

apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: false
clientConnection:
  kubeconfig: "/etc/kubernetes/admin.conf"
profiles:
- schedulerName: sample-scheduler
  plugins:
    preFilter:
      enabled:
        - name: SampleScheduler
    postBind:
      enabled:
        - name: SampleScheduler

• Put the created plugin in plugins. <extension point> .enabled
• Set the name of this scheduler with schedulerName: sample-scheduler. Used to specify the scheduler when creating a pod.

Start the scheduler

sudo ./sample-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kubernetes/scheduler.conf --config=scheduler-config.yaml --secure-port=10260

• Run on master
• Pass the startup settings earlier with --config =
• --Secure-port = 10260 is the listening port setting. It will start with the default of 10259 without it, but if there is a port conflict, you can avoid it by specifying it like this.
• --authentication-kubeconfig =, --authorization-kubeconfig = specifies the kubeconfig used by the scheduler. This time, we will use /etc/kubernetes/scheduler.conf.
• The state after execution is the state where both default-scheduler and sample-scheduler are running.

Pod creation for operation check yaml

apiVersion: v1
kind: Pod
metadata:
  name: sample-scheduler-pod
  labels:
    name: scheduler-example
spec:
  schedulerName: sample-scheduler
  containers:
  - name: container1
    image: k8s.gcr.io/pause:2.0

• schedulerName: Specify the scheduler created by sample-scheduler
• If it works normally, the following will be output to the standard output of the terminal executed by the scheduler.
• You can see that the klog.Infof ("pod% v is bounded to% v ", pod.Name, nodeName) described in the extension point has been executed.

I1225 07:09:30.512923   10010 sample-scheduler.go:27] pre filter called for pod sample-scheduler-pod
I1225 07:09:30.516019   10010 sample-scheduler.go:34] pod sample-scheduler-pod is binded to node-hoge-hoge

More advanced examples

The above sample has too little processing, and I think there is a big gap with the practical level. So, as a reference, I will briefly introduce how the k8s official extension plugin coscheduler uses the scheduling framework. The repository is kubernetes-sigs/scheduler-plugins

What is coscheduling?

The default scheduler schedules pods one by one. However, if you want to launch multiple linked pods, The following deadlocks can occur:

Mechanism of deadlock

Two groups of pods linked by three pods are in the Pending state

Two pods are launched from the blue group

Two pods are launched from the red group

• This can happen because the pods are scheduled one by one.

If one more pod is started in both red and blue, the process can be started, but it cannot be started because the resources are full.

• As a result, the booted pods will continue to wait for the rest of the boots, and the Pending pods will continue to wait for resources to become free, resulting in a deadlock.

Avoiding deadlocks with coscheduling

The above deadlock occurs because you start the pods one by one. Therefore, you can avoid it by starting them all at once. This is called coschedule (or gang scheduling).

A rough description of the implementation of the Coscheduling plugin

• Define a custom resource called PodGroup. Have them list the PodGroup name and the minimum number of pods required.
• Have the pod group that belongs to the yaml annotation when creating the pod be listed.
• The Coscheduling plugin does the following with Pod's Prefilter extension:
• Get the PodGroup name of the pod requested for scheduling from annotation. Get PodGroup settings from that PodGroup name using the k8s client library
• Check if the minimum required number of pods in the pod group can be started.
• Are there enough pods thrown at k8s?
• Is there enough space on each node?
• If it cannot be started, interrupt the scheduling. (Returns an Unschedulable error in the prefilter method)
• If a conflict with another scheduler causes resources to run out and the required number of pods cannot be started, cancel the scheduling of all pods in the group to avoid deadlock (I will omit it, but Permit and unreserve). I use extension points such as)

By using a combination of custom resources and scheduler plug-ins in this way, advanced scheduling is achieved.

Remarks

How to make the created scheduler the default scheduler?

• For now, it seems that you have to delete the default scheduler and register your own scheduler with the name default-scheduler.