SoFunction
Updated on 2025-03-03

go parsing YAML files (multi-document parsing)

Overview

Abstract: In kubernetes secondary development, yaml files are commonly used to create resources, such as Pod and deployment. Before creating a resource using client-go, you need to convert the yaml file to the resource object obj. This article introduces the encoding and decoding of yaml files.

text

Yaml documentation introduction

YAML is a lightweight data serialization format, used to transfer data between different platforms, and also used in scenarios such as configuration files. In YAML, the entire data file is called a "document", and in general, each YAML file contains only one document. In kubernetes, in order to create resource convenience, the commonly used YAML files of kubectl do include multiple documents.
A document can contain one or more "nodes", each node containing one or more key-value pairs. Relationship between key-value pairscolon(:)Indicates that the key and value of the key-value pair are separated by spaces, and the entire line of the key and value pair is at the end of the line.There can't be extra spaces. A node can be usedMinus sign (-)The beginning means that it is a sequence, i.e. a list.

# This is a YAML documentname:  Tom
age: 30
hobbies: 
  - reading
  - running
  - swimming

In this document, use"#"symbolRepresents comments, each node's key and value are separated by a colon, and nodes are separated by a newline character. In the "hobbies" node, use the "-" symbol to indicate that this is a sequence (i.e. a list), the items in the listUse the "-" symbolStart to indicate

YAML documentation features include：

  • Ease of readability:YAML uses indented and structured formats, making it very readable. It uses spaces instead of special symbols to represent hierarchical relationships, making the document readable.

  • Simplicity:YAML uses concise syntax to represent data structures, avoiding redundancy and duplication. It supports the use of features such as references and folding blocks to simplify the writing of documents.

  • Nestingability: YAML supports nested structures that can contain other nodes in one node, representing more complex data structures. This nested structure makes the organization of data more flexible and clear.

  • Cross-platform:YAML is a programming language-independent data format that can be easily exchanged and shared between different programming languages ​​and platforms.

  • Strong data expression ability: YAML supports representing various data types, including strings, integers, floating point numbers, boolean values, dates, time, regular expressions, etc. It also supports data structures such as lists and hash tables.

  • Comments: YAML allows the inclusion of comments in a document, which begins with the "#" symbol and can be used to provide additional instructions or comments about the data.

  • Scalability: YAML allows users to expand their functions by customizing tags and types to adapt them to more application scenarios and needs.

  • Compatibility with profiles: YAML is very popular in profiles due to its legibility and simplicity. It is widely used in configuration files of various software and frameworks, such asKubernetes、Docker Compose、Ansiblewait.

    Overall, YAML is a simple, readable and feature-rich data serialization format that is ideal for configuration files and data exchange.

Commonly used three-party libraries for processing YAML documents"/yaml.v2"

Common methods include

  • () Convert structure object to YAML string

  • () Convert YAML string to structure object

Code Example

Conversion between yaml file and structure

Create a YAML file for testing

---
name: xianwei
age: 18
city: shenzhen

Writing test code

// go test -run="^TestPersonStruct$"  -v
func TestPersonStruct(t *) {
  // Define a structure	type Person struct {
		Name string `yaml:"name"`
		Age  int    `yaml:"int"`
		City string `yaml:"city"`
	}
	byteStream, err := ("yamldir/")
	if err != nil {
		("cann not open =%v\n", err)
		return
	}

	// Convert yaml string to person object	p := new(Person)
	if err := (byteStream, p); err != nil {
		("=%v\n", err)
		return
	}
	("person= %+v\n", p)

	// Convert Person object to yaml string	yamlstr, err := (p)
	if err != nil {
		("=%v\n", err)
		return
	}
	("yamlstr= %+v\n", string(yamlstr))
}

The above code implements:

  • Convert yaml string to person object
  • Person object converted to yaml string

Code output

=== RUN   TestPersonStruct
person= &{Name:xianwei Age:0 City:shenzhen}
yamlstr= name: xianwei
int: 0
city: shenzhen

--- PASS: TestPersonStruct (0.00s)
PASS

Parses yaml file and create pod

When creating a pod using yaml, it is important to note that you need to convert the yaml string to a json string, and then the json string Unmarshal into a pod object. The reason is that in the type definition (source code is shown below), only the conversion of the defined string is specified.

type Pod struct {
	 `json:",inline"`
	 `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
	Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
	Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

Create a YAML file for testing

---
apiVersion: v1
kind: Pod
metadata:
  name: mynginx
spec:
  containers:
    - name: nginx
      image: nginx:1.14.2

Writing test code

// go test -run="^TestYamlfilePod$"  -v
func TestYamlfilePod(t *) {
	// Convert yaml file to pod object	byteStream, err := ("yamldir/")
	if err != nil {
		("cann not open =%v\n", err)
		return
	}

	// Convert yaml string to json string	pod := new()
	// yaml2 means the three-party library "/apimachinery/pkg/util/yaml"	jsonstr, err := (byteStream)
	if err != nil {
		("=%v\n", err)
		return
	}
	("jsonstr= %v\n", string(jsonstr))

	// Convert json string to pod structure	err = (jsonstr, &pod)
	if err != nil {
		("=%v\n", err)
		return
	}
	("pod= %v\n", pod)

	// clientset get	restConfig, err := ("", "/Users/80280051/.kube/dg11test/config")
	if err != nil {
		panic(err)
	}
	K8sClientSet, err := (restConfig)
	if err != nil {
		panic(err)
	}

	// Create pod	_, err = K8sClientSet.CoreV1().Pods("default").Create(pod)
	if err != nil {
		("cann not create =%v\n", err)
		return
	}
	
}

The above code implements:

  • Convert yaml string to json string
  • Then convert the pod object from json string
  • Finally, create pod using clientset

Code output

=== RUN   TestYamlfilePod
jsonstr= {"apiVersion":"v1","kind":"Pod","metadata":{"name":"mynginx"},"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx"}]}}
pod= &Pod{ObjectMeta:{mynginx      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:PodSpec{Volumes:[]Volume{},Containers:[]Container{Container{Name:nginx,Image:nginx:1.14.2,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{},HostAliases:[]HostAlias{},PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:nil,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}
--- PASS: TestYamlfilePod (0.05s)
PASS

Processing of multiple documents in yaml files

When creating a Pod using kubectl or client-go, multiple YAML documents are often written to a YAML file. In this case, it can be divided by string "\n-" and processed separately.

Create a YAML file for testing

---
apiVersion: v1
kind: Pod
metadata:
  name: mynginx
spec:
  containers:
    - name: nginx
      image: nginx:1.14.2
---
apiVersion: v1
type: Opaque
stringData:
  userdata: "eGlhbndlaTExMQo="
kind: Secret
metadata:
  name: mySecret

Writing test code

// go test -run="^TestYamlfileMultiDocument" -mod=vendor -v
func TestYamlfileMultiDocument(t *) {
	// Convert yaml file to yaml string	fp, err := ("yamldir/")
	if err != nil {
		("cann not open =%v\n", err)
		return
	}
	// Define the separator	const yamlSeparator = "\n---"

	pod := new()
	secret := new()
	yamlsplit := (string(fp), yamlSeparator)
	("len of yamlsplit = ", len(yamlsplit))
	for index, yamlObj := range yamlsplit {
		// Process the first yaml document and convert it into a pod object		if index == 0 {
			// Convert to object			("obj=", yamlObj)
			if err = ([]byte(yamlObj), pod); err != nil {
				("yaml unmarshal error of  = %v\n", err)
				return
			}
			("pod obj =%+v\n", pod)
		}
		// Process the second yaml document and convert it into a secret object		if index == 1 {
			("obj=", yamlObj)
			// Convert to object			if err = ([]byte(yamlObj), secret); err != nil {
				("yaml unmarshal error of  = %v\n", err)
				return
			}
			("secret obj =%+v\n", secret)
		}

	}

}

The above code implements:

  • Segment the yaml string by "\n-"
  • Convert the first yaml document after segmentation into a pod object
  • Convert the second yaml document after splitting into a secret object

Code output

=== RUN   TestYamlfileMultiDocument
len of yamlsplit =  2
obj= ---
apiVersion: v1
kind: Pod
metadata:
  name: mynginx
spec:
  containers:
    - name: nginx
      image: nginx:1.14.2
pod obj =&Pod{ObjectMeta:{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:PodSpec{Volumes:[]Volume{},Containers:[]Container{Container{Name:nginx,Image:nginx:1.14.2,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{},HostAliases:[]HostAlias{},PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:nil,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:,Conditions:[]PodCondition{},Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[]ContainerStatus{},QOSClass:,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{},EphemeralContainerStatuses:[]ContainerStatus{},},}
obj= 
apiVersion: v1
type: Opaque
stringData:
  userdata: "eGlhbndlaTExMQo="
kind: Secret
metadata:
  name: mySecret
secret obj =&Secret{ObjectMeta:{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Data:map[string][]byte{},Type:Opaque,StringData:map[string]string{},}
--- PASS: TestYamlfileMultiDocument (0.00s)
PASS

Summarize

This article introduces the characteristics of YAML documents and uses code to demonstrate the processing of YAML documents by GO language, especially how to deal with it when there are multiple YAML documents in the YAML file.

This is the end of this article about go parsing YAML files (multi-document parsing). For more related go parsing YAML files, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!