SoFunction
Updated on 2025-03-01

Detailed explanation of the network storage protocol based on go instance

Main content

1. What is network storage?

2. What is iSCSI?

3.What is RDMA?

4. What is NVME-oF?

1. What is network storage?

Network storage is a technology that connects storage resources to a network so that multiple computers can share and access these storage resources. Network storage can be a local network-based storage (such as a local area network) or a wide area network-based storage (such as cloud storage).

Network storage usually uses protocols such as network file system (NFS) or storage area network (SAN) to achieve sharing and access to storage resources. Network storage has the advantages of high availability, scalability and flexibility, and can meet the needs of storage resources in different scenarios.

2. What is iSCSI?

iSCSI (Internet Small Computer System Interface) is a network-based storage protocol that maps remote storage devices to local disks, allowing computers to access remote storage devices just like local disks. The iSCSI protocol can run on Ethernet, Fibre Channel and other networks. It can establish a virtual SCSI bus between computers and storage devices, thereby realizing data transmission and management. The iSCSI protocol can simplify the management and deployment of storage systems and improve the reliability and performance of storage systems.

iSCSI can run in user mode or kernel mode. In Linux systems, the implementation of iSCSI is usually done using kernel modules, i.e. iscsi_tcp.ko and scsi_transport_iscsi.ko, which handle iSCSI transmission and SCSI command processing. However, there are also some user-mode iSCSI implementations, such as open-iscsi, which uses user-mode iSCSI daemon (iscsid) and iSCSI libraries (libiscsi). User-mode iSCSI implementations can provide more flexible configuration and management, but generally perform less than kernel-mode implementations.

Go-implemented iSCSI libraries and tools can be used to build and manage iSCSI storage systems, such as:

  • go-iscsi: An iSCSI library based on the Go language that supports the creation and management of iSCSI targets and LUNs (logical units). It can be used to build your own iSCSI storage system or integrate it into other applications.
  • iscsid: A Go-based iSCSI daemon that can be used to create and manage iSCSI targets and LUNs. It supports multiple backend storage engines, including local file system, Ceph, GlusterFS, etc.
  • open-iscsi: an open source iSCSI implementation, including an iSCSI client (initiator) and a server (target). It is implemented in C, but there are some Go tools and libraries that can be used to interact with it.
  • dqlite: A distributed SQLite database based on Go language implementation, supporting multi-node replication and high availability. It can be used to build distributed iSCSI storage systems or other distributed applications.

iscsid

iscsid is a daemon of the iSCSI protocol, which is mainly responsible for communicating with the iSCSI initiate (client) and forwarding iSCSI requests to the SCSI device. In file systems, iscsid is usually implemented using kernel modules.

The kernel module of iscsid is usually composed of two parts: iscsi_tcp.ko and scsi_transport_iscsi.ko. Among them, iscsi_tcp.ko is responsible for handling iSCSI's transport layer protocol, such as establishing and maintaining TCP connections, handling iSCSI login and logout, etc. scsi_transport_iscsi.ko is responsible for processing iSCSI SCSI commands, such as forwarding iSCSI requests to SCSI devices, processing responses to SCSI commands, etc.

When iscsid is started, it loads the iscsi_tcp.ko and scsi_transport_iscsi.ko kernel modules and creates corresponding character device files, such as /dev/sda, /dev/sdb, etc. These character device files correspond to SCSI devices on the iSCSI target (server). They can be used as ordinary hard disks and can be formatted, mounted, read and write, etc.

When the iSCSI initiator sends a SCSI command, the iscsid daemon forwards the command to the corresponding character device file, such as /dev/sda. The character device file forwards commands to a SCSI device, such as a hard disk. The SCSI device executes the command and returns the result. The character device file returns the result to the iscsid daemon. The iscsid daemon returns the result to the iSCSI initiate.

3.What is RDMA?

RDMA (Remote Direct Memory Access) is a high-performance, low-latency network transmission technology. It allows computers to directly access each other's memory, thereby avoiding unnecessary copying and context switching of data during transmission. RDMA technology mainly includes two protocols: InfiniBand and RoCE (RDMA over Converged Ethernet).

In RDMA technology, data transmission between computers no longer requires the intervention of the CPU, but is implemented directly through DMA transmission between the network adapter and the memory controller. This method can greatly reduce the CPU load and data transmission delay, thereby improving transmission efficiency and performance.

The RDMA workflow usually includes the following steps:

  • The application writes data to a buffer in memory.
  • The application sends data transfer requests through the RDMA API.
  • The network adapter sends a data transfer request to the target computer.
  • The network adapter of the target computer receives a data transfer request.
  • The memory controller of the target computer reads the requested data from memory and transfers the data to the memory controller of the source computer through a network adapter.
  • The memory controller of the source computer writes data to a buffer in memory and notifies the application that the transfer is completed.

RDMA technology requires hardware and software support, including network adapters, memory controllers, drivers, and RDMA APIs. In addition, RDMA technology also has some limitations and precautions, such as ensuring the order and consistency of data transmission, and avoiding memory leaks and data overflows.

The Go-IPoIB library is an RDMA over IPoIB protocol library implemented in Go language. Its underlying principle is mainly to realize RDMA communication through the RDMA-CM API. RDMA-CM is an RDMA communication manager based on the InfiniBand network. It provides a common RDMA communication interface that can run on different RDMA networks. The RDMA-CM API includes the RDMA-CM library and the RDMA-CM header file, which can be used on Linux systems.

The Go-IPoIB library provides a method to implement the RDMA over IPoIB protocol using the RDMA-CM API. When using the Go-IPoIB library, you need to use functions to create an RDMA connection and specify local and remote IP addresses and port numbers. Then, the Write and Read functions of RDMA communication can be used to send and receive data. During data transmission, the Go-IPoIB library automatically encapsulates and deencapsulates IPoIB packets and uses the RDMA-CM API to manage RDMA communication.

The implementation of the Go-IPoIB library involves multiple aspects such as network protocols, hardware devices, system calls, etc., which are relatively complex. Using the Go-IPoIB library requires a certain computer network and system programming foundation, and you need to understand in detail the usage methods of RDMA-CM API and related network protocol specifications.

Here is a simple example program that implements a basic RDMA write operation:

package main
import (
	"fmt"
	"log"
	"net"
	"time"
	"/infiniband/go-ipoib"
)
func main() {
	// Create an RDMA connection	conn, err := ("192.168.1.1", "ib0", "192.168.1.2", "ib0", 1024)
	if err != nil {
		(err)
	}
	defer ()
	// Send data	data := []byte("hello world")
	err = (data)
	if err != nil {
		(err)
	}
	// Receive data	buf := make([]byte, len(data))
	err = (buf)
	if err != nil {
		(err)
	}
	(string(buf))
}

In this example, we use the function to create an RDMA connection, specify the local and remote IP addresses and port numbers, and set the buffer size to 1024 bytes. We then use the function to send a message to the other party and receive a reply message from the other party using the function. Finally, we convert the received data into a string and print it out.

In practical applications, more complex RDMA communication implementations need to be carried out according to specific needs, such as RDMA read operations, management of multiple connections, etc. At the same time, it is also necessary to consider the security and stability of RDMA communication to avoid data loss and deadlock.

4. What is NVME-oF?

NVMe-oF (Non-Volatile Memory Express over Fabrics) is a technology that extends the NVMe protocol to network storage. It allows remote computers to access local storage devices over the network, enabling high-performance, low-latency, and high-availability storage access.

The implementation of NVMe-oF technology requires hardware and software support, including network adapters, storage controllers, drivers and protocol stacks. In the NVMe-oF architecture, storage devices are divided into two parts: NVMe-oF Target and NVMe-oF Initiator.

NVMe-oF Target is the server of the storage device, which is responsible for responding to requests from the NVMe-oF Initiator and transferring data to the local storage device. The NVMe-oF Initiator is the client of the storage device, which is responsible for sending requests to the NVMe-oF Target and reading data from the local storage device.

In NVMe-oF technology, data transmission is achieved through RDMA (Remote Direct Memory Access) technology, which can achieve high bandwidth and low latency data transmission. At the same time, NVMe-oF technology also supports a variety of transmission protocols, including RoCE (RDMA over Converged Ethernet), iWARP (Internet Wide Area RDMA Protocol), etc.

In general, NVMe-oF technology is a technology that extends the NVMe protocol to network storage, which can achieve high-performance, low-latency, and high-availability storage access. If you need more detailed information, you can view relevant documents or refer to other relevant materials. Thanks!

To implement NVMe-oF in Go, you can use the GoNVMe library. The GoNVMe library is an NVMe command line tool and library implemented based on the Go language, which can be used to manage and test NVMe devices and drivers.

When using the GoNVMe library, you need to first install the NVMe device and drivers, and install the GoNVMe library. Then, you can use the GoNVMe command line tool to perform various NVMe operations, such as listing NVMe devices, reading properties of NVMe devices, executing NVMe commands, etc.

Here is an example program that uses the GoNVMe library to read NVMe device properties:

package main
import (
	"fmt"
	"/chenjie199234/GoNVMe/nvme"
)
func main() {
	// Turn on NVMe device	dev, err := ("/dev/nvme0")
	if err != nil {
		panic(err)
	}
	defer ()
	// Read NVMe device properties	id, err := ()
	if err != nil {
		panic(err)
	}
	// Output NVMe device properties	("%+v\\n", id)
}

In this example, we open an NVMe device using a function and read the properties of the NVMe device using the function. Then, we convert the read attribute information into a string and print it out.

Using the GoNVMe library requires a certain foundation in computer storage and system programming, and you need to understand the NVMe protocol and related storage technologies. At the same time, NVMe devices and driver support is also a prerequisite for using the GoNVMe library.

The above is the detailed explanation of the detailed content based on the go instance network storage protocol. For more information about the go network storage protocol, please pay attention to my other related articles!