Detailed explanation of the example of implementing in-memory database based on Golang

GO implements memory database

Implement Redis's database layer (core layer: process commands and return)

/csgopher/go-redis

This article involves the following documents: dict: some methods to define dictionary

sync_dict: implement dict
db: database
command: define command
ping, keys, string: specific processing logic of instructions
database: standalone version of database

datastruct/dict/

type Consumer func(key string, val interface{}) bool

type Dict interface {
   Get(key string) (val interface{}, exists bool)
   Len() int
   Put(key string, val interface{}) (result int)
   PutIfAbsent(key string, val interface{}) (result int)
   PutIfExists(key string, val interface{}) (result int)
   Remove(key string) (result int)
   ForEach(consumer Consumer)
   Keys() []string
   RandomKeys(limit int) []string
   RandomDistinctKeys(limit int) []string
   Clear()
}

Dict interface: The interface of the Redis data structure. Here we use it as a dictionary implementation. If you want to use another data structure, you can change it to another implementation.

Consumer: Iterate through all key-value pairs in the dictionary, the return value is Boolean, true continues to traverse, false stops traversal

datastruct/dict/sync_dict.go

type SyncDict struct {
   m 
}

func MakeSyncDict() *SyncDict {
   return &SyncDict{}
}

func (dict *SyncDict) Get(key string) (val interface{}, exists bool) {
   val, ok := (key)
   return val, ok
}

func (dict *SyncDict) Len() int {
   length := 0
   (func(k, v interface{}) bool {
      length++
      return true
   })
   return length
}

func (dict *SyncDict) Put(key string, val interface{}) (result int) {
   _, existed := (key)
   (key, val)
   if existed {
      return 0
   }
   return 1
}

func (dict *SyncDict) PutIfAbsent(key string, val interface{}) (result int) {
   _, existed := (key)
   if existed {
      return 0
   }
   (key, val)
   return 1
}

func (dict *SyncDict) PutIfExists(key string, val interface{}) (result int) {
   _, existed := (key)
   if existed {
      (key, val)
      return 1
   }
   return 0
}

func (dict *SyncDict) Remove(key string) (result int) {
   _, existed := (key)
   (key)
   if existed {
      return 1
   }
   return 0
}

func (dict *SyncDict) ForEach(consumer Consumer) {
   (func(key, value interface{}) bool {
      consumer(key.(string), value)
      return true
   })
}

func (dict *SyncDict) Keys() []string {
   result := make([]string, ())
   i := 0
   (func(key, value interface{}) bool {
      result[i] = key.(string)
      i++
      return true
   })
   return result
}

func (dict *SyncDict) RandomKeys(limit int) []string {
   result := make([]string, limit)
   for i := 0; i < limit; i++ {
      (func(key, value interface{}) bool {
         result[i] = key.(string)
         return false
      })
   }
   return result
}

func (dict *SyncDict) RandomDistinctKeys(limit int) []string {
   result := make([]string, limit)
   i := 0
   (func(key, value interface{}) bool {
      result[i] = key.(string)
      i++
      if i == limit {
         return false
      }
      return true
   })
   return result
}

func (dict *SyncDict) Clear() {
   *dict = *MakeSyncDict()
}

Implementing the Dict interface using

database/

type DB struct {
	index int
	data  
}

type ExecFunc func(db *DB, args [][]byte) 

type CmdLine = [][]byte

func makeDB() *DB {
	db := &amp;DB{
		data: (),
	}
	return db
}

func (db *DB) Exec(c , cmdLine [][]byte)  {
	cmdName := (string(cmdLine[0]))
	cmd, ok := cmdTable[cmdName]
	if !ok {
		return ("ERR unknown command '" + cmdName + "'")
	}
	if !validateArity(, cmdLine) {
		return (cmdName)
	}
	fun := 
	return fun(db, cmdLine[1:]) // Cut off the set in set k v}

func validateArity(arity int, cmdArgs [][]byte) bool {
	argNum := len(cmdArgs)
	if arity &gt;= 0 {
		return argNum == arity
	}
	return argNum &gt;= -arity
}

func (db *DB) GetEntity(key string) (*, bool) {
	raw, ok := (key)
	if !ok {
		return nil, false
	}
	entity, _ := raw.(*)
	return entity, true
}

func (db *DB) PutEntity(key string, entity *) int {
	return (key, entity)
}

func (db *DB) PutIfExists(key string, entity *) int {
	return (key, entity)
}

func (db *DB) PutIfAbsent(key string, entity *) int {
	return (key, entity)
}

func (db *DB) Remove(key string) {
	(key)
}

func (db *DB) Removes(keys ...string) (deleted int) {
	deleted = 0
	for _, key := range keys {
		_, exists := (key)
		if exists {
			(key)
			deleted++
		}
	}
	return deleted
}

func (db *DB) Flush() {
	()
}

Implementing the database in Redis

ExecFunc: All Redis instructions are written in this type

validateArity method:

Fixed length：set k v => arity=3；
Get longer: exists k1 k2 k3 ... => arity=-2, indicating parameters >=2

database/

var cmdTable = make(map[string]*command)

type command struct {
   executor ExecFunc
   arity    int 
}

func RegisterCommand(name string, executor ExecFunc, arity int) {
   name = (name)
   cmdTable[name] = &command{
      executor: executor,
      arity:    arity,
   }
}

command: Each command structure is a command, such as ping, keys, etc.

arity: number of parameters
cmdTable: record the relationship between all commands and command structure
RegisterCommand: The implementation of the registration command, in the program

database/

func Ping(db *DB, args [][]byte)  {
    if len(args) == 0 {
        return &{}
    } else if len(args) == 1 {
        return (string(args[0]))
    } else {
        return ("ERR wrong number of arguments for 'ping' command")
    }
}

func init() {
    RegisterCommand("ping", Ping, 1)
}

init method: This method will be called when starting the program, used for initialization

database/

func execDel(db *DB, args [][]byte)  {
   keys := make([]string, len(args))
   for i, v := range args {
      keys[i] = string(v)
   }

   deleted := (keys...)
   return (int64(deleted))
}

func execExists(db *DB, args [][]byte)  {
   result := int64(0)
   for _, arg := range args {
      key := string(arg)
      _, exists := (key)
      if exists {
         result++
      }
   }
   return (result)
}

func execFlushDB(db *DB, args [][]byte)  {
   ()
   return &{}
}

func execType(db *DB, args [][]byte)  {
   key := string(args[0])
   entity, exists := (key)
   if !exists {
      return ("none")
   }
   switch .(type) {
   case []byte:
      return ("string")
   }
   return &{}
}

func execRename(db *DB, args [][]byte)  {
   if len(args) != 2 {
      return ("ERR wrong number of arguments for 'rename' command")
   }
   src := string(args[0])
   dest := string(args[1])
   
   entity, ok := (src)
   if !ok {
      return ("no such key")
   }
   (dest, entity)
   (src)
   return &{}
}

func execRenameNx(db *DB, args [][]byte)  {
   src := string(args[0])
   dest := string(args[1])

   _, exist := (dest)
   if exist {
      return (0)
   }

   entity, ok := (src)
   if !ok {
      return ("no such key")
   }
   (src, dest)
   (dest, entity)
   return (1)
}

func execKeys(db *DB, args [][]byte)  {
   pattern := (string(args[0]))
   result := make([][]byte, 0)
   (func(key string, val interface{}) bool {
      if (key) {
         result = append(result, []byte(key))
      }
      return true
   })
   return (result)
}

func init() {
   RegisterCommand("Del", execDel, -2)
   RegisterCommand("Exists", execExists, -2)
   RegisterCommand("Keys", execKeys, 2)
   RegisterCommand("FlushDB", execFlushDB, -1)
   RegisterCommand("Type", execType, 2)
   RegisterCommand("Rename", execRename, 3)
   RegisterCommand("RenameNx", execRenameNx, 3)
}

Implement the following instructions:

execDel：del k1 k2 k3 ...
execExists：exist k1 k2 k3 ...
execFlushDB：flushdb
execType：type k1
execRename：rename k1 k2
execRenameNx：renamenx k1 k2
execKeys: keys (a tool class that depends on lib package)

database/

func execGet(db *DB, args [][]byte)  {
   key := string(args[0])
   bytes, err := (key)
   if err != nil {
      return err
   }
   if bytes == nil {
      return &{}
   }
   return (bytes)
}

func (db *DB) getAsString(key string) ([]byte, ) {
   entity, ok := (key)
   if !ok {
      return nil, nil
   }
   bytes, ok := .([]byte)
   if !ok {
      return nil, &{}
   }
   return bytes, nil
}

func execSet(db *DB, args [][]byte)  {
   key := string(args[0])
   value := args[1]
   entity := &{
      Data: value,
   }
   (key, entity)
   return &{}
}

func execSetNX(db *DB, args [][]byte)  {
   key := string(args[0])
   value := args[1]
   entity := &{
      Data: value,
   }
   result := (key, entity)
   return (int64(result))
}

func execGetSet(db *DB, args [][]byte)  {
   key := string(args[0])
   value := args[1]

   entity, exists := (key)
   (key, &{Data: value})
   if !exists {
      return ()
   }
   old := .([]byte)
   return (old)
}

func execStrLen(db *DB, args [][]byte)  {
   key := string(args[0])
   entity, exists := (key)
   if !exists {
      return ()
   }
   old := .([]byte)
   return (int64(len(old)))
}

func init() {
   RegisterCommand("Get", execGet, 2)
   RegisterCommand("Set", execSet, -3)
   RegisterCommand("SetNx", execSetNX, 3)
   RegisterCommand("GetSet", execGetSet, 3)
   RegisterCommand("StrLen", execStrLen, 2)
}

Implement the following instructions:

execGet：get k1
execSet：set k v
execSetNX：setnex k v
execGetSet: getset k v returns the old value
execStrLen：strlen k

database/

type Database struct {
   dbSet []*DB
}

func NewDatabase() *Database {
   mdb := &Database{}
   if  == 0 {
       = 16
   }
    = make([]*DB, )
   for i := range  {
      singleDB := makeDB()
       = i
      [i] = singleDB
   }
   return mdb
}

func (mdb *Database) Exec(c , cmdLine [][]byte) (result ) {
   defer func() {
      if err := recover(); err != nil {
         (("error occurs: %v\n%s", err, string(())))
      }
   }()

   cmdName := (string(cmdLine[0]))
   if cmdName == "select" {
      if len(cmdLine) != 2 {
         return ("select")
      }
      return execSelect(c, mdb, cmdLine[1:])
   }
   dbIndex := ()
   selectedDB := [dbIndex]
   return (c, cmdLine)
}

func execSelect(c , mdb *Database, args [][]byte)  {
   dbIndex, err := (string(args[0]))
   if err != nil {
      return ("ERR invalid DB index")
   }
   if dbIndex >= len() {
      return ("ERR DB index is out of range")
   }
   (dbIndex)
   return ()
}

func (mdb *Database) Close() {
}

func (mdb *Database) AfterClientClose(c ) {
}

Database: a collection of dbs
Exec: Execute the switch db command or other commands
execSelect method: select db (instruction: select 2)

resp/handler/

import (
	database2 "go-redis/database"
)

func MakeHandler() *RespHandler {
   var db 
   db = ()
   return &RespHandler{
      db: db,
   }
}

Modify the database implementation of the protocol layer handler

Architecture summary

The TCP layer serves the connection of TCP, and then hands the connection to the handler of the RESP protocol layer. The handler listens to the client's connection, parses the instructions and sends them to the pipeline, and transfers the pipeline to the database layer (database/). The core layer executes different methods according to the command type, and then returns.

This is the end of this article about the detailed explanation of the examples of implementing in-memory databases based on Golang. For more related content of Golang in-memory database, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!