Detailed explanation of how to implement ssh&scp in Go language

Preface

Recently, I encountered a temporary requirement that a service's daily log in a customer environment needs to be processed in a series of complex processes and generate data reports. Because the data processing logic is complex and needs to be stored in the database, it is used in the customer environment.shellThe script cannot be processed, so the log needs to be copied to the local area first and then processed; at the same time, in order to avoid manual copying of logs every day, automation is needed, and the entire link is automatically executed without manual intervention. Usually usedGoThere are many languages, which leads toGolanguagesshConnect to remote client server and usescpA series of operations to copy the data.

Note: The examples in this article are all based on Go1.17 64-bit machines

Connect to the remote server and execute commands (ssh)

Use as given belowUsername + PasswordConnect to the remote server and execute it/usr/bin/whoamiExample of the command, the steps are as follows:

GenerateClientConfig: If you want to connect to a remote server, you must specify at least one implementation.AuthofAuthMethod, We use passwords here; at the same time, we need to provide a method for secure verification of remote server keys.HostKeyCallback, we are using the method of non-checking here.(), it is recommended to use it in production(key PublicKey)；
CallDial ：DialMethod establishes a connection with the remote server and returns aclient ；
NewSession：NewSessionMethod to open a session, which can be passed in one sessionRunMethod executes a command.

import (
 "bytes"
 "fmt"
 "log"
  
  "/x/crypto/ssh"
)

func main() {

 var (
  username = "your username"
  password = "your password"
  addr     = "ip:22"
 )
 
 config := &amp;{
  User: username,
  Auth: []{
   (password),
  },
  HostKeyCallback: (),
 }
 client, err := ("tcp", addr, config)
 if err != nil {
  ("Failed to dial: ", err)
 }
 defer ()

 // Open a session to execute a command session, err := ()
 if err != nil {
  ("Failed to create session: ", err)
 }
 defer ()

 // Execute the command and write the execution result into b var b 
  = &amp;b
  
  // You can also use() to integrate the output if err := ("/usr/bin/whoami"); err != nil {
  ("Failed to run: " + ())
 }
 (())  // root
}

In the above example, we areRunA command is passed into the method, and the remote server will return the execution result to us. If it is a complex operation, it will be more troublesome to pass in the command. For example, the requirements mentioned above require me tok8sCopy the daily log of the service in the container. The steps after disassembly are: obtain multiple services.k8s podName, copy the log file from multiple containers according to the current date, and then integrate it into one log file. For complex operations, we can write a script on the remote server, and thenRunPass the command to execute the script in the method.

To give a brief example, we wrote a script on the remote server., put it/optIn the directory, the script content and call methods are as follows:

# Script File#!/bin/bash
today=$(date +"%Y-%m-%d")
# Write data to file$(df -h &gt; $)

package main

import (
 "fmt"
 "log"
  
  "/x/crypto/ssh"
)

func main() {

 var (
  username = "your username"
  password = "your password"
  addr     = "ip:22"
 )

 config := &amp;{
  User: username,
  Auth: []{
   (password),
  },
  HostKeyCallback: (),
 }
 client, err := ("tcp", addr, config)
 if err != nil {
  ("Failed to dial: ", err)
 }
 defer ()


 session, err := ()
 if err != nil {
  ("Failed to create session: ", err)
 }
 defer ()

  // Call remote server script script res, err := ("sh /opt/")
 if err != nil {
  ("Failed to run: " + ())
 }
 (string(res))
  
  /*
  Filesystem      Size  Used Avail Use% Mounted on
  devtmpfs        909M     0  909M   0% /dev
  tmpfs           919M   24K  919M   1% /dev/shm
  tmpfs           919M  540K  919M   1% /run
  tmpfs           919M     0  919M   0% /sys/fs/cgroup
  /dev/vda1        50G  6.9G   40G  15% /
  tmpfs           184M     0  184M   0% /run/user/0
  */
}

Copy remote server files to local (scp)

The steps to copy files are relatively simple:

Establishssh client
based onssh clientCreatesftp client
Open the remote server file and copy it to the local area

package main

import (
 "io"
 "log"
 "os"
 "time"
  
  "/pkg/sftp"
 "/x/crypto/ssh"
)

func main() {

 var (
  username = "your username"
  password = "your password"
  addr     = "ip:22"
 )

 // 1. Create ssh client config := &amp;{
  User: username,
  Auth: []{
   (password),
  },
  HostKeyCallback: (),
 }
 client, err := ("tcp", addr, config)
 if err != nil {
  ("Failed to dial: ", err)
 }
 defer ()

 // 2. Create sftp client based on ssh client sftpClient, err := (client)
 if err != nil {
  ("Failed to init sftp client: ", err)
 }
 defer ()

 // 3. Open the remote server file filename := ().Format("2006-01-02") + ".log"
 source, err := ("/opt/" + filename)
 if err != nil {
  ("Failed to open remote file: ", err)
 }
 defer ()

 // 4. Create local files target, err := (filename, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644)
 if err != nil {
  ("Failed to open local file: ", err)
 }
 defer ()

 // 5. Data copy n, err := (target, source)
 if err != nil {
  ("Failed to copy file: ", err)
 }
 ("Succeed to copy file: ", n)

}

existsftp clientThere are many other methods, such asWalk、ReadDir、Stat、Mkdiretc. There are also documentsRead、Write、WriteTo、ReadFromSuch methods are as convenient as operating the local file system.

In simple package

package main

import (
 "fmt"
 "io"
 "log"
 "os"
 "time"

 "/pkg/sftp"
 "/x/crypto/ssh"
)

type Cli struct {
 user   string
 pwd    string
 addr   string
 client *
}


func NewCli(user, pwd, addr string) Cli {
 return Cli{
  user: user,
  pwd:  pwd,
  addr: addr,
 }
}

// Connect Connect Connect to the remote serverfunc (c *Cli) Connect() error {
 config := &amp;{
  User: ,
  Auth: []{
   (),
  },
  HostKeyCallback: (),
 }
 client, err := ("tcp", , config)
 if nil != err {
  return ("connect server error: %w", err)
 }
  = client
 return nil
}

// Run commandfunc (c Cli) Run(shell string) (string, error) {
 if  == nil {
  if err := (); err != nil {
   return "", err
  }
 }

 session, err := ()
 if err != nil {
  return "", ("create new session error: %w", err)
 }
 defer ()

 buf, err := (shell)
 return string(buf), err
}

// Scp Copy filefunc (c Cli) Scp(srcFileName, targetFileName string) (int64, error) {
 if  == nil {
  if err := (); err != nil {
   return 0, err
  }
 }

 sftpClient, err := ()
 if err != nil {
  return 0, ("new sftp client error: %w", err)
 }
 defer ()

 source, err := (srcFileName)
 if err != nil {
  return 0, ("sftp client open file error: %w", err)
 }
 defer ()

 target, err := (targetFileName, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644)
 if err != nil {
  return 0, ("open local file error: %w", err)
 }
 defer ()

 n, err := (target, source)
 if err != nil {
  return 0, ("copy file error: %w", err)
 }
 return n, nil
}


// Call testfunc main() {
 var (
  username = "your username"
  password = "your password"
  addr     = "ip:22"
 )

 // Initialization client := NewCli(username, password, addr)

 // ssh and run the script _, err := ("sh /opt/")
 if err != nil {
  ("failed to run shell,err=[%v]\n", err)
  return
 }

 // scp file to local filename := ().Format("2006-01-02") + ".log"
 n, err := ("/opt/"+filename, filename)
 if err != nil {
  ("failed to scp file,err=[%v]\n", err)
  return
 }
 ("Succeed to scp file, size=[%d]\n", n)

 // Process the file and delete the local file...}

Through the above series of operations, my needs can be achieved:

1. Write the program:

Connect to the client server
Execute scripts of remote servers to generate log files
Copy the log files of the remote server to the local
Process log files
Delete local files

2. Start a timed task on the server to run the program

The above is a detailed explanation of the method of implementing ssh&scp in Go language. For more information about implementing ssh scp in Go language, please pay attention to my other related articles!