I won't say much nonsense, let's just read the code~
func ReadLine(fileName string) ([]string,error){
f, err := (fileName)
if err != nil {
return nil,err
}
buf := (f)
var result []string
for {
line, err := ('\n')
line = (line)
if err != nil {
if err == { //EOF will be reported after reading return result,nil
}
return nil,err
}
result = append(result,line)
}
return result,nil
}
Supplement: Golang's solution to read files and handle super large files
There are many ways to read files in Golang, and the applicable scenarios are also different. Here we divide the reading of files into the following types:
Overall file reading
File shard reading (block-level reading)
File line-level reading
The system configuration is different, and the execution time is also different. Here is a reference
System configuration:
OS : Windows10
Memory : 16G
CPU (Intel) Intel® Core™ i3-4370 CPU @ 3.80GHz (3800 MHz)
1. Overall file reading
The overall reading of the file is to read the file at one time. Understand that the content of the file is read for the first time.
Use scenarios:
It is more suitable for small files (large files also consume a lot of time to read space and time)
It is also suitable for files with strong integrity (the files should not be too large)
Code Example 1
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"time"
)
// Test text file size 11Mvar m11 string = `G:\runtime\log\ccapi\`
// The text file for testing is 400M in sizevar m400 string = `G:\runtime\log\ccapi\`
// Read the entire filefunc readAll(filePath string) {
start1 := ()
(filePath)
("readAll spend : ", ().Sub(start1))
}
func main() {
readAll(m11)
readAll(m400)
}
$ go run readAll spend : 6.9999ms readAll spend : 358.8014ms
Code Example 2
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"time"
)
// Test text file size 11Mvar m11 string = `G:\runtime\log\ccapi\`
// The text file for testing is 400M in sizevar m400 string = `G:\runtime\log\ccapi\`
// Read the file intactfunc readAllBuff(filePath string) {
start1 := ()
// Open the file FileHandle, err := (filePath)
if err != nil {
(err)
return
}
// Close the file defer ()
// Get the current file information fileInfo, err := ()
if err != nil {
(err)
return
}
buffer := make([]byte, ())
// Read the file content and write it to the buffer n, err := (buffer)
if err != nil {
(err)
}
// Print all slices (string(buffer[:n]))
("readAllBuff spend : ", ().Sub(start1))
}
func main() {
readAllBuff(m11)
readAllBuff(m400)
}
2. File fragmentation reading
Read part of the file gradually until the file is fully read
PS: The size of the file is read according to the set shard size, so when reading text type files (for example: log files)
It is not necessarily output line by line as you expect, because the line breaks at the end of the text will not be processed, but the content will be read according to the size of the shard.
Use scenarios:
Reading large files is very suitable
It is very suitable to read binary type files (such as audio and video files or resource type files, etc.)
Code Example
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"time"
)
// Test text file size 11Mvar m11 string = `G:\runtime\log\ccapi\`
// The text file for testing is 400M in sizevar m400 string = `G:\runtime\log\ccapi\`
// Read files one by onefunc readBlock(filePath string) {
start1 := ()
FileHandle, err := (filePath)
if err != nil {
(err)
return
}
defer ()
// Set the number of bytes read per time buffer := make([]byte, 1024)
for {
n, err := (buffer)
// Control conditions, adjust according to actual conditions if err != nil && err != {
(err)
}
if n == 0 {
break
}
// The following code prints out the file blocks (bytes) read each time //(string(buffer[:n]))
}
("readBolck spend : ", ().Sub(start1))
}
func main() {
readBlock(m11)
readBlock(m400)
}
$ go run readBolck spend : 31.9814ms readBolck spend : 1.0889488s
3. Read the file line by line
Read the file line by line until it reaches the end of the file
Use scenarios:
It is very suitable to read super large files (for example: super large log files, etc.)
The files you read should be with a line break (if you use a large file composed of single-line files, you need to pay attention)
For large files that need to be analyzed
Count the number of occurrences of certain data
Query if some data exists
Find data for a specified row
Example Code 1
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"time"
)
// Test text file size 11Mvar m11 string = `G:\runtime\log\ccapi\`
// The text file for testing is 400M in sizevar m400 string = `G:\runtime\log\ccapi\`
// Read every line of the filefunc readEachLineReader(filePath string) {
start1 := ()
FileHandle, err := (filePath)
if err != nil {
(err)
return
}
defer ()
lineReader := (FileHandle)
for {
// Methods that can be used in the same usage scenario // func (b *Reader) ReadLine() (line []byte, isPrefix bool, err error)
// func (b *Reader) ReadBytes(delim byte) (line []byte, err error)
// func (b *Reader) ReadString(delim byte) (line string, err error)
line, _, err := ()
if err == {
break
}
//The following are some business logic operations // The following code prints the file line contents read each time (string(line))
}
("readEachLineReader spend : ", ().Sub(start1))
}
func main(){
readEachLineReader(m11)
readEachLineReader(m400)
}
$ go run readEachLineReader spend : 16.9902ms readEachLineReader spend : 537.9683ms
Code Example 2
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"time"
)
// Test text file size 11Mvar m11 string = `G:\runtime\log\ccapi\`
// The text file for testing is 400M in sizevar m400 string = `G:\runtime\log\ccapi\`
// Read every line of the filefunc readEachLineScanner(filePath string) {
start1 := ()
FileHandle, err := (filePath)
if err != nil {
(err)
return
}
defer ()
lineScanner := (FileHandle)
for () {
// In the same usage scenario, the following methods can be used // func (s *Scanner) Bytes() []byte
// func (s *Scanner) Text() string
// Actual logic: perform certain business operations on the read content // The following code prints the file line contents read each time (())
}
("readEachLineScanner spend : ", ().Sub(start1))
}
func main() {
readEachLineScanner(m11)
readEachLineScanner(m400)
}
$ go run readEachLineScanner spend : 17.9895ms readEachLineScanner spend : 574.1722ms
4. Summary
Common problems similar to reading large files in interviews, we usually use a fragmented reading or line-by-line reading scheme.
A similar solution can be used to upload large files. Each time some content of the file is read is uploaded (write) into the network interface until the file is read.
If you have ordinary small files and do not have much operation on the content, you can use overall reading and the speed is relatively fast.
If you have operations on file content, use fragmented reading and line-by-line reading.
The binary type file is more suitable to use the fragmented or overall reading scheme.
File reading is not only a local file, but also a file that needs to be accessed when reading files on the network (various documents, audio and video, pictures, and other types of files). Or you can use three ways to read the file contents to
func ReadAll(r ) ([]byte, error) complete read file
func Copy(dst Writer, src Reader) (writeten int64, err error) File read and write
type Reader interface {
Read(p []byte) (n int, err error)
}
Read through the Read method of the Reader interface
The above is personal experience. I hope you can give you a reference and I hope you can support me more. If there are any mistakes or no complete considerations, I would like to give you advice.