This article introduces Python3 concurrently write the principle of analysis of the file, the text of the sample code through the introduction of the very detailed, for everyone to learn or work with certain reference learning value, you can refer to the following friends
When using python2 for concurrent writes, I found that the file would be messed up, that is, other lines would be inserted in the middle of one line.
But when writing concurrently with python3, neither multiprocessing, nor multithreading, this problem does not occur, is it a feature of python3?
import time
import os
import multiprocessing
from import Pool as ThreadPool
def write(val, file):
w = open(file, "a")
for i in range(100):
("%s\n" % val)
(0.001)
def thread_write(file):
res, pools = [], ThreadPool(10)
for i in range(10):
val = str(i) * 1000
(pools.apply_async(func=write, args=(val, file, )))
while res:
for ret in res:
if ():
(ret)
(0.01)
def mutil_write(file):
pools = (processes=10)
res = []
for i in range(100):
(pools.apply_async(thread_write, args=(file, )))
while res:
for ret in res:
if ():
(ret)
(0.01)
if __name__ == '__main__':
file = "./write_test"
mutil_write(file)
with open(file) as fb:
lines = 0
line_len = []
for line in fb:
lines += 1
line = ()
line_len.append(len(line))
if len(line) != 1000:
raise(Exception("error line: %s, len: %d" % (line, len(line))))
print("lines:%d, max len:%d, min:%d, avg:%.2f" % (lines, max(line_len), min(line_len), sum(line_len)/len(line_len)))
(file)
The above code, multi-process concurrently write the end of the verification of the length of each line is the length of the set. With python3 run repeatedly, all through the test without exception.
$ python3 --version Python 3.7.4 $ python3 lines:10000, max len:1000, min:1000, avg:1000.00
If you use python2, you will get an exception:
$ python2 --version
Python 2.7.15
$ python2
Traceback (most recent call last):
File "", line 49, in <module>
raise(Exception("error line: %s, len: %d" % (line, len(line))))
Exception: error line: 333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333330000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, len: 1092
This is the whole content of this article.