Detailed explanation of how to use Python multi-process locks

Many times, we need to write a file in multiple processes at the same time. If we do not add the lock mechanism, it will cause the file writing errors.

At this time, we can use ()

I used it like this at the beginning:

import multiprocessing
lock = ()
class MatchProcess():
  def __init__(self, threadId, mfile, lock):
    .__init__(self)
     = threadId
     = mfile
     = lock
  def run(self):
    while True:
       ()
       try:
         ('111111111111111111' + '\n')
       finally:
         ()
 
if __name__ == '__main__':
  mf = open('', 'w')
  for i in range(15):
    p = MatchProcess(i, mf, lock)
    () 

Found that this way, the lock did not work, and the file content was still incorrect (note that the 1111 I wrote here is an example, and other contents actually written by my code)

So in this way, although the lock is passed to each process through parameters, we know that there is no shared memory between processes, so I understand that the locks obtained by each process are actually different, so it cannot lock the file.

Is the process pool feasible? So I made the following attempts

def run(line):
  ()
    try:
      ('111111111111111111' + '\n')
  finally:
    ()
sf = open('', 'r')
data_lst = list()
for line in sf: 
  line = ()
  data_lst.append(line)
pool = Pool(15)
pool.map_async(run, data_lst) The #map_async method will pass each element in the iterable object data_lst into the run method to execute iterate.()
()
print 'over'

But note:

()
()

These two lines of code are essential, otherwise, the main process will exit after execution, causing the entire process to end

Therefore, after the entire process is executed, over will be printed

But this way, it is found that the lock still does not work

Finally, the following method was adopted:

def run(line):
  mfile = open('', 'a')
  ()
  try:
    ('111111111111111111' + '\n')
  finally:
    ()
 
sf = open('', 'r')
data_lst = list()
for line in sf: 
  line = ()
  data_lst.append(line)
 
pList = []
for line in line_lst:
  p = (target=run, args=(line, lock))
  ()
  (p)
 
for p in pList:
  ()

It was found that this method did work. The speed was very slow when the amount of data was written every time.

But a rather disgusting problem is that I initially tried to pass the args parameter of the Process object into the run method after opening the file, but found that the data could not be written into the file. Hell, I haven't figured out this problem yet

I can't help but take the stupid method above. Open and write every time I write. This is definitely not a wise approach. If there is a better way, please leave me a message.

In other words, passing in the file after opening is invalid, so you can pass the file name in, and then open it first every time you write in the run method, and closing it after writing should also be feasible.

But why is the first method my article used, which is also passed in after the file is opened, is feasible.

The above detailed explanation of how to use Python multi-process locks is all the content I have shared with you. I hope you can give you a reference and I hope you can support me more.