Python线程和代码的一般问题

问题1 :您正在写入文件缓冲区-只有当缓冲区已满、文件句柄已关闭或您显式调用时,它才会刷新到实际文件 flush() self.text.flush() )

问题2

如果您只想控制到STDOUT的输出,只要它不干扰线程执行,您可以捕获想要打印的内容,并最终在互斥锁下打印出来(因此一次只有一个线程写入),甚至可以通过管道将其返回到主线程,并让它管理对STDOUT的访问。一个简单的互斥体示例是:

PRINT_MUTEX = threading.Lock()

def compare(self, hashed, path):  # never mind the inefficiency, we'll get to that later
    out = []  # hold our output buffer
    with open(os.getcwd() + "/database.csv", 'r') as f:
        for row in f:
            row = row.split(':')
            if row[1] == hashed:
                out.append("Suspicious File!")
                out.append("Suspecfs: " + row[2] + "File name : " + path)
    if out:
        with self.PRINT_MUTEX:  # use a lock to print out the results
            print("\n".join(out))

compare 结果一次一个,而不是散布结果。如果您想让主线程/进程控制STDOUT,尤其是因为您想将其转换为多处理代码,请选中 this answer .

:您的线程永远不会退出,因为它们卡在 while True Queue.task_done() 因为它是用来给其他“听众”发信号的(如果他们被 Queue.join()

你应该使用 threading.Event queue.Queue 只有您可以创建一个特殊属性来表示队列的结束,然后在没有更多要处理的内容时将其放置在队列中,然后让线程在遇到此特殊属性时退出其循环。让我们首先修复代码中的一个大疏忽-您根本没有存储对线程的引用,而是用最后一个线程覆盖它,因此您无法真正控制执行流-与其将最后一个线程引用存储在变量中,不如将所有引用存储在列表中。此外,如果要等待一切结束,请不要使用守护进程线程:

def __init__(self):
    self.text = open(os.getcwd()+"/FileScanLogs.txt", "a+")  # consider os.path.join()
    self.hashlist = queue.Queue()
    self.filelist = queue.Queue()
    self.hashers = []  # hold the md5hash thread references
    self.comparators = []  # hold the threader thread references
    self.top = '/home/'
    for _ in range(12):  # you might want to consider a ThreadPool instead
        t = threading.Thread(target=self.md5hash)
        t.start()
        self.hashers.append(t)
    for _ in range(4):
        t = threading.Thread(target=self.threader)
        t.start()
        self.comparators.append(t)
    main.body(self)

现在我们可以修改 main.body() 方法,以便将上述特殊值添加到队列的末尾,以便工作线程知道何时停止:

QUEUE_CLOSE = object()  # a 'special' object to denote end-of-data in our queues

def body(self):
    start = time.time()
    self.text.write("Time:  " + time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + "\n")
    for root, dirs, files in os.walk(self.top):
        for f in files:
            path = os.path.join(root, f)
            self.filelist.put(path)
    self.filelist.put(self.QUEUE_CLOSE)  # no more files, signal the end of the filelist
    for t in self.hashers:  # let's first wait for our hashing threads to exit
        t.join()
    # since we're not going to be receiving any new hashes, we can...
    self.hashlist.put(self.QUEUE_CLOSE)  # ... signal the end of the hashlist as well
    for t in self.comparators:  # let's wait for our comparator threads to exit
        t.join()
    self.text.write("Total: " + str(time.time() - start) + "\n")
    self.text.close()  # close the log file (this will also flush the previous content)
    print("Log file is created as " + os.getcwd() + "/FileScanLogs.txt")

因此,我们需要修改工作线程,使其在遇到队列末尾时退出:

def md5hash(self):
    while self.filelist:
        entry = self.filelist.get()
        if entry is self.QUEUE_CLOSE:  # end of queue encountered
            self.filelist.put(self.QUEUE_CLOSE)  # put it back for the other threads
            break  # break away the processing
        finalhash = whatever_is_your_hash_code(entry)
        lists = finalhash + ',' + entry
        self.hashlist.put(lists)

def threader(self):
    while True:
        item = self.hashlist.get()
        if item is self.QUEUE_CLOSE:  # end of queue encountered
            self.hashlist.put(self.QUEUE_CLOSE)  # put it back for the other threads
            break  # break away the queue
        hashes = item.split(',')[0]
        path = item.split(',')[1]
        self.compare(hashes, path)

main.compare() hash<=>whatever dict 然后在现场进行比较(即。 if hashed in your_map )相反。

this answer ,但这只是一个主要的PITA,大多数时候不值得经历这些麻烦)。