代码之家  ›  专栏  ›  技术社区  ›  shpokas

异步dns查找,但结果指向同一文件

  •  0
  • shpokas  · 技术社区  · 8 年前

    我必须做大量的dns naptr查找(每分钟数千次)。 我使用dnspython运行一个python脚本,读取一个文件并写回另一个文件。请求速率约为每秒300个请求。 我尝试在python中使用异步dns aiodns ,但数字是一样的。 可能我的剧本有缺陷。请看下面。这是Python3.4。

    但是如果结果必须返回到一个文件,那么甚至可以异步查找吗?

    import asyncio
    import aiodns
    
    ...
    
    loop = asyncio.get_event_loop()
    resolver = aiodns.DNSResolver(loop=loop)
    resolver.nameservers = ['x.y.w.z']
    
    ...
    
    @asyncio.coroutine
    def getsip(number):
    
        try:
            strQuery = str(dns.e164.from_e164("+" + number))
            answer = yield from resolver.query(strQuery, 'NAPTR')
    
            for rdata in answer:
                return rdata.regex
    
        except:
            return ""
    
    with open(filename, 'r') as fread, open(filenameOut, 'w') as fwrite:
        reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
        reader.fieldnames = fieldnamesIn
    
        writer = csv.DictWriter(fwrite, fieldnames = fieldnamesOut, delimiter='|')
    
        for row in reader:
            sys.stdout.write("Processing record number: %d \r" % (total) )
            sys.stdout.flush()
            total+=1
            answer = loop.run_until_complete(getsip(row['NUM']))
            if answer == "":
                missingAnswers+=1
    
            writer.writerow({'NUM': row['NUM'], 'SIP': answer})
    
    print("Records not found: " + str(missingAnswers) + " of total " +  str(total) + " records.")
    
    1 回复  |  直到 8 年前
        1
  •  1
  •   user4815162342    8 年前

    但是如果结果必须返回到一个文件,那么甚至可以异步查找吗?

    如果您不关心结果的顺序,那么实现异步查找很简单。例如,您可以使用 asyncio.as_completed 要计划所有协同程序并行运行,并在每次完成时收到通知,请执行以下操作:

    @asyncio.coroutine
    def process():
        with open(filename, 'r') as fread:
            reader = csv.DictReader(fread, delimiter='|', quoting=csv.QUOTE_NONE)
            reader.fieldnames = fieldnamesIn
            rows = list(reader)
    
        with open(filenameOut, 'w') as fwrite:
            writer = csv.DictWriter(fwrite, fieldnames=fieldnamesOut, delimiter='|')
            missingAnswers = 0
    
            loop = asyncio.get_event_loop()
            tasks = [loop.create_task(getsip(row['NUM'])) for row in rows]
            for done_coro in asyncio.as_completed(tasks):
                answer = yield from done_coro
                if answer == ""
                    missingAnswers += 1
                writer.writerow({'NUM': row['NUM'], 'SIP': answer})
    
        print("Records not found: %d of total %d records"
              % (missingAnswers, len(rows)))
    
    loop = asyncio.get_event_loop()
    loop.run_until_complete(process())