需要對一個上百萬行的json文件進行清洗，并需要將清洗好的結(jié)構(gòu)化數(shù)據(jù)重新存儲為csv文件。嘗試使用pandas的dataframe轉(zhuǎn)存清洗好的數(shù)據(jù)條目，卻發(fā)現(xiàn)常規(guī)的清洗一條寫入一條的速度太慢了，速度主要卡在每次數(shù)據(jù)的寫入，于是為此專門定義了一個 asyn writeline，同時利用 async readline 每次生成 100 個協(xié)程處理100行數(shù)據(jù)，然而測試結(jié)果卻和常規(guī)的按序處理無差別，平均每條數(shù)據(jù)處理速度都在 0.5 s 左右，感覺是自己的 asyn writeline 有問題，還請大神賜教。

測試代碼如下：

import pandas as pd
import json
import time
import asyncio

def trop():
    tropicos = pd.DataFrame()
    with open(r"/tropicosbase.json", "r") as yn:
        count = 0
        tropicos['tag'] = None
        tropicos.loc[0] = None
        async def readline(line):
            nonlocal count
            js = json.loads(line)
            await writeline(js, tropicos, count)
            count += 1
            tropicos.loc[count] = None
        cs = yn.readlines()[:100]
        tasks = [asyncio.ensure_future(readline(line)) for line in cs]
        loop = asyncio.get_event_loop()
        start = time.time()
        loop.run_until_complete(asyncio.wait(tasks))
        end = time.time()
        print(end - start)

    tropicos.to_csv(r'/tropicos.csv', index=None)


async def writeline(js, tropicos, count):
    for k, v in js.items():
        try:
            tropicos[k][count] = v
        except KeyError:
            if k == 'detailsdiv':
                pass
            else:
                tropicos[k] = pd.Series()
                tropicos[k][count] = v

trop()