python讀取一個(gè)大于10G的txt文件的方法
前言
用python 讀取一個(gè)大于10G 的文件，自己電腦只有8G內(nèi)存，一運(yùn)行就報(bào)內(nèi)存溢出：MemoryError
python 如何用open函數(shù)讀取大文件呢？
讀取大文件
首先可以自己先制作一個(gè)大于10G的txt文件
a = '''
2021-02-02 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/
2021-02-02 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2021-02-02 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version ('HTTP')
2021-02-02 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2021-02-02 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0
2021-02-02 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876
2021-02-02 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2021-02-02 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2021-02-02 22:16:21,229 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2021-02-02 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2021-02-02 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2021-02-02 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2021-02-02 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2021-02-02 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2021-02-02 22:16:26,509 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/report_list/1/ HTTP/1.1" 200 14649
2021-02-02 22:16:51,496 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-02 22:16:51,721 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-02 22:16:59,707 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-03 22:16:59,909 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-03 22:17:01,306 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/edit_case/1/ HTTP/1.1" 200 36504
2021-02-03 22:17:06,265 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/add_project/ HTTP/1.1" 200 17737
2021-02-03 22:17:07,825 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/project_list/1/ HTTP/1.1" 200 29789
2021-02-03 22:17:13,116 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/add_config/ HTTP/1.1" 200 24816
2021-02-03 22:17:19,671 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/config_list/1/ HTTP/1.1" 200 19532
'''
while True:
with open("xxx.log", "a", encoding="utf-8") as fp:
fp.write(a)
循環(huán)寫入到 xxx.log 文件，運(yùn)行 3-5 分鐘，pycharm 打開查看文件大小大于 10G
于是我用open函數(shù) 直接讀取
f = open("xxx.log", 'r')
print(f.read())
f.close()
拋出內(nèi)存溢出異常:MemoryError
Traceback (most recent call last):
File "D:/2021kecheng06/demo/txt.py", line 35, in <module>
print(f.read())
MemoryError
運(yùn)行的時(shí)候可以看下自己電腦的內(nèi)存已經(jīng)占了100%， cpu高達(dá)91% ，不掛掉才怪了！
這種錯(cuò)誤的原因在于，read()方法執(zhí)行操作是一次性的都讀入內(nèi)存中，顯然文件大于內(nèi)存就會(huì)報(bào)錯(cuò)。
read() 的幾種方法
1.read() 方法可以帶參數(shù) n， n 是每次讀取的大小長度，也就是可以每次讀一部分，這樣就不會(huì)導(dǎo)致內(nèi)存溢出
f = open("xxx.log", 'r')
print(f.read(2048))
f.close()
運(yùn)行結(jié)果
2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/
2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version ('HTTP')
2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0
2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876
2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2019-10-24 22:16:21,229 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2019-10-24 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2019-10-24 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2019-10-24 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2019-10-24 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2019-10-24 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2019-10-24 22:16:26,509 [django.server:124] [basehttp:log_message] [I
這樣就只讀取了2048個(gè)字符，全部讀取的話，循環(huán)讀就行
f = open("xxx.log", 'r')
while True:
block = f.read(2048)
print(block)
if not block:
break
f.close()
2.readline()：每次讀取一行，這個(gè)方法也不會(huì)報(bào)錯(cuò)
f = open("xxx.log", 'r')
while True:
line = f.readline()
print(line, end="")
if not line:
break
f.close()
3.readlines()：讀取全部的行，生成一個(gè)list，通過list來對(duì)文件進(jìn)行處理，顯然這種方式依然會(huì)造成：MemoyError
真正 Pythonic 的方法
真正 Pythonci 的方法，使用 with 結(jié)構(gòu)打開文件，fp 是一個(gè)可迭代對(duì)象，可以用 for 遍歷讀取每行的文件內(nèi)容
with open("xxx.log", 'r') as fp:
for line in fp:
print(line, end="")
yield 生成器讀取大文件
前面一篇講yield 生成器的時(shí)候提到讀取大文件，函數(shù)返回一個(gè)可迭代對(duì)象，用next()方法讀取文件內(nèi)容
def read_file(fpath):
BLOCK_SIZE = 1024
with open(fpath, 'rb') as f:
while True:
block = f.read(BLOCK_SIZE)
if block:
yield block
else:
return
if __name__ == '__main__':
a = read_file("xxx.log")
print(a) # generator objec
print(next(a)) # bytes類型
print(next(a).decode("utf-8")) # str
運(yùn)行結(jié)果
<generator object read_file at 0x00000226B3005258>
b'\r\n2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/\r\n2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678\r\n2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version (\'HTTP\')\r\n2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -\r\n2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0\r\n2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876\r\n2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801\r\n2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638\r\n2019-10-24 22:16:21,229 [django.server:124] '
[basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2019-10-24 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2019-10-24 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2019-10-24 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2019-10-24 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2019-10-24 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2019-10-24 22:16:26,509 [django.server:124] [basehtt
到此這篇關(guān)于python讀取一個(gè)大于10G的txt文件的方法的文章就介紹到這了,更多相關(guān)python讀取大于10G文件內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！
相關(guān)文章
Python中flatten( )函數(shù)及函數(shù)用法詳解
flatten是numpy.ndarray.flatten的一個(gè)函數(shù)，即返回一個(gè)一維數(shù)組。這篇文章主要介紹了Python中flatten( )函數(shù),需要的朋友可以參考下2018-11-11
在matplotlib的圖中設(shè)置中文標(biāo)簽的方法
今天小編就為大家分享一篇在matplotlib的圖中設(shè)置中文標(biāo)簽的方法，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過來看看吧2018-12-12
學(xué)習(xí)python之編寫簡單乘法口訣表實(shí)現(xiàn)代碼
這篇文章主要介紹了學(xué)習(xí)python之編寫簡單乘法口訣表實(shí)現(xiàn)代碼,需要的朋友可以參考下2016-02-02
使用Python和Pillow實(shí)現(xiàn)圖片馬賽克功能
在這篇博客中,我們將探討如何使用Python創(chuàng)建一個(gè)簡單而有趣的桌面應(yīng)用程序,我們的目標(biāo)是構(gòu)建一個(gè)應(yīng)用,允許用戶選擇一張照片,然后在照片的右下角添加馬賽克效果,感興趣的小伙伴跟著小編一起來看看吧2024-08-08
Anaconda虛擬環(huán)境配置Python庫與Spyder編譯器的方法
在文章創(chuàng)建Anaconda虛擬Python環(huán)境的方法中,我們介紹了在Anaconda環(huán)境下,創(chuàng)建、使用與刪除Python虛擬環(huán)境的方法,而創(chuàng)建虛擬環(huán)境后,就需要在對(duì)應(yīng)的環(huán)境內(nèi)配置各類庫與軟件,本文就對(duì)這些操作加以介紹,感興趣的朋友感興趣的朋友一起看看吧2024-04-04
Django框架模板語言實(shí)例小結(jié)【變量，標(biāo)簽，過濾器，繼承，html轉(zhuǎn)義】
這篇文章主要介紹了Django框架模板語言,結(jié)合實(shí)例形式總結(jié)分析了Django框架中變量，標(biāo)簽，過濾器，繼承，html轉(zhuǎn)義等相關(guān)模板語言操作技巧,需要的朋友可以參考下2019-05-05
Python根據(jù)給定模型進(jìn)行特征權(quán)值計(jì)算
在機(jī)器學(xué)習(xí)中,特征權(quán)重的計(jì)算是理解模型如何做出預(yù)測(cè)的重要步驟,本文將詳細(xì)介紹如何使用Python根據(jù)給定模型計(jì)算特征權(quán)重,希望對(duì)大家有一定的幫助2024-11-11
這篇文章主要介紹了Python中的路徑問題，幫助大家更好的理解和學(xué)習(xí)python，感興趣的朋友可以了解下2020-09-09
這篇文章主要介紹了python中的默認(rèn)編碼使用方式,具有很好的參考價(jià)值,希望對(duì)大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-06-06
python編程scrapy簡單代碼實(shí)現(xiàn)搜狗圖片下載器
這篇文章主要為大家介紹了使用python scrapy簡單代碼實(shí)現(xiàn)搜狗圖片下載器示例詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助2021-11-11