快捷導(dǎo)航

python實(shí)現(xiàn)數(shù)據(jù)清洗(缺失值與異常值處理)

更新時(shí)間：2019年12月02日 09:21:59 作者：諾坎普奇跡

今天小編就為大家分享一篇python實(shí)現(xiàn)數(shù)據(jù)清洗(缺失值與異常值處理)，具有很好的參考價(jià)值，希望對(duì)大家有所幫助。一起跟隨小編過來看看吧

1。將本地sql文件寫入mysql數(shù)據(jù)庫

本文寫入的是python數(shù)據(jù)庫的taob表

source [本地文件]

其中總數(shù)據(jù)為9616行，列分別為title,link,price,comment

2。使用python鏈接并讀取數(shù)據(jù)

查看數(shù)據(jù)概括

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
            user='root',
            passwd='123456',
            db='python')#鏈接本地?cái)?shù)據(jù)庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數(shù)據(jù)
print(data.describe())

說明數(shù)據(jù)的導(dǎo)入是正確的,簡單的分析發(fā)現(xiàn)問題并不是這么簡單，因?yàn)閏omment均值562可能偏大，最大評(píng)論數(shù)454037也可能出現(xiàn)錯(cuò)誤，price價(jià)格為0也不太可能出現(xiàn)。

 price    comment
count 9616.00000  9616.000000
mean   64.49324   562.239601
std   176.10901  6078.909643
min    0.00000    0.000000
25%   20.00000   16.000000
50%   36.00000   58.000000
75%   66.00000   205.000000
max  7940.00000 454037.000000

3。缺失值處理

將價(jià)格為0的值設(shè)置為中位數(shù)36

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
            user='root',
            passwd='123456',
            db='python')#鏈接本地?cái)?shù)據(jù)庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數(shù)據(jù)

data['price'][data['price']==0]=None
x = 0
for i in data.columns:
  for j in range(len(data)):
    if (data[i].isnull()) [j]:
      data[i][j]='36'
      x+=1
print(x)
#44

結(jié)果顯示修改了44行的數(shù)據(jù)。

4。異常值處理

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
            user='root',
            passwd='123456',
            db='python')#鏈接本地?cái)?shù)據(jù)庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數(shù)據(jù)
#缺失值處理
data['price'][data['price']==0]=None
x = 0
for i in data.columns:
  for j in range(len(data)):
    if (data[i].isnull()) [j]:
      data[i][j]='36'
      x+=1
print(x)
#異常值處理
#繪制散點(diǎn)圖,價(jià)格為橫軸
data1 = data.T#轉(zhuǎn)置
price = data1.values[2]
comment = data1.values[3]
plt.plot(price,comment,'o')
plt.show()
#print(price)

結(jié)果如下圖，價(jià)格為0左右時(shí)comment很大可能為異常值，comments為0時(shí)，價(jià)格極大這個(gè)有可能的。

接下來處理評(píng)論數(shù)異常值，假設(shè)異常值分割線設(shè)置為20w，

#-*- coding:utf-8 -*-
#author：M10
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import mysql.connector
conn = mysql.connector.connect(host='localhost',
            user='root',
            passwd='123456',
            db='python')#鏈接本地?cái)?shù)據(jù)庫
sql = 'select * from taob'#sql語句
data = pd.read_sql(sql,conn)#獲取數(shù)據(jù)
#缺失值處理
data['price'][data['price']==0]=None
x = 0
for i in data.columns:
  for j in range(len(data)):
    if (data[i].isnull()) [j]:
      data[i][j]='36'
      x+=1
print(x)
#異常值處理
da = data.values#重新賦值data
#異常值處理，將commments大于200000的數(shù)據(jù)comments設(shè)置為58
cont_clou = len(da)#獲取行數(shù)
#遍歷數(shù)據(jù)進(jìn)行處理
for i in range(0,cont_clou):
  if(data.values[i][3]>200000):
    #print(data.values[i][3])
    da[i][3]='58'
    #print(da[i][3])

#繪制散點(diǎn)圖,價(jià)格為橫軸
data1 = da.T#轉(zhuǎn)置
price = data1[2]
comment = data1[3]
plt.plot(price,comment,'o')
plt.xlabel('price')
plt.ylabel('comments')
plt.show()

處理后的輸出結(jié)果為：