标签：day   定义   ...   equal   with   category   agg   pass   元素   

数据清理

简书大神SeanCheney的译作，我作了些格式调整和文章目录结构的变化，更适合自己阅读，以后翻阅是更加方便自己查找吧

import pandas as pd
import numpy as np

设定最大列数和最大行数

pd.set_option(‘max_columns‘,5 , ‘max_rows‘, 5)

1 宽格式转长格式

state_fruit = pd.read_csv(‘data/state_fruit.csv‘, index_col=0)

state_fruit

1.1 stack

DataFrame.stack(level=-1, dropna=True)

• Stack the prescribed level(s) from columns to index.
• 将列索引转化为行索引
  

• Return a reshaped DataFrame or Series having a multi-level index with one or more new inner-most levels compared to the current DataFrame. The new inner-most levels are created by pivoting the columns of the current dataframe:

  • if the columns have a single level, the output is a Series;
  • if the columns have multiple levels, the new index level(s) is (are) taken from the prescribed level(s) and the output is a DataFrame.

1.1.1 stack将列索引转化为行索引

stack方法可以将所有列名，转变为垂直的一级行索引

state_fruit.stack()

Texas    Apple      12
         Orange     10
                  ... 
Florida  Orange     14
         Banana    190
Length: 9, dtype: int64

使用reset_index()，将结果变为DataFrame

state_fruit_tidy = state_fruit.stack().reset_index()
state_fruit_tidy

重命名列名

state_fruit_tidy.columns = [‘state‘, ‘fruit‘, ‘weight‘]
state_fruit_tidy

也可以使用rename_axis给不同的行索引层级命名

state_fruit.stack().rename_axis([‘state‘, ‘fruit‘])

state    fruit 
Texas    Apple      12
         Orange     10
                  ... 
Florida  Orange     14
         Banana    190
Length: 9, dtype: int64

再次使用reset_index方法

state_fruit.stack().rename_axis([‘state‘, ‘fruit‘])           .reset_index(name=‘weight‘)

1.1.2 同时stack多组变量

即stack后将所有的列名，按照规则分为两列，或者多列

movie = pd.read_csv(‘data/movie.csv‘)
actor = movie[[
    ‘movie_title‘, ‘actor_1_name‘, ‘actor_2_name‘, 
    ‘actor_3_name‘, ‘actor_1_facebook_likes‘, 
    ‘actor_2_facebook_likes‘, ‘actor_3_facebook_likes‘]]

创建一个自定义函数，用来改变列名。wide_to_long要求分组的变量要有相同的数值结尾

def change_col_name(col_name):
    col_name = col_name.replace(‘_name‘, ‘‘)
    if ‘facebook‘ in col_name:
        fb_idx = col_name.find(‘facebook‘)
        col_name = col_name[:5] + col_name[fb_idx - 1:] + col_name[5:fb_idx-1]
    return col_name

actor2 = actor.rename(columns=change_col_name)
actor2.iloc[:5,:5]

使用wide_to_long函数，同时stack两列actor和Facebook

stubs = [‘actor‘, ‘actor_facebook_likes‘]
actor2_tidy = pd.wide_to_long(actor2, 
    stubnames=stubs, 
    i=[‘movie_title‘], 
    j=‘actor_num‘, 
    sep=‘_‘)
actor2_tidy.head(10)

1.2 melt

pandas.melt(frame, id_vars=None, value_vars=None, var_name=None, value_name=‘value‘, col_level=None)

• This function is useful to massage a DataFrame into a format where one or more columns are identifier variables (id_vars), while all other columns, considered measured variables (value_vars), are “unpivoted” to the row axis, leaving just two non-identifier columns, ‘variable’ and ‘value’.
• 这个函数是用来转化dataframe的格式，新的格式将满足一列或者多列作为id列，一列作为测量指标列，最后一列作为前面几列对应的测量值列
• id_vars : tuple, list, or ndarray, optional
  • Column(s) to use as identifier variables.
• value_vars : tuple, list, or ndarray, optional
  • Column(s) to unpivot. If not specified, uses all columns that are not set as id_vars.
• var_name : scalar
  • Name to use for the ‘variable’ column. If None it uses frame.columns.name or ‘variable’.
• value_name : scalar, default ‘value’
  • Name to use for the ‘value’ column.
• col_level : int or string, optional
  • If columns are a MultiIndex then use this level to melt.

读取state_fruit2数据集

state_fruit2 = pd.read_csv(‘data/state_fruit2.csv‘)
state_fruit2

melt可以将原先的列名作为变量，原先的值作为值

var_name和value_name可以用来重命名新生成的变量列和值的列

state_fruit2.melt(id_vars=[‘State‘],value_vars=[‘Apple‘, ‘Orange‘, ‘Banana‘])

随意设定一个行索引

state_fruit2.index=list(‘abc‘)
state_fruit2.index.name = ‘letter‘
state_fruit2

var_name和value_name可以用来重命名新生成的变量列和值的列
var_name对应value_vars 默认为variable
value_name对应原有表格中的值 默认为value

state_fruit2.melt(id_vars=[‘State‘],
    value_vars=[‘Apple‘, ‘Orange‘, ‘Banana‘],
    var_name=‘Fruit‘,
    value_name=‘Weight‘)

如果想让所有值都位于一列，旧的列标签位于另一列，可以直接使用melt

state_fruit2.melt()

要指明id变量，只需使用id_vars参数

state_fruit2.melt(id_vars=‘State‘)

2 长格式转宽格式

2.1 unstack

• DataFrame.unstack(level=-1, fill_value=None)
• level : int, string, or list of these, default -1 (last level)
  • Level(s) of index to unstack, can pass level name
  • 将行索引中的哪一级别展开
• fill_value : replace NaN with this value if the unstack produces
  • missing values
  • 缺失值怎样填充

读取college数据集，学校名作为行索引，只选取本科生的列

usecol_func = lambda x: ‘UGDS_‘ in x or x == ‘INSTNM‘
college = pd.read_csv(‘data/college.csv‘,index_col=‘INSTNM‘, usecols=usecol_func)

用stack方法，将所有水平列名，转化为垂直的行索引

college_stacked = college.stack()
college_stacked.head(18)

INSTNM                                         
Alabama A & M University             UGDS_WHITE    0.0333
                                     UGDS_BLACK    0.9353
                                                    ...  
University of Alabama at Birmingham  UGDS_NRA      0.0179
                                     UGDS_UNKN     0.0100
Length: 18, dtype: float64

unstack方法可以将其还原

college_stacked.unstack().head()

2.2 pivot

DataFrame.pivot(index=None, columns=None, values=None)

• Return reshaped DataFrame organized by given index / column values.

• 通过给的index和column返回一个重塑的dataframe

• index : string or object, optional
  • Column to use to make new frame’s index. If None, uses existing index.
  • 原有的dataframe中的哪些列的值作为新的datarame索引
• columns : string or object
  • Column to use to make new frame’s columns.
  • 原有的dataframe中的哪些列中的值作为新的datarame的列
• values : string, object or a list of the previous, optional
  • Column(s) to use for populating new frame’s values. If not specified, all remaining columns will be used and the result will have hierarchically indexed columns.
  • 原有的datarame哪个列的值，作为新的表中的值

另一种方式是先用melt，再用pivot。先加载数据，不指定行索引名

college2 = pd.read_csv(‘data/college.csv‘, usecols=usecol_func)

college_melted = college2.melt(id_vars=‘INSTNM‘, var_name=‘Race‘,value_name=‘Percentage‘)
college_melted.head()

用pivot还原

melted_inv = college_melted.pivot(index=‘INSTNM‘,columns=‘Race‘,values=‘Percentage‘)
melted_inv.head()

3 透视表

3.1 使用pivot_table

pandas.pivot_table(data, values=None, index=None, columns=None, aggfunc=‘mean‘, fill_value=None, margins=False, dropna=True, margins_name=‘All‘)

Create a spreadsheet-style pivot table as a DataFrame. The levels in the pivot table will be stored in MultiIndex objects (hierarchical indexes) on the index and columns of the result DataFrame

类似pivot

• data : DataFrame
• values : column to aggregate, optional
• index : column, Grouper, array, or list of the previous
  • If an array is passed, it must be the same length as the data. The list can contain any of the other types (except list). Keys to group by on the pivot table index. If an array is passed, it is being used as the same manner as column values.
• columns : column, Grouper, array, or list of the previous
  • If an array is passed, it must be the same length as the data. The list can contain any of the other types (except list). Keys to group by on the pivot table column. If an array is passed, it is being used as the same manner as column values.
  • 当colomns传递的列表有多个元素时，也会建立入index的多级索引
• aggfunc : function, list of functions, dict, default numpy.mean
  • If list of functions passed, the resulting pivot table will have hierarchical columns whose top level are the function names (inferred from the function objects themselves) If dict is passed, the key is column to aggregate and value is function or list of functions
  • 对所选取的值做聚合
• fill_value : scalar, default None
  • Value to replace missing values with

flights = pd.read_csv(‘data/flights.csv‘)
flights.head()

用pivot_table方法求出每条航线每个始发地的被取消的航班总数

fp = flights.pivot_table(index=‘AIRLINE‘, 
    columns=‘ORG_AIR‘, 
    values=‘CANCELLED‘, 
    aggfunc=‘sum‘,
    fill_value=0).round(2)
fp.head()

3.2 使用groupby方法

groupby聚合不能直接复现这张表,需要先按所有index和columns的列聚合.

fg = flights.groupby([‘AIRLINE‘, ‘ORG_AIR‘])[‘CANCELLED‘].sum()
fg.head()

AIRLINE  ORG_AIR
AA       ATL         3
         DEN         4
         DFW        86
         IAH         3
         LAS         3
Name: CANCELLED, dtype: int64

再使用unstack，将ORG_AIR这层索引作为列名

fg_unstack = fg.unstack(‘ORG_AIR‘, fill_value=0)
fg_unstack.head()

3.3 两种方式的比较

fg_unstack = fg.unstack(‘ORG_AIR‘, fill_value=0)
fp.equals(fg_unstack)

True

fp2 = flights.pivot_table(index=[‘AIRLINE‘, ‘MONTH‘],
    columns=[‘ORG_AIR‘, ‘CANCELLED‘],
    values=[‘DEP_DELAY‘, ‘DIST‘],
    aggfunc=[np.mean, np.sum],
    fill_value=0)
fp2

用groupby和unstack复现上面的方法

flights.groupby([‘AIRLINE‘, ‘MONTH‘, ‘ORG_AIR‘, ‘CANCELLED‘])[‘DEP_DELAY‘, ‘DIST‘]     .agg([‘mean‘, ‘sum‘])     .unstack([‘ORG_AIR‘, ‘CANCELLED‘], fill_value=0)     .swaplevel(0, 1, axis=‘columns‘)     .head()

4 数据清理TIPS

一些数据分析的案例和技巧

4.1 为了更容易reshaping，重新命名索引层

读取college数据集，分组后，统计本科生的SAT数学成绩信息

college = pd.read_csv(‘data/college.csv‘)

cg = college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘, ‘SATMTMID‘].agg([‘count‘, ‘min‘, ‘max‘]).head(6)
cg

行索引的两级都有名字，而列索引没有名字。用rename_axis给列索引的两级命名

cg = cg.rename_axis([‘AGG_COLS‘, ‘AGG_FUNCS‘], axis=‘columns‘)
cg

将AGG_FUNCS列移到行索引

cg.stack(‘AGG_FUNCS‘).head()

stack默认是将列放到行索引的最内层，可以使用swaplevel改变层级

cg.stack(‘AGG_FUNCS‘).swaplevel(‘AGG_FUNCS‘, ‘STABBR‘, axis=‘index‘).head()

在此前的基础上再做sort_index

cg.stack(‘AGG_FUNCS‘)     .swaplevel(‘AGG_FUNCS‘, ‘STABBR‘, axis=‘index‘)     .sort_index(level=‘RELAFFIL‘, axis=‘index‘)     .sort_index(level=‘AGG_COLS‘, axis=‘columns‘).head(6)

对一些列做stack，对其它列做unstack

cg.stack(‘AGG_FUNCS‘).unstack([‘RELAFFIL‘, ‘STABBR‘])

对所有列做stack，会返回一个Series

cg.stack([‘AGG_FUNCS‘, ‘AGG_COLS‘]).head(12)

STABBR  RELAFFIL  AGG_FUNCS  AGG_COLS
AK      0         count      UGDS         7.0
                             SATMTMID     0.0
                                         ... 
AL      0         count      UGDS        71.0
                             SATMTMID    13.0
Length: 12, dtype: float64

删除行和列索引所有层级的名称

cg.rename_axis([None, None], axis=‘index‘).rename_axis([None, None], axis=‘columns‘)

4.2 当多个变量被存储为列名时进行清理

当多个变量被存储为列名时进行清理

读取weightlifting数据集

weightlifting = pd.read_csv(‘data/weightlifting_men.csv‘)
weightlifting

用melt方法，将sex_age放入一个单独的列

wl_melt = weightlifting.melt(id_vars=‘Weight Category‘, 
    var_name=‘sex_age‘, 
    value_name=‘Qual Total‘)
wl_melt.head()

用split方法将sex_age列分为两列

sex_age = wl_melt[‘sex_age‘].str.split(expand=True)
sex_age.head()

sex_age.columns = [‘Sex‘, ‘Age Group‘]
sex_age.head()

只取出字符串中的M

sex_age[‘Sex‘] = sex_age[‘Sex‘].str[0]
sex_age.head()

用concat方法，将sex_age,与wl_cat_total连接起来

wl_cat_total = wl_melt[[‘Weight Category‘, ‘Qual Total‘]]
wl_tidy = pd.concat([sex_age, wl_cat_total], axis=‘columns‘)
wl_tidy.head()

上面的结果也可以如下实现

cols = [‘Weight Category‘, ‘Qual Total‘]
sex_age[cols] = wl_melt[cols]

也可以通过assign的方法，动态加载新的列

age_group = wl_melt.sex_age.str.extract(‘(\d{2}[-+](?:\d{2})?)‘, expand=False)
sex = wl_melt.sex_age.str[0]
new_cols = {‘Sex‘:sex,‘Age Group‘: age_group}
wl_tidy2 = wl_melt.assign(**new_cols).drop(‘sex_age‘, axis=‘columns‘)
wl_tidy2.head()

4.3 当多个变量被存储为列的值时进行清理

读取restaurant_inspections数据集，将Date列的数据类型变为datetime64

inspections = pd.read_csv(‘data/restaurant_inspections.csv‘, parse_dates=[‘Date‘])
inspections.head(10)

4.3.1 stack方式

inspections.set_index([‘Name‘,‘Date‘, ‘Info‘]).unstack(‘Info‘).head()

用reset_index方法，使行索引层级与列索引相同

insp_tidy = inspections.set_index([‘Name‘,‘Date‘, ‘Info‘])     .unstack(‘Info‘)     .reset_index(col_level=-1)
insp_tidy.head()

除掉列索引的最外层，重命名行索引的层为None

insp_tidy.columns = insp_tidy.columns.droplevel(0).rename(None)
insp_tidy.head()

4.3.2 pivot_table方式

pivot_table需要传入聚合函数，才能产生一个单一值

inspections.pivot_table(index=[‘Name‘, ‘Date‘], 
         columns=‘Info‘, 
         values=‘Value‘, 
         aggfunc=‘first‘)     .reset_index()    .rename_axis(None, axis=‘columns‘)

# inspections.pivot(index=[‘Name‘, ‘Date‘], columns=‘Info‘, values=‘Value‘)
# 运行pivot会报错，因为没有聚合函数，通过[‘Name‘, ‘Date‘]索引和columns=‘Info‘对应的是多个值

4.4 当两个或多个值存储于一个单元格时进行清理

读取texas_cities数据集

cities = pd.read_csv(‘data/texas_cities.csv‘)
cities

将Geolocation分解为四个单独的列

geolocations = cities.Geolocation.str.split(pat=‘. ‘, expand=True)
geolocations.columns = [‘latitude‘, ‘latitude direction‘, ‘longitude‘, ‘longitude direction‘]
geolocations

转变数据类型

geolocations = geolocations.astype({‘latitude‘:‘float‘, ‘longitude‘:‘float‘})
geolocations.dtypes

latitude               float64
latitude direction      object
longitude              float64
longitude direction     object
dtype: object

将新列与原先的city列连起来

cities_tidy = pd.concat([cities[‘City‘], geolocations], axis=‘columns‘)
cities_tidy

函数to_numeric可以将每列自动变为整数或浮点数

temp = geolocations.apply(pd.to_numeric, errors=‘ignore‘)
temp

|符，可以对多个标记进行分割

 cities.Geolocation.str.split(pat=‘° |, ‘, expand=True)

更复杂的提取方式

cities.Geolocation.str.extract(‘([0-9.]+). (N|S), ([0-9.]+). (E|W)‘, expand=True)

4.5 当多个变量被存储为列名和列值时进行清理

读取sensors数据集

sensors = pd.read_csv(‘data/sensors.csv‘)
sensors

用melt清理数据

sensors.melt(id_vars=[‘Group‘, ‘Property‘], var_name=‘Year‘).head(6)

用pivot_table，将Property列转化为新的列名

sensors.melt(id_vars=[‘Group‘, ‘Property‘], var_name=‘Year‘)     .pivot_table(index=[‘Group‘, ‘Year‘], columns=‘Property‘, values=‘value‘)     .reset_index()     .rename_axis(None, axis=‘columns‘)

用stack和unstack实现上述方法

sensors.set_index([‘Group‘, ‘Property‘])     .stack()     .unstack(‘Property‘)     .rename_axis([‘Group‘, ‘Year‘], axis=‘index‘)     .rename_axis(None, axis=‘columns‘)     .reset_index()

4.6 当多个观察单位被存储于同一张表时进行清理

就是将一张表拆成多张表

读取movie_altered数据集

movie = pd.read_csv(‘data/movie_altered.csv‘)
movie.head()

插入新的列，用来标识每一部电影

movie.insert(0, ‘id‘, np.arange(len(movie)))

用wide_to_long，将所有演员放到一列，将所有Facebook likes放到一列

stubnames = [‘director‘, ‘director_fb_likes‘, ‘actor‘, ‘actor_fb_likes‘]

movie_long = pd.wide_to_long(movie, stubnames=stubnames, i=‘id‘, j=‘num‘, sep=‘_‘).reset_index()
movie_long[‘num‘] = movie_long[‘num‘].astype(int)
movie_long.head(9)

movie.columns

Index([‘id‘, ‘title‘, ‘rating‘, ‘year‘, ‘duration‘, ‘director_1‘,
       ‘director_fb_likes_1‘, ‘actor_1‘, ‘actor_2‘, ‘actor_3‘,
       ‘actor_fb_likes_1‘, ‘actor_fb_likes_2‘, ‘actor_fb_likes_3‘],
      dtype=‘object‘)

movie_long.columns

Index([‘id‘, ‘num‘, ‘year‘, ‘duration‘, ‘rating‘, ‘title‘, ‘director‘,
       ‘director_fb_likes‘, ‘actor‘, ‘actor_fb_likes‘],
      dtype=‘object‘)

将这个数据分解成多个小表

movie_table = movie_long[[‘id‘,‘title‘, ‘year‘, ‘duration‘, ‘rating‘]]
director_table = movie_long[[‘id‘, ‘director‘, ‘num‘, ‘director_fb_likes‘]]
actor_table = movie_long[[‘id‘, ‘actor‘, ‘num‘, ‘actor_fb_likes‘]]

做一些去重和去除缺失值的工作

movie_table = movie_table.drop_duplicates().reset_index(drop=True)
director_table = director_table.dropna().reset_index(drop=True)
actor_table = actor_table.dropna().reset_index(drop=True)

Pandas Cookbook -- 08数据清理

标签：day   定义   ...   equal   with   category   agg   pass   元素   

原文地址：https://www.cnblogs.com/shiyushiyu/p/9800795.html