python panda庫從基礎到高級操作分析
1. Pandas 概述
Pandas 是 Python 數(shù)據(jù)科學領域的核心庫,專為處理結構化數(shù)據(jù)而設計。它提供了兩種核心數(shù)據(jù)結構:Series(一維數(shù)組)和DataFrame(二維表格),支持高效的數(shù)據(jù)操作、清洗和分析。Pandas 的主要優(yōu)勢包括:
- 高效處理大型數(shù)據(jù)集
- 靈活的數(shù)據(jù)清洗和轉換功能
- 強大的分組和聚合操作
- 無縫對接其他科學計算庫(如 NumPy、Matplotlib)
import pandas as pd
import numpy as np
# 創(chuàng)建Series(一維數(shù)據(jù)結構)
s = pd.Series([1, 3, 5, np.nan, 6, 8])
print("Series示例:")
print(s)
# 創(chuàng)建DataFrame(二維表格)
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35],
'City': ['New York', 'London', 'Paris']
})
print("\nDataFrame示例:")
print(df)輸出結果:
Series示例:
0 1.0
1 3.0
2 5.0
3 NaN
4 6.0
5 8.0
dtype: float64DataFrame示例:
Name Age City
0 Alice 25 New York
1 Bob 30 London
2 Charlie 35 Paris
2. 基本操作:數(shù)據(jù)讀取與查看
Pandas 提供了豐富的接口用于讀取不同格式的數(shù)據(jù),并支持便捷的數(shù)據(jù)探查功能。
# 構造示例數(shù)據(jù)
data = {
'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
'Age': [25, 30, 35, 28, 22],
'City': ['New York', 'London', 'Paris', 'New York', 'Berlin'],
'Salary': [5000, 6000, 7000, 5500, 4500]
}
df = pd.DataFrame(data)
# 保存為CSV文件
df.to_csv('sample_data.csv', index=False)
# 從CSV讀取數(shù)據(jù)
df = pd.read_csv('sample_data.csv')
# 查看數(shù)據(jù)基本信息
print("數(shù)據(jù)基本信息:")
df.info()
# 查看數(shù)據(jù)前幾行
print("\n數(shù)據(jù)前5行:")
print(df.head())
# 查看數(shù)據(jù)統(tǒng)計摘要
print("\n數(shù)據(jù)統(tǒng)計摘要:")
print(df.describe())
# 查看數(shù)據(jù)形狀
rows, columns = df.shape
print(f"\n數(shù)據(jù)形狀: {rows}行{columns}列")輸出結果:
數(shù)據(jù)基本信息:
<class 'pandas.core.frameworks.dataframe.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 5 non-null object
1 Age 5 non-null int64
2 City 5 non-null object
3 Salary 5 non-null int64
dtypes: int64(2), object(2)
memory usage: 240.0+ bytes數(shù)據(jù)前5行:
Name Age City Salary
0 Alice 25 New York 5000
1 Bob 30 London 6000
2 Charlie 35 Paris 7000
3 David 28 New York 5500
4 Eve 22 Berlin 4500數(shù)據(jù)統(tǒng)計摘要:
Age Salary
count 5.0 5.0
mean 28.0 5600.0
std 5.0 953.9
min 22.0 4500.0
25% 25.0 5000.0
50% 28.0 5500.0
75% 30.0 6000.0
max 35.0 7000.0數(shù)據(jù)形狀: 5行4列
3. 索引操作:精準定位數(shù)據(jù)
Pandas 支持多種索引方式,包括位置索引、標簽索引和布爾索引,滿足不同場景的數(shù)據(jù)訪問需求。
# 位置索引 (iloc)
print("第一行數(shù)據(jù) (iloc[0]):")
print(df.iloc[0])
print("\n前兩行數(shù)據(jù) (iloc[:2]):")
print(df.iloc[:2])
# 標簽索引 (loc)
df.set_index('Name', inplace=True)
print("\n以Name為索引后,查詢Alice的記錄 (loc['Alice']):")
print(df.loc['Alice'])
# 布爾索引
print("\n年齡大于30的記錄:")
print(df[df['Age'] > 30])
# 組合索引
print("\n查詢New York的高收入人群 (Salary > 5000):")
print(df[(df['City'] == 'New York') & (df['Salary'] > 5000)])輸出結果:
第一行數(shù)據(jù) (iloc[0]):
Name Alice
Age 25
City New York
Salary 5000
Name: 0, dtype: object前兩行數(shù)據(jù) (iloc[:2]):
Name Age City Salary
0 Alice 25 New York 5000
1 Bob 30 London 6000以Name為索引后,查詢Alice的記錄 (loc['Alice']):
Age 25
City New York
Salary 5000
Name: Alice, dtype: object年齡大于30的記錄:
Age City Salary
Name
Charlie 35 Paris 7000查詢New York的高收入人群 (Salary > 5000):
Age City Salary
Name
David 28 New York 5500
4. GroupBy 操作:分組聚合分析
GroupBy 是 Pandas 中最強大的功能之一,支持按指定列分組后進行各種聚合計算。
# 按City分組,計算平均年齡
city_age_mean = df.groupby('City')['Age'].mean()
print("各城市平均年齡:")
print(city_age_mean)
# 多列分組并應用多個聚合函數(shù)
grouped = df.groupby(['City', 'Age']).agg({
'Salary': ['mean', 'sum']
})
print("\n按城市和年齡分組后的薪資統(tǒng)計:")
print(grouped)
# 分組后篩選 - 只保留平均薪資大于5500的城市
filtered_groups = df.groupby('City').filter(lambda x: x['Salary'].mean() > 5500)
print("\n平均薪資大于5500的城市記錄:")
print(filtered_groups)
# 分組后應用自定義函數(shù)
def salary_range(x):
return x.max() - x.min()
city_salary_range = df.groupby('City')['Salary'].apply(salary_range)
print("\n各城市薪資范圍:")
print(city_salary_range)輸出結果:
各城市平均年齡:
City
Berlin 22.0
London 30.0
New York 26.5
Paris 35.0
Name: Age, dtype: float64按城市和年齡分組后的薪資統(tǒng)計:
Salary
mean sum
City Age
Berlin 22.0 4500.0 4500
London 30.0 6000.0 6000
New York 25.0 5000.0 5000
28.0 5500.0 5500
Paris 35.0 7000.0 7000平均薪資大于5500的城市記錄:
Age City Salary
Name
Bob 30 London 6000
Charlie 35 Paris 7000各城市薪資范圍:
City
Berlin 0
London 0
New York 500
Paris 0
Name: Salary, dtype: int64
5. 數(shù)值運算:向量化計算
Pandas 支持高效的向量化運算,避免顯式循環(huán),大幅提高計算效率。
# 單列運算 - 所有年齡加1
df['Age'] = df['Age'] + 1
print("年齡加1后的數(shù)據(jù)集:")
print(df)
# 多列運算 - 添加年薪列(假設Salary是月薪)
df['Annual_Salary'] = df['Salary'] * 12
print("\n添加年薪列后的數(shù)據(jù)集:")
print(df)
# 統(tǒng)計函數(shù)
print("\n各城市平均年齡:")
print(df.groupby('City')['Age'].mean())
print("\n各城市平均年薪:")
print(df.groupby('City')['Annual_Salary'].mean())
# 相關系數(shù)計算
print("\nAge與Salary的相關系數(shù):")
print(df['Age'].corr(df['Salary']))輸出結果:
年齡加1后的數(shù)據(jù)集:
Age City Salary Annual_Salary
Name
Alice 26 New York 5000 60000
Bob 31 London 6000 72000
Charlie 36 Paris 7000 84000
David 29 New York 5500 66000
Eve 23 Berlin 4500 54000添加年薪列后的數(shù)據(jù)集:
Age City Salary Annual_Salary
Name
Alice 26 New York 5000 60000
Bob 31 London 6000 72000
Charlie 36 Paris 7000 84000
David 29 New York 5500 66000
Eve 23 Berlin 4500 54000各城市平均年齡:
City
Berlin 23.0
London 31.0
New York 27.5
Paris 36.0
Name: Age, dtype: float64各城市平均年薪:
City
Berlin 54000.0
London 72000.0
New York 63000.0
Paris 84000.0
Name: Annual_Salary, dtype: float64Age與Salary的相關系數(shù):
0.9411252628281636
6. 對象操作:數(shù)據(jù)增刪改查
Pandas 提供了靈活的接口用于數(shù)據(jù)框的結構修改,包括列和行的增刪改查。
# 添加新列 - 年齡段分類
df['Age_Category'] = pd.cut(df['Age'], bins=[20, 25, 30, 35, 40],
labels=['Young', 'Early Career', 'Mid Career', 'Senior'])
print("添加年齡段分類后的數(shù)據(jù)集:")
print(df)
# 刪除列 - 刪除Age列
df.drop('Age', axis=1, inplace=True)
print("\n刪除Age列后的數(shù)據(jù)集:")
print(df)
# 修改數(shù)據(jù) - 將New York的薪資提高5%
df.loc[df['City'] == 'New York', 'Salary'] *= 1.05
print("\n調整New York薪資后的數(shù)據(jù)集:")
print(df)
# 插入行
new_row = pd.DataFrame({
'Name': ['Frank'],
'City': ['Sydney'],
'Salary': [6500],
'Annual_Salary': [6500*12],
'Age_Category': ['Mid Career']
}, index=['Frank'])
df = pd.concat([df, new_row])
print("\n插入新行后的數(shù)據(jù)集:")
print(df)輸出結果:
添加年齡段分類后的數(shù)據(jù)集:
Age City Salary Annual_Salary Age_Category
Name
Alice 26 New York 5000 60000 Early Career
Bob 31 London 6000 72000 Mid Career
Charlie 36 Paris 7000 84000 Senior
David 29 New York 5500 66000 Early Career
Eve 23 Berlin 4500 54000 Young刪除Age列后的數(shù)據(jù)集:
City Salary Annual_Salary Age_Category
Name
Alice New York 5000 60000 Early Career
Bob London 6000 72000 Mid Career
Charlie Paris 7000 84000 Senior
David New York 5500 66000 Early Career
Eve Berlin 4500 54000 Young調整New York薪資后的數(shù)據(jù)集:
City Salary Annual_Salary Age_Category
Name
Alice New York 5250 63000 Early Career
Bob London 6000 72000 Mid Career
Charlie Paris 7000 84000 Senior
David New York 5775 69300 Early Career
Eve Berlin 4500 54000 Young插入新行后的數(shù)據(jù)集:
City Salary Annual_Salary Age_Category
Name
Alice New York 5250 63000 Early Career
Bob London 6000 72000 Mid Career
Charlie Paris 7000 84000 Senior
David New York 5775 69300 Early Career
Eve Berlin 4500 54000 Young
Frank Sydney 6500 78000 Mid Career
7. Merge 操作:數(shù)據(jù)合并
Pandas 支持多種方式合并不同的數(shù)據(jù)框,包括內連接、外連接等常見數(shù)據(jù)庫操作。
# 創(chuàng)建第二個數(shù)據(jù)框
df2 = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Charlie', 'Frank'],
'Department': ['Engineering', 'Marketing', 'Engineering', 'HR']
})
# 內連接 - 只保留兩個數(shù)據(jù)框都有的Name
merged_inner = pd.merge(df, df2, on='Name', how='inner')
print("內連接結果:")
print(merged_inner)
# 左連接 - 保留左數(shù)據(jù)框的所有記錄
merged_left = pd.merge(df, df2, on='Name', how='left')
print("\n左連接結果:")
print(merged_left)
# 右連接 - 保留右數(shù)據(jù)框的所有記錄
merged_right = pd.merge(df, df2, on='Name', how='right')
print("\n右連接結果:")
print(merged_right)
# 全連接 - 保留兩個數(shù)據(jù)框的所有記錄
merged_outer = pd.merge(df, df2, on='Name', how='outer')
print("\n全連接結果:")
print(merged_outer)輸出結果:
內連接結果:
City Salary Annual_Salary Age_Category Department
Name
Alice New York 5250 63000 Early Career Engineering
Bob London 6000 72000 Mid Career Marketing
Charlie Paris 7000 84000 Senior Engineering
Frank Sydney 6500 78000 Mid Career HR左連接結果:
City Salary Annual_Salary Age_Category Department
Name
Alice New York 5250 63000 Early Career Engineering
Bob London 6000 72000 Mid Career Marketing
Charlie Paris 7000 84000 Senior Engineering
David New York 5775 69300 Early Career NaN
Eve Berlin 4500 54000 Young NaN
Frank Sydney 6500 78000 Mid Career HR右連接結果:
City Salary Annual_Salary Age_Category Department
Name
Alice New York 5250 63000 Early Career Engineering
Bob London 6000 72000 Mid Career Marketing
Charlie Paris 7000 84000 Senior Engineering
Frank Sydney 6500 78000 Mid Career HR全連接結果:
City Salary Annual_Salary Age_Category Department
Name
Alice New York 5250 63000 Early Career Engineering
Bob London 6000 72000 Mid Career Marketing
Charlie Paris 7000 84000 Senior Engineering
David New York 5775 69300 Early Career NaN
Eve Berlin 4500 54000 Young NaN
Frank Sydney 6500 78000 Mid Career HR
8. 數(shù)據(jù)透視表:多維數(shù)據(jù)分析
數(shù)據(jù)透視表是分析多維數(shù)據(jù)的強大工具,支持在行、列維度上同時進行分組和聚合。
# 創(chuàng)建示例數(shù)據(jù)
data = {
'Year': [2020, 2020, 2020, 2021, 2021, 2021],
'Month': [1, 2, 3, 1, 2, 3],
'City': ['New York', 'New York', 'London', 'New York', 'London', 'London'],
'Sales': [100, 120, 110, 130, 140, 150]
}
sales_df = pd.DataFrame(data)
# 創(chuàng)建基本數(shù)據(jù)透視表 - 按年和月分組,計算Sales總和
pivot_sales = sales_df.pivot_table(
values='Sales',
index='Year',
columns='Month',
aggfunc='sum'
)
print("按年和月分組的銷售數(shù)據(jù)透視表:")
print(pivot_sales)
# 多層數(shù)據(jù)透視表 - 同時按年、月和城市分組
multi_pivot = sales_df.pivot_table(
values='Sales',
index=['Year', 'City'],
columns='Month',
aggfunc='sum'
)
print("\n按年、城市和月分組的銷售數(shù)據(jù)透視表:")
print(multi_pivot)
# 應用多個聚合函數(shù)
pivot_agg = sales_df.pivot_table(
values='Sales',
index='Year',
columns='City',
aggfunc=['sum', 'mean', 'count']
)
print("\n應用多個聚合函數(shù)的數(shù)據(jù)透視表:")
print(pivot_agg)輸出結果:
按年和月分組的銷售數(shù)據(jù)透視表:
Month 1 2 3
Year
2020 100 120 110
2021 130 140 150按年、城市和月分組的銷售數(shù)據(jù)透視表:
Month 1 2 3
Year City
2020 New York 100 120 NaN
London NaN NaN 110
2021 New York 130 NaN NaN
London NaN 140 150應用多個聚合函數(shù)的數(shù)據(jù)透視表:
sum mean count
City New York London New York London New York London
Year
2020 220 110 2 1 2 1
2021 130 290 1 2 1 2
9. 時間序列處理
Pandas 對時間序列數(shù)據(jù)提供了強大的支持,包括日期解析、頻率轉換和時間差計算。
# 創(chuàng)建帶時間序列的數(shù)據(jù)
dates = pd.date_range(start='2023-01-01', periods=6, freq='M')
sales = [100, 120, 110, 130, 140, 150]
ts_df = pd.DataFrame({
'Date': dates,
'Sales': sales
})
# 轉換為datetime類型
ts_df['Date'] = pd.to_datetime(ts_df['Date'])
print("時間序列數(shù)據(jù):")
print(ts_df)
# 提取時間特征
ts_df['Year'] = ts_df['Date'].dt.year
ts_df['Month'] = ts_df['Date'].dt.month
ts_df['Quarter'] = ts_df['Date'].dt.quarter
print("\n添加時間特征后:")
print(ts_df)
# 時間序列重采樣 - 按季度聚合
ts_df.set_index('Date', inplace=True)
quarterly_sales = ts_df.resample('Q').sum()
print("\n按季度聚合的銷售數(shù)據(jù):")
print(quarterly_sales)
# 計算時間差
ts_df['Previous_Sales'] = ts_df['Sales'].shift(1)
ts_df['Sales_Growth'] = ts_df['Sales'] - ts_df['Previous_Sales']
print("\n添加銷售增長數(shù)據(jù)后:")
print(ts_df)輸出結果:
時間序列數(shù)據(jù):
Date Sales
0 2023-01-31 100
1 2023-02-28 120
2 2023-03-31 110
3 2023-04-30 130
4 2023-05-31 140
5 2023-06-30 150添加時間特征后:
Date Sales Year Month Quarter
0 2023-01-31 100 2023 1 1
1 2023-02-28 120 2023 2 1
2 2023-03-31 110 2023 3 1
3 2023-04-30 130 2023 4 2
4 2023-05-31 140 2023 5 2
5 2023-06-30 150 2023 6 2按季度聚合的銷售數(shù)據(jù):
Sales Year Month Quarter
Date
2023-03-31 330 2023 3 1
2023-06-30 420 2023 6 2添加銷售增長數(shù)據(jù)后:
Sales Year Month Quarter Previous_Sales Sales_Growth
Date
2023-01-31 100 2023 1 1 NaN NaN
2023-02-28 120 2023 2 1 100.0 20.0
2023-03-31 110 2023 3 1 120.0 -10.0
2023-04-30 130 2023 4 2 110.0 20.0
2023-05-31 140 2023 5 2 130.0 10.0
2023-06-30 150 2023 6 2 140.0 10.0
10. 大數(shù)據(jù)處理技巧
處理大規(guī)模數(shù)據(jù)時,Pandas 提供了多種優(yōu)化方法,包括分塊讀取、數(shù)據(jù)類型優(yōu)化和內存管理。
# 分塊讀取大文件示例
def process_chunk(chunk):
"""處理數(shù)據(jù)塊的示例函數(shù)"""
return chunk.groupby('City')['Salary'].mean()
# 模擬大文件分塊讀取
chunksize = 2
results = []
for chunk in pd.read_csv('sample_data.csv', chunksize=chunksize):
results.append(process_chunk(chunk))
# 合并分塊處理結果
final_result = pd.concat(results)
print("分塊處理結果:")
print(final_result)
# 優(yōu)化數(shù)據(jù)類型以減少內存占用
df['Salary'] = df['Salary'].astype('int32') # 從int64轉為int32
print("\n優(yōu)化數(shù)據(jù)類型后內存使用:")
df.info()
# 使用category類型處理分類數(shù)據(jù)
df['City'] = df['City'].astype('category')
print("\n使用category類型后內存使用:")
df.info()
# 稀疏數(shù)據(jù)處理
sparse_data = pd.Series([0, 0, 0, 5, 0, 0, 10, 0], dtype='Sparse[int]')
print("\n稀疏數(shù)據(jù)示例:")
print(sparse_data)輸出結果:
分塊處理結果:
City
New York 5000.0
London 6000.0
New York 5500.0
Berlin 4500.0
dtype: float64優(yōu)化數(shù)據(jù)類型后內存使用:
<class 'pandas.core.frameworks.dataframe.DataFrame'>
Index: 5 entries, Alice to Frank
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 City 5 non-null object
1 Salary 5 non-null int32
2 Annual_Salary 5 non-null int64
3 Age_Category 5 non-null object
dtypes: int32(1), int64(1), object(2)
memory usage: 240.0+ bytes使用category類型后內存使用:
<class 'pandas.core.frameworks.dataframe.DataFrame'>
Index: 5 entries, Alice to Frank
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 City 5 non-null category
1 Salary 5 non-null int32
2 Annual_Salary 5 non-null int64
3 Age_Category 5 non-null object
dtypes: category(1), int32(1), int64(1), object(1)
memory usage: 228.0+ bytes稀疏數(shù)據(jù)示例:
0 0
1 0
2 0
3 5
4 0
5 0
6 10
7 0
dtype: Sparse[int, 0]
學習總結:Pandas 核心函數(shù)與方法
1. 數(shù)據(jù)結構創(chuàng)建
pd.Series(data):創(chuàng)建一維 Series 對象pd.DataFrame(data):創(chuàng)建二維 DataFrame 對象pd.read_csv(filepath):從 CSV 文件讀取數(shù)據(jù)pd.read_excel(filepath):從 Excel 文件讀取數(shù)據(jù)pd.date_range(start, periods, freq):創(chuàng)建日期范圍
2. 數(shù)據(jù)查看與信息
df.info():查看數(shù)據(jù)基本信息df.head(n):查看前 n 行數(shù)據(jù)df.tail(n):查看后 n 行數(shù)據(jù)df.describe():生成描述性統(tǒng)計df.shape:獲取數(shù)據(jù)行列數(shù)
3. 索引與篩選
df.iloc[]:基于位置的索引df.loc[]:基于標簽的索引df[condition]:布爾索引df.set_index(col):設置指定列為索引df.reset_index():重置索引
4. 數(shù)據(jù)操作
df.drop(col, axis=1):刪除列df.assign(new_col=value):添加新列df.rename(columns={}):重命名列df.append(row):添加行df.concat([df1, df2]):拼接數(shù)據(jù)框
5. 分組與聚合
df.groupby(col):按列分組grouped.agg(func):應用聚合函數(shù)grouped.apply(func):應用自定義函數(shù)df.pivot_table():創(chuàng)建數(shù)據(jù)透視表grouped.filter(func):分組后篩選
6. 數(shù)據(jù)合并
pd.merge(df1, df2, on=col):合并數(shù)據(jù)框df.join(df2):按索引合并pd.concat([df1, df2]):拼接數(shù)據(jù)框df.append(row):添加行數(shù)據(jù)
7. 時間序列
pd.to_datetime(col):轉換為日期時間df.resample(freq):時間序列重采樣df.shift(n):數(shù)據(jù)位移df.diff():計算差分pd.date_range():生成日期范圍
8. 性能優(yōu)化
df.chunked = pd.read_csv(..., chunksize=):分塊讀取df.astype(dtype):優(yōu)化數(shù)據(jù)類型df[col] = df[col].astype('category'):使用分類類型pd.SparseSeries():處理稀疏數(shù)據(jù)
到此這篇關于python panda庫從基礎到高級操作的文章就介紹到這了,更多相關python panda庫內容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關文章希望大家以后多多支持腳本之家!
相關文章
selenium2.0中常用的python函數(shù)匯總
這篇文章主要介紹了selenium2.0中常用的python函數(shù),總結分析了selenium2.0中常用的python函數(shù)的功能、原理與基本用法,需要的朋友可以參考下2019-08-08
python中isdigit() isalpha()用于判斷字符串的類型問題
這篇文章主要介紹了python中isdigit() isalpha()用于判斷字符串的類型問題,具有很好的參考價值,希望對大家有所幫助。如有錯誤或未考慮完全的地方,望不吝賜教2022-11-11
python判斷所輸入的任意一個正整數(shù)是否為素數(shù)的兩種方法
今天小編就為大家分享一篇python判斷所輸入的任意一個正整數(shù)是否為素數(shù)的兩種方法,具有很好的參考價值,希望對大家有所幫助。一起跟隨小編過來看看吧2019-06-06

