一個Python案例帶你掌握xpath數(shù)據(jù)解析方法

更新時間：2022年02月23日 14:59:09 作者：Python數(shù)據(jù)分析實(shí)例

xpath解析是最常用且最便捷高效的一種解析方式，通用性強(qiáng)。本文將通過一個Python爬蟲案例帶你詳細(xì)了解一下xpath數(shù)據(jù)解析方法，需要的可以參考一下

xpath基本概念

xpath解析：最常用且最便捷高效的一種解析方式。通用性強(qiáng)。

xpath解析原理

1.實(shí)例化一個etree的對象，且需要將被解析的頁面源碼數(shù)據(jù)加載到該對象中

2.調(diào)用etree對象中的xpath方法結(jié)合xpath表達(dá)式實(shí)現(xiàn)標(biāo)簽的定位和內(nèi)容的捕獲。

環(huán)境安裝

pip?install?lxml

如何實(shí)例化一個etree對象

from?lxml?import?etree

1.將本地的html文件中的遠(yuǎn)嗎數(shù)據(jù)加載到etree對象中：

etree.parse(filePath)

2.可以將從互聯(lián)網(wǎng)上獲取的原碼數(shù)據(jù)加載到該對象中：

etree.HTML(‘page_text')

xpath(‘xpath表達(dá)式’)

1./:表示的是從根節(jié)點(diǎn)開始定位。表示一個層級

2.//:表示多個層級。可以表示從任意位置開始定位

3.屬性定位：//div[@class='song'] tag[@attrName='attrValue']

4.索引定位：//div[@class='song']/p[3] 索引從1開始的

5.取文本：

/text()獲取的是標(biāo)簽中直系的文本內(nèi)容
//text()標(biāo)簽中非直系的文本內(nèi)容（所有文本內(nèi)容）

6.取屬性：/@attrName ==>img/src

xpath爬取58二手房實(shí)例

完整代碼

from?lxml?import?etree
import?requests

if?__name__?==?'__main__':
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/84.0.4147.105?Safari/537.36'
????}
????url?=?'https://xa.58.com/ershoufang/'
????page_text?=?requests.get(url=url,headers=headers).text
????tree?=?etree.HTML(page_text)
????div_list?=?tree.xpath('//section[@class="list"]/div')
????fp?=?open('./58同城二手房.txt','w',encoding='utf-8')
????for?div?in?div_list:
????????title?=?div.xpath('.//div[@class="property-content-title"]/h3/text()')[0]
????????print(title)
????????fp.write(title+'\n'+'\n')

xpath圖片解析下載實(shí)例

完整代碼

import?requests,os
from?lxml?import?etree

if?__name__?==?'__main__':
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/84.0.4147.105?Safari/537.36'
????}
????url?=?'https://pic.netbian.com/4kmeinv/'
????page_text?=?requests.get(url=url,headers=headers).text
????tree?=?etree.HTML(page_text)
????li_list?=?tree.xpath('//div[@class="slist"]/ul/li/a')
????if?not?os.path.exists('./piclibs'):
????????os.mkdir('./piclibs')
????for?li?in?li_list:
????????detail_url?='https://pic.netbian.com'?+?li.xpath('./img/@src')[0]
????????detail_name?=?li.xpath('./img/@alt')[0]+'.jpg'
????????detail_name?=?detail_name.encode('iso-8859-1').decode('GBK')
????????detail_path?=?'./piclibs/'?+?detail_name
????????detail_data?=?requests.get(url=detail_url,?headers=headers).content
????????with?open(detail_path,'wb')?as?fp:
????????????fp.write(detail_data)
????????????print(detail_name,'seccess!!')

xpath爬取全國城市名稱實(shí)例

完整代碼

import?requests
from?lxml?import?etree

if?__name__?==?'__main__':
????url?=?'https://www.aqistudy.cn/historydata/'
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/87.0.4280.141?Safari/537.36',
????}
????page_text?=?requests.get(url=url,headers=headers).content.decode('utf-8')
????tree?=?etree.HTML(page_text)
????#熱門城市???//div[@class="bottom"]/ul/li
????#全部城市???//div[@class="bottom"]/ul/div[2]/li
????a_list?=?tree.xpath('//div[@class="bottom"]/ul/li?|?//div[@class="bottom"]/ul/div[2]/li')
????fp?=?open('./citys.txt','w',encoding='utf-8')
????i?=?0
????for?a?in?a_list:
????????city_name?=?a.xpath('.//a/text()')[0]
????????fp.write(city_name+'\t')
????????i=i+1
????????if?i?==?6:
????????????i?=?0
????????????fp.write('\n')
????print('爬取成功')

xpath爬取簡歷模板實(shí)例

完整代碼

import?requests,os
from?lxml?import?etree

if?__name__?==?'__main__':
????url?=?'https://sc.chinaz.com/jianli/free.html'
????headers?=?{
????????'User-Agent':?'Mozilla/5.0?(Windows?NT?10.0;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/87.0.4280.141?Safari/537.36',
????}
????page_text?=?requests.get(url=url,headers=headers).content.decode('utf-8')
????tree?=?etree.HTML(page_text)
????a_list?=?tree.xpath('//div[@class="box?col3?ws_block"]/a')
????if?not?os.path.exists('./簡歷模板'):
????????os.mkdir('./簡歷模板')
????for?a?in?a_list:
????????detail_url?=?'https:'+a.xpath('./@href')[0]
????????detail_page_text?=?requests.get(url=detail_url,headers=headers).content.decode('utf-8')
????????detail_tree?=?etree.HTML(detail_page_text)
????????detail_a_list?=?detail_tree.xpath('//div[@class="clearfix?mt20?downlist"]/ul/li[1]/a')
????????for?a?in?detail_a_list:
????????????download_name?=?detail_tree.xpath('//div[@class="ppt_tit?clearfix"]/h1/text()')[0]
????????????download_url?=?a.xpath('./@href')[0]
????????????download_data?=?requests.get(url=download_url,headers=headers).content
????????????download_path?=?'./簡歷模板/'+download_name+'.rar'
????????????with?open(download_path,'wb')?as?fp:
????????????????fp.write(download_data)
????????????????print(download_name,'success!!')