快捷導(dǎo)航

一文教你Python如何快速精準(zhǔn)抓取網(wǎng)頁數(shù)據(jù)

更新時間：2025年04月30日 16:19:07 作者：Python_trys

這篇文章主要為大家詳細(xì)介紹了如何利用Python實現(xiàn)快速精準(zhǔn)抓取網(wǎng)頁數(shù)據(jù),文中的示例代碼簡潔易懂,具有一定的借鑒價值,有需要的小伙伴可以了解下

1. 準(zhǔn)備工作

首先安裝必要的庫：

pip install requests beautifulsoup4

2. 基礎(chǔ)爬蟲實現(xiàn)

import requests
from bs4 import BeautifulSoup
import time
import random

def get_csdn_articles(keyword, pages=1):
    """
    抓取CSDN上指定關(guān)鍵詞的文章
    :param keyword: 搜索關(guān)鍵詞
    :param pages: 要抓取的頁數(shù)
    :return: 文章列表，包含標(biāo)題、鏈接、簡介等信息
    """
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
    }
    
    base_url = "https://so.csdn.net/so/search"
    articles = []
    
    for page in range(1, pages + 1):
        params = {
            'q': keyword,
            't': 'blog',
            'p': page
        }
        
        try:
            response = requests.get(base_url, headers=headers, params=params)
            response.raise_for_status()
            
            soup = BeautifulSoup(response.text, 'html.parser')
            items = soup.find_all('div', class_='search-item')
            
            for item in items:
                title_tag = item.find('a', class_='title')
                if not title_tag:
                    continue
                    
                title = title_tag.get_text().strip()
                link = title_tag['href']
                
                # 獲取簡介
                desc_tag = item.find('p', class_='content')
                description = desc_tag.get_text().strip() if desc_tag else '無簡介'
                
                # 獲取閱讀數(shù)和發(fā)布時間
                info_tags = item.find_all('span', class_='date')
                read_count = info_tags[0].get_text().strip() if len(info_tags) > 0 else '未知'
                publish_time = info_tags[1].get_text().strip() if len(info_tags) > 1 else '未知'
                
                articles.append({
                    'title': title,
                    'link': link,
                    'description': description,
                    'read_count': read_count,
                    'publish_time': publish_time
                })
            
            print(f"已抓取第 {page} 頁，共 {len(items)} 篇文章")
            
            # 隨機(jī)延遲，避免被封
            time.sleep(random.uniform(1, 3))
            
        except Exception as e:
            print(f"抓取第 {page} 頁時出錯: {e}")
            continue
    
    return articles

if __name__ == '__main__':
    # 示例：抓取關(guān)于"Python爬蟲"的前3頁文章
    keyword = "

3. 高級功能擴(kuò)展

3.1 抓取文章詳情

def get_article_detail(url):
    """抓取文章詳情內(nèi)容"""
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
    }
    
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # 獲取文章主體內(nèi)容
        content = soup.find('article')
        if content:
            # 清理不必要的標(biāo)簽
            for tag in content(['script', 'style', 'iframe', 'nav', 'footer']):
                tag.decompose()
            return content.get_text().strip()
        
        return "無法獲取文章內(nèi)容"
    except Exception as e:
        print(f"抓取文章詳情出錯: {e}")
        return None

3.2 保存數(shù)據(jù)到文件

import json
import csv

def save_to_json(data, filename):
    """保存數(shù)據(jù)到JSON文件"""
    with open(filename, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=2)

def save_to_csv(data, filename):
    """保存數(shù)據(jù)到CSV文件"""
    if not data:
        return
        
    keys = data[0].keys()
    
    with open(filename, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=keys)
        writer.writeheader()
        writer.writerows(data)

4. 完整示例

if __name__ == '__main__':
    # 抓取文章列表
    keyword = "Python爬蟲"
    articles = get_csdn_articles(keyword, pages=2)
    
    # 抓取前3篇文章的詳情
    for article in articles[:3]:
        article['content'] = get_article_detail(article['link'])
        time.sleep(random.uniform(1, 2))  # 延遲
    
    # 保存數(shù)據(jù)
    save_to_json(articles, 'csdn_articles.json')
    save_to_csv(articles, 'csdn_articles.csv')
    
    print("數(shù)據(jù)抓取完成并已保存！")

5. 反爬蟲策略應(yīng)對

1.設(shè)置請求頭：模擬瀏覽器訪問

2.隨機(jī)延遲：避免請求過于頻繁

3.使用代理IP：防止IP被封

4.處理驗證碼：可能需要人工干預(yù)

5.遵守robots.txt：尊重網(wǎng)站的爬蟲規(guī)則

到此這篇關(guān)于一文教你Python如何快速精準(zhǔn)抓取網(wǎng)頁數(shù)據(jù)的文章就介紹到這了,更多相關(guān)Python抓取網(wǎng)頁數(shù)據(jù)內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！

您可能感興趣的文章: