.NET 快速实现网页数据抓取
作者:EquatorCoco
- 2024-05-23 福建
本文字数:3074 字
阅读完需:约 10 分钟
网页数据抓取需求
本文我们以抓取博客园 10 天推荐排行榜第一页的文章标题、文章简介和文章地址为示例,并把抓取下来的数据保存到对应的 txt 文本中。
![](https://static001.geekbang.org/infoq/60/60815c5cf01e5d15ee7c0428bf555d19.png)
创建控制台应用
创建名为DotnetSpiderExercise
的控制台应用。
![](https://static001.geekbang.org/infoq/df/df33dd3ac692830eda8e04a73bf285e7.png)
![](https://static001.geekbang.org/infoq/3a/3a5b84a00b74f3066f64bb2ab927f302.png)
![](https://static001.geekbang.org/infoq/f8/f8ed02d2bf3c7fc0264a9e35d5a768d2.png)
安装 DotnetSpider NuGet 包
NuGet 包管理器搜索:DotnetSpider
![](https://static001.geekbang.org/infoq/f0/f02779541559fc67438f0172ddded105.png)
添加 Serilog 日志组件
NuGet 包管理器搜索:Serilog.AspNetCore
![](https://static001.geekbang.org/infoq/66/665f10f8490f83f0354a0825487dddbb.png)
添加 RecommendedRankingModel
namespace DotnetSpiderExercise
{
public class RecommendedRankingModel
{
/// <summary>
/// 文章标题
/// </summary>
public string ArticleTitle { get; set; }
/// <summary>
/// 文章简介
/// </summary>
public string ArticleSummary { get; set; }
/// <summary>
/// 文章地址
/// </summary>
public string ArticleUrl { get; set; }
}
}
复制代码
添加 RecommendedRankingSpider
网页数据抓取的业务逻辑都在这里面。
using DotnetSpider.DataFlow.Parser;
using DotnetSpider.DataFlow;
using DotnetSpider.Downloader;
using DotnetSpider.Http;
using DotnetSpider.Scheduler.Component;
using DotnetSpider.Selector;
using DotnetSpider;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Serilog;
using DotnetSpider.Scheduler;
using Microsoft.Extensions.Hosting;
using System.Reflection;
namespace DotnetSpiderExercise
{
public class RecommendedRankingSpider : Spider
{
public RecommendedRankingSpider(IOptions<SpiderOptions> options,
DependenceServices services,
ILogger<Spider> logger) : base(options, services, logger)
{
}
public static async Task RunAsync()
{
var builder = Builder.CreateDefaultBuilder<RecommendedRankingSpider>();
builder.UseSerilog();
builder.UseDownloader<HttpClientDownloader>();
builder.UseQueueDistinctBfsScheduler<HashSetDuplicateRemover>();
await builder.Build().RunAsync();
}
protected override async Task InitializeAsync(CancellationToken stoppingToken = default)
{
//添加自定义解析
AddDataFlow(new Parser());
//使用控制台存储器
AddDataFlow(new ConsoleStorage());
//添加采集请求:博客园10天推荐排行榜
await AddRequestsAsync(new Request("https://www.cnblogs.com/aggsite/topdiggs")
{
//请求超时10秒
Timeout = 10000
});
}
class Parser : DataParser
{
public override Task InitializeAsync()
{
return Task.CompletedTask;
}
protected override Task ParseAsync(DataFlowContext context)
{
var recommendedRankingList = new List<RecommendedRankingModel>();
// 网页数据解析
var number = 1;
var recommendedList = context.Selectable.SelectList(Selectors.XPath(".//article[@class='post-item']"));
foreach (var news in recommendedList)
{
var articleTitle = news.Select(Selectors.XPath(".//a[@class='post-item-title']"))?.Value;
var articleSummary = news.Select(Selectors.XPath(".//p[@class='post-item-summary']"))?.Value?.Replace("\n", "").Replace(" ", "");
var articleUrl = news.Select(Selectors.XPath(".//a[@class='post-item-title']/@href"))?.Value;
Console.WriteLine($"第{number}篇文章 标题:{articleTitle}");
recommendedRankingList.Add(new RecommendedRankingModel
{
ArticleTitle = articleTitle,
ArticleSummary = articleSummary,
ArticleUrl = articleUrl
});
number++;
}
using (StreamWriter sw = new StreamWriter("RecommendedRanking.txt"))
{
foreach (RecommendedRankingModel model in recommendedRankingList)
{
string line = $"文章标题:{model.ArticleTitle}\r\n文章简介:{model.ArticleSummary}\r\n文章地址:{model.ArticleUrl}";
sw.WriteLine(line + "\r\n ========================================================================================== \r\n");
}
}
return Task.CompletedTask;
}
}
}
}
复制代码
Program 执行数据抓取
namespace DotnetSpiderExercise
{
public class Program
{
static async Task Main(string[] args)
{
Console.WriteLine("网页数据抓取开始...");
await RecommendedRankingSpider.RunAsync();
Console.WriteLine("网页数据抓取完成...");
}
}
}
复制代码
![](https://static001.geekbang.org/infoq/e4/e4e41a3678e61ac0cf3c85a5103371c1.png)
抓取数据和页面数据对比
抓取数据
![](https://static001.geekbang.org/infoq/e5/e5e8d36c62fd982a30cb78fbc4eefc71.png)
页面数据
![](https://static001.geekbang.org/infoq/9f/9f8de0b94e0ecbd0457c1d5fb3786b6a.png)
文章转载自:追逐时光者
划线
评论
复制
发布于: 刚刚阅读数: 2
![用户头像](http://static001.geekbang.org/account/avatar/00/37/bf/d7/e92abddd.jpg?x-oss-process=image/resize,w_200,h_200)
EquatorCoco
关注
还未添加个人签名 2023-06-19 加入
还未添加个人简介
评论