mapreduce_mapreduce技术文章

mapreduce

0 人感兴趣 · 47 次引用

关注

最新
推荐

https://static001.geekbang.org/infoq/16/16ade6bf7562e17e3f6aedbfd0e01283.jpeg?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

MapReduce 作业生命周期

穿过生命散发芬芳

2022-10-24

MapReduce作业作为一种分布式应用程序，可直接运行在Hadoop资源管理系统YARN之上（MapReduce On YARN）。每个MapReduce应用程序由一个MRAppMaster以及一系列MapTask和ReduceTask构成，它们通过ResourceManager获得资源，并由NodeManager启动运行。

mapreduce 10月月更

https://static001.geekbang.org/infoq/9f/9f79347afebc4259d61717f2ab089eee.gif?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

在 Hadoop 环境里面统计西游记文章的词组（hdfs 实验）

王小王-123

2022-06-29

在Hadoop的环境里面我们测试一个文件，对西游记这个书籍里面的所有词语，进行统计排序。hadoop包括两个核心组成：HDFS：分布式文件系统，存储海量的数据MapReduce：并行处理框架，实现任务分解和调度。

mapreduce hadoop统计词频 hadoop项目文本统计分析

开发一个不需要重写成 Hive QL 的大数据 SQL 引擎

华为云开发者联盟

2022-04-29

开发一款能支持标准数据库SQL的大数据仓库引擎，让那些在Oracle上运行良好的SQL可以直接运行在Hadoop上，而不需要重写成Hive QL。

sql 大数据 mapreduce SQL引擎大数据仓库

带你了解极具弹性的 Spark 架构的原理

华为云开发者联盟

2022-04-22

相比MapReduce僵化的Map与Reduce分阶段计算相比，Spark的计算框架更加富有弹性和灵活性，运行性能更佳。

spark mapreduce 内存 RDD Spark架构

不 care 工具，在大数据平台中 Hive 能自动处理 SQL

华为云开发者联盟

2022-04-19

有没有更简单的办法，可以直接将SQL运行在大数据平台？

sql 大数据 mapreduce hive 数据分析

坐实大数据资源调度框架之王，Yarn 为何这么牛

华为云开发者联盟

2022-04-12

Yarn的出现伴随着Hadoop的发展，使Hadoop从一个单一的大数据计算引擎，成为大数据的代名词。

大数据 hadoop mapreduce YARN 资源调度框架

https://static001.geekbang.org/infoq/26/2669016f6f4c8e8832a9c16990449d16.jpeg?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

MapReduce 学习思考

2022-04-04

MapReduce是我最近知晓的一个编程模型，简单学习了一下，发现它的设计思路和之前Rob Pike演讲的go语言并发模式还有一点异曲同工之妙，所以在此整理归纳一下。

mapreduce

大数据培训 hive 和 mapreduce 的区别

@零度

2022-03-23

考虑到维护的成本的增大,慢慢的开始准备将MR的作业,逐渐的移植到一些脚本平台上去，hive成了我们的首选_大数据培训。

mapreduce hive 大数据开发

https://static001.geekbang.org/infoq/60/608778d210ce6b4290568a810dbb25c3.png?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

Map-Reduce 思想在 ABAP 编程中的一个实际应用案例

Jerry Wang

2022-03-18

Map-Reduce 是一种编程模型和相关实现，用于在集群上使用并行分布式算法，生成和处理大规模数据集。ABAP 编程语言从语言层面支持对大规模数据的 REDUCE 操作。本文分享了笔者工作中使用Map-Reduce思路处理大规模数据集的一个实际案例，希望能给大家带来启发。

mapreduce abap CRM系统企业级应用 3月月更

Hadoop 中 mapreduce 作业日志是如何生成的

华为云开发者联盟

2022-01-11

摘要：本篇博客介绍了hadoop中mapreduce类型的作业日志是如何生成的。主要介绍日志生成的几个关键过程，不涉及过多细节性的内容。

hadoop hdfs mapreduce YARN 作业日志

Hadoop 学习过程中遇到的错误及解决方法

阿丞

2021-12-28

在自己学习搭建并使用Hadoop集群的时候，遇到的一些错误及解决方法。

hadoop hdfs mapreduce YARN

MapReduce 示例：减少 Hadoop MapReduce 中的侧连接

华为云开发者联盟

2021-09-17

摘要：在排序和reducer 阶段，reduce 侧连接过程会产生巨大的网络I/O 流量，在这个阶段，相同键的值被聚集在一起。

hadoop mapreduce 排序侧连接联接

https://static001.geekbang.org/infoq/6f/6fa13c501de472b158bd5b0119774fc2.png?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

Spark 必读！总有一些 Spark 知识点你需要知道

Qunar技术沙龙

2021-08-27

一文梳理 Spark 必知必会知识点既可以快速对 Spark 有个简单清晰的认知；也可以回顾 Spark 基础概念和原理；还可以进行知识点查缺补漏。

spark hdfs mapreduce Web UI 数据倾斜

云小课｜MRS 基础原理之 ClickHouse 组件介绍

华为云开发者联盟

2021-08-20

阅识风云是华为云信息大咖，擅长将复杂信息多元化呈现，其出品的一张图(云图说)、深入浅出的博文(云小课)或短视频(云视厅)总有一款能让您快速上手华为云。更多精彩内容请单击此处。

mapreduce 开源 Clickhouse EI企业智能列式数据库

MapReduce 排序以及序列化实践

五分钟学大数据

2021-07-14

序列化（Serialization）是指把结构化对象转化为字节流。

mapreduce 7月日更

https://static001.geekbang.org/infoq/24/243acf13475ad61cc92837d92b08b722.jpeg?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

腾讯云 EMR 基于 YARN 针对云原生容器化的优化与实践

腾讯云大数据

2021-06-28

HADOOP资源调度器YARN在容器环境中能多有效率？

mapreduce

https://static001.geekbang.org/infoq/29/2942095ecc9aefda5e10f385e171ddcd.jpeg?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

MapReduce 排序以及序列化

五分钟学大数据

2021-05-12

序列化（Serialization）是指把结构化对象转化为字节流。

大数据 hadoop mapreduce

https://static001.geekbang.org/infoq/1e/1ed4815548718a63507fb8e744dd215d.jpeg?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

MapReduce 的运行机制详解

五分钟学大数据

2021-03-23

整个Map阶段流程大体如上图所示。

大数据 mapreduce 3月日更

12 周架构

FreeOcean

2021-02-12

大数据概要总结

hadoop mapreduce YARN

https://static001.geekbang.org/infoq/71/71dae3826e559ebc0b9688c2bec3efbf.jpeg?x-oss-process=image%2Fresize%2Cw_416%2Ch_234

MapReduce 练习案例 4 - 求共同好友

小马哥

2021-01-20

大数据专栏系列文章: 今天告诉你如何通过玩转 MapReduce 计数器技术, 观察MR任务的执行

大数据 mapreduce 七日更

{ "code": 0, "data": { "list": [ { "icon": "", "id": 0, "link": "https://xie.infoq.cn/square/10", "name": "活动🔥", "type": 2 }, { "icon": "", "id": 3, "link": "", "name": "Java", "type": 1 }, { "icon": "", "id": 211, "link": "", "name": "开源", "type": 1 }, { "icon": "", "id": 238, "link": "", "name": "架构", "type": 1 }, { "icon": "", "id": 9434, "link": "", "name": "OpenHarmony", "type": 1 }, { "icon": "", "id": 303, "link": "", "name": "算法", "type": 1 }, { "icon": "", "id": 14135, "link": "", "name": "元宇宙", "type": 1 }, { "icon": "", "id": 84, "link": "", "name": "MySQL", "type": 1 }, { "icon": "", "id": 5237, "link": "", "name": "移动开发", "type": 1 }, { "icon": "", "id": 221, "link": "", "name": "学习方法", "type": 1 }, { "icon": "", "id": 221, "link": "", "name": "学习方法", "type": 1 }, { "icon": "", "id": 17178, "link": "", "name": "Web3.0", "type": 1 }, { "icon": "", "id": 203, "link": "", "name": "高效工作", "type": 1 }, { "icon": "", "id": 92, "link": "", "name": "数据库", "type": 1 }, { "icon": "", "id": 8, "link": "", "name": "Python", "type": 1 }, { "icon": "", "id": 661, "link": "", "name": "音视频", "type": 1 }, { "icon": "", "id": 371, "link": "", "name": "前端", "type": 1 }, { "icon": "", "id": 242, "link": "", "name": "AI", "type": 1 }, { "icon": "", "id": 143, "link": "", "name": "大数据", "type": 1 }, { "icon": "", "id": 204, "link": "", "name": "团队管理", "type": 1 }, { "icon": "", "id": 232, "link": "", "name": "程序员", "type": 1 }, { "icon": "", "id": 317, "link": "", "name": "运维", "type": 1 }, { "icon": "", "id": 200, "link": "", "name": "深度思考", "type": 1 }, { "icon": "", "id": 1081, "link": "", "name": "低代码", "type": 1 }, { "icon": "", "id": 97, "link": "", "name": "redis", "type": 1 }, { "icon": "", "id": 19, "link": "", "name": "golang", "type": 1 }, { "icon": "", "id": 294, "link": "", "name": "微服务架构", "type": 1 }, { "icon": "", "id": 21, "link": "", "name": "flutter", "type": 1 } ] }, "error": { }, "extra": { "cost": 0.000670645, "request-id": "5d7befc2bf92437f8ea101d35f0ecc02@3@xie" } }

{ "code": 0, "data": { "bot": [ { "list": [ { "name": "关于我们", "link": "https://www.infoq.cn/about", "image": "" }, { "name": "我要投稿", "link": "https://www.infoq.cn/contribute", "image": "" }, { "name": "合作伙伴", "link": "https://www.geekbang.org/partner", "image": "" }, { "name": "加入我们", "link": "https://www.lagou.com/gongsi/j43775.html", "image": "" }, { "name": "关注我们", "link": "https://infoq.cn/official/account", "image": "" } ], "name": "InfoQ" }, { "list": [ { "name": "内容投稿：editors@geekbang.com", "link": "mailto:editors@geekbang.com", "image": "" }, { "name": "业务合作：hezuo@geekbang.com", "link": "mailto:hezuo@geekbang.com", "image": "" }, { "name": "反馈投诉：feedback@geekbang.com", "link": "mailto:feedback@geekbang.com", "image": "" }, { "name": "加入我们：zhaopin@geekbang.com", "link": "mailto:zhaopin@geekbang.com", "image": "" }, { "name": "联系电话：18600430890", "link": "", "image": "" }, { "name": "地址：北京市朝阳区望京北路9号2幢7层A701", "link": "", "image": "" } ], "name": "联系我们" }, { "list": [ { "name": "ArchSummit全球架构师峰会 2023年3月17-18日", "link": "https://archsummit.infoq.cn/202303/beijing/?utm_source=infoq&utm_medium=footer", "image": "https://static001.infoq.cn/resource/image/62/f5/628b5e750e918f52cda61a19fdc186f5.png" }, { "name": "ArchSummit全球架构师峰会 2023年4月21-22日", "link": "https://archsummit.infoq.cn/2023/shanghai/?utm_source=infoq&utm_medium=footer", "image": "https://static001.infoq.cn/resource/image/68/86/682e88b72d998d472686410a05c67986.png" }, { "name": "QCon全球软件开发大会 2023年5月26-27日", "link": "https://qcon.infoq.cn/2023/guangzhou/?utm_source=infoq&utm_medium=footer", "image": "https://static001.infoq.cn/resource/image/b5/8c/b5214dd411ae938ab2ddfb504e7d078c.png" } ], "name": "InfoQ 近期会议" }, { "list": [ { "name": "InfoQ En", "link": "https://www.infoq.com/", "image": "https://static001.infoq.cn/resource/image/55/38/55cd81623e36f5ab7a7db74d60b74838.png" }, { "name": "InfoQ Jp", "link": "https://www.infoq.com/jp/", "image": "https://static001.infoq.cn/resource/image/95/13/95fe851c02c86120e9037eada6a36d13.png" }, { "name": "InfoQ Fr", "link": "http://www.infoq.com/fr/", "image": "https://static001.infoq.cn/resource/image/2a/3e/2aa440b6d94e94f64c508f16da38933e.png" }, { "name": "InfoQ Br", "link": "http://www.infoq.com/br/", "image": "https://static001.infoq.cn/resource/image/4e/1e/4e737ce82bc7c8a1c2f2307bcea9a11e.png" } ], "name": "全球 InfoQ" } ], "bus": [ { "end_time": 1703865599, "img": "https://static001.infoq.cn/resource/image/db/3c/db58e00a7fc616fcdd32c777726b6c3c.jpg", "link": "https://qcon.infoq.cn/2023/shanghai?utm_source=infoqweb&utm_medium=huodongdabenying", "location": "上海", "name": "QCon 上海 2023", "start_time": 1703692800 }, { "end_time": 1702483200, "img": "https://static001.infoq.cn/resource/image/06/ff/06730d2377fa0677cd103e11f3409dff.png", "link": "https://www.infoq.cn/form/?id=1862&utm_source=1&sign=iq_652681f86704a", "location": "线上", "name": "《云上新视界》火山引擎视频云系列公开课", "start_time": 1701273600 }, { "end_time": 1703952000, "img": "https://static001.infoq.cn/resource/image/8b/ed/8b130ca27d9c71d61e3f766eff44d1ed.jpeg", "link": "https://github.com/orgs/secretflow/discussions/1007", "location": "线上", "name": "隐语开源共建计划 OSCP 第三期", "start_time": 1700064000 }, { "end_time": 1703260799, "img": "https://static001.infoq.cn/resource/image/3a/b8/3a54e625d4c5bef52ee34822dcf09eb8.jpg", "link": "https://aistudio.baidu.com/competition/detail/1073/0/introduction", "location": "线上", "name": "飞桨黑客马拉松第五期", "start_time": 1695571200 }, { "end_time": 1702137599, "img": "https://static001.infoq.cn/resource/image/b6/af/b6f6e813aae99c579c1617b41274e5af.jpg", "link": "https://www.infoq.cn/form/?id=1928&utm_source=web&sign=iq_655ec0806c6d5", "location": "线下", "name": "2023全球AI前沿科技大会", "start_time": 1702051200 }, { "end_time": 1701273600, "img": "https://static001.infoq.cn/resource/image/fa/2e/fa9e3975a19e07e20684ce166116ec2e.png", "link": "https://www.infoq.cn/event/?id=276&utm_source=hddby&sign=iq_654dacd3d53a5", "location": "线上", "name": "【有奖征文】赢取丰厚稿酬 11-30 线上", "start_time": 1699545600 }, { "end_time": 1702742400, "img": "https://static001.infoq.cn/resource/image/20/db/206b8a3dyyf45d3b9d2b25d8d66d83db.jpg", "link": "https://aistudio.baidu.com/competition/detail/1103/0/introduction", "location": "首钢园创新中心D座", "name": "飞桨 x 文心大模型大模型黑客松", "start_time": 1700496000 }, { "end_time": 1697903999, "img": "https://static001.infoq.cn/resource/image/6f/91/6fa11c8f3550ccf09bd47c32da187a91.jpg", "link": "https://doris-summit.org.cn/?utm_source=InfoQ&utm_medium=banner&utm_id=2023", "location": "北京·丽亭华苑酒店", "name": "Doris Summit Asia 2023：与创新者同行", "start_time": 1697817600 }, { "end_time": 1695311999, "img": "https://static001.infoq.cn/resource/image/21/55/21e55fc4b6775544e13f7683a8118f55.png", "link": "https://www.infoq.cn/minibook/Iwk2LLuMFSV4AisWG8jR", "location": "线上", "name": "互联网行业再进化——云上AI时代", "start_time": 1695225600 }, { "end_time": 1701359999, "img": "https://static001.infoq.cn/resource/image/0d/ab/0dc1c972f72decab4eaea377238e7dab.jpg", "link": "https://cloud.baidu.com/qianfandev/bac?track=infoq", "location": "线上", "name": "百度智能云千帆大模型平台应用开发挑战赛", "start_time": 1694966400 }, { "end_time": 1695657599, "img": "https://static001.infoq.cn/resource/image/47/20/472634d97a6a5e980f9d3e3261cd9520.jpg", "link": "https://cn.selectdb.com/events/1", "location": "线上", "name": "新内核新图景 - 飞轮科技全系产品发布会", "start_time": 1695571200 }, { "end_time": 1695571199, "img": "https://static001.infoq.cn/resource/image/65/bf/657e16d927a221653d2c9cd7byy11fbf.png", "link": "https://hdxu.cn/c1Jg6", "location": "上海", "name": "MatrixOne Meetup", "start_time": 1695484800 }, { "end_time": 1698508799, "img": "https://static001.infoq.cn/resource/image/38/cd/387a0735e11d8ed6a39270aa93f27bcd.jpeg", "link": "https://www.paddlepaddle.org.cn/PaddlePaddleFoundathon-2023-8", "location": "线上", "name": "AGI Foundathon大模型创业松", "start_time": 1698422400 }, { "end_time": 1693324800, "img": "https://static001.infoq.cn/resource/image/ae/3d/ae7d0247fda95f8fe60645ff6dff4a3d.jpg", "link": "https://www.infoq.cn/event/?id=273", "location": "线上", "name": "Geekathon 大模型技术应用创新大赛", "start_time": 1688054400 }, { "end_time": 1695139199, "img": "https://static001.infoq.cn/resource/image/c8/2d/c8a645e0632d33293b7f2bd8068fa12d.jpg", "link": "https://live.polyv.cn/watch/4175199?promoteId=LgSt21", "location": "线上", "name": "9.19 青云科技 AI 算力发布会", "start_time": 1695052800 }, { "end_time": 1695398400, "img": "https://static001.infoq.cn/resource/image/40/9b/4099a26e6400d52ffae41e1354b7729b.jpg", "link": "https://season-share.7-event.cn/FormTemplate/index.html#/?formID=368&showID=10458&trace_code=MW9ibpqQFva4", "location": "广州", "name": "数聚·源力 2023PRO_大会", "start_time": 1693238400 }, { "end_time": 1694015999, "img": "https://static001.infoq.cn/resource/image/65/45/65079701ba555721b6bd01c527586c45.jpg", "link": "https://www.infoq.cn/form/?id=1777&utm_source=hd&sign=iq_64e886900c011", "location": "线上", "name": "存储难题新解法揭秘极致易用的HBlock", "start_time": 1693324800 }, { "end_time": 1693583999, "img": "https://static001.infoq.cn/resource/image/a6/11/a6007122e73b33144c2989075dc22311.png", "link": "https://www.infoq.cn/form/?id=1769&utm_source=5&sign=iq_64db3b0ef25e1", "location": "深圳", "name": "APP开发者个人信息保护培训宣讲会·深圳站", "start_time": 1693497600 }, { "end_time": 1692719999, "img": "https://static001.infoq.cn/resource/image/01/79/01c91f1230ce7b4e9e8dbe816d3ee879.jpg", "link": "https://volcengine2023.cevent.com.cn/VideoCloud&AIGC.html?s=infoQ", "location": "线上", "name": "火山引擎视频云&AIGC技术大会", "start_time": 1692633600 }, { "end_time": 1693929599, "img": "https://static001.infoq.cn/resource/image/7b/f6/7b0291ea5e4f9f088ffa2630fe3711f6.png", "link": "https://qcon.infoq.cn/202309/beijing?utm_source=infoqweb&utm_medium=homehdbanner&utm_campaign=8&utm_term=0607", "location": "北京", "name": "启航·AIGC软件工程变革", "start_time": 1693670400 }, { "end_time": 1693497599, "img": "https://static001.infoq.cn/resource/image/d4/57/d498e5cbf950ae00a3babf87384fab57.jpg", "link": "https://www.infoq.cn/event/?id=271&utm_source=6&sign=iq_64bddefbaa31b", "location": "线上", "name": "2小时学会10个组件“取代”传统Java开发", "start_time": 1690128000 }, { "end_time": 1691510400, "img": "https://static001.infoq.cn/resource/image/4d/59/4de42267c4daf12130d7901892b2a759.jpg", "link": "https://www.infoq.cn/form/?id=1746&utm_source=4&sign=iq_64c9f1e213e56", "location": "线上", "name": "大模型时代的 AI 基础设施——百度AI大底座", "start_time": 1690905600 }, { "end_time": 1693238399, "img": "https://static001.infoq.cn/resource/image/d2/70/d289a3fde03151ddc21e9a134a843b70.png", "link": "https://sf.163.com/lcapcompetition?fromnsf=qzzu_SELF_INFO_m_infoq_23ddmdsinfoq", "location": "线上", "name": "2023网易低代码大赛", "start_time": 1686844800 }, { "end_time": 1684252799, "img": "https://static001.infoq.cn/resource/image/59/8b/59c58407651ca6275a3c3b7f40db318b.jpg", "link": "https://live.infoq.cn/room/1758?utm_source=infoq-web&utm_medium=rightbanner", "location": "线上", "name": "金融企业的数字化转型秘钥", "start_time": 1684166400 }, { "end_time": 1683993599, "img": "https://static001.infoq.cn/resource/image/72/13/7270cf7fb9782b02550cf3291d43a513.jpg", "link": "https://www.infoq.cn/event/?id=262&utm_source=guanwang&sign=iq_6440e2058c94b", "location": "北京", "name": "出海领航｜数据战略创新驱动业务增长", "start_time": 1683907200 }, { "end_time": 1682870399, "img": "https://static001.infoq.cn/resource/image/e1/99/e1eabe46d33074c03b3f91b0c9939999.jpg", "link": "https://www.infoq.cn/minibook/Is534v4mBvUmgC1uJEdc", "location": "线上", "name": "腾讯大规模云原生技术实践案例集", "start_time": 1679932800 }, { "end_time": 1679500799, "img": "https://static001.infoq.cn/resource/image/d4/d0/d451e80da1422c401c7c69a64a7791d0.jpg", "link": "https://www.nvidia.cn/gtc-global/sessions/china-ai-day/?ncid=pa-dis-othe-205289", "location": "线上", "name": "GTC 大会热门会议精选", "start_time": 1679414400 }, { "end_time": 1682870399, "img": "https://static001.infoq.cn/resource/image/30/25/3012263d0391f1d6eb801ff836639025.jpg", "link": "https://www.industrial-bigdata.com/Competition", "location": "线上", "name": "大数据建模预测技术实赛演练正在进行中！", "start_time": 1676390400 }, { "end_time": 1672156799, "img": "https://static001.infoq.cn/resource/image/bf/33/bf86f775aa09a22d6e87d0462449b633.jpg", "link": "https://dev.amazoncloud.cn/activity/activityDetail?id=63959a6132e83b0359766ff9", "location": "线上", "name": "数据与机器学习：面向未来的数据战略", "start_time": 1672070400 }, { "end_time": 1671551999, "img": "https://static001.infoq.cn/resource/image/d1/7a/d1dc55ef96919aa6286767a0e31f137a.jpg", "link": "https://www.awsevents.cn/reInvent2022/roving-exhibition.aspx?code=b29c46ea7b874cbb8310b53921670f9e&s=7982&smid=15648", "location": "北京", "name": "构建者之夜：用一朵云探索技术新边界", "start_time": 1671465600 }, { "end_time": 1668787199, "img": "https://static001.infoq.cn/resource/image/d4/3e/d414d2f3c0feb7cae2a32803c9b1953e.jpg", "link": "https://www.infoq.cn/form/?id=1265", "location": "线上", "name": "《中国企业数智化创新TOP50 》", "start_time": 1667750400 }, { "end_time": 1681833599, "img": "https://static001.infoq.cn/resource/image/69/f1/694cc0a66b82c55216fe5be7b575e5f1.jpg", "link": "https://www.volcengine.com/event/force-spring?utm_source=social&utm_medium=chuanbo_nocontact&utm_term=infoq&utm_campaign=20230418&utm_content=springforce", "location": "线上", "name": "2023春季火山引擎FORCE原动力大会", "start_time": 1681747200 }, { "end_time": 1682524799, "img": "https://static001.infoq.cn/resource/image/11/f0/11ce3bc00389596a57d09fa9873900f0.jpg", "link": "https://www.infoq.cn/event/?id=264&utm_source=hddby&sign=iq_643e136988c34", "location": "杭州", "name": "蚂蚁集团数字科技开发者大会 · 2023", "start_time": 1682438400 }, { "end_time": 1681919999, "img": "https://static001.infoq.cn/resource/image/30/50/30979d04a9b03377ddbcd01a1f1d1850.jpg", "link": "https://www.infoq.cn/event/?id=263", "location": "北京", "name": "OpenHarmony 开发者大会 2023", "start_time": 1681833600 }, { "end_time": 1672156799, "img": "https://static001.infoq.cn/resource/image/c3/cd/c3af01a108dbdbc66127843a61531dcd.jpg", "link": "https://app.jingsocial.com/microFrontend/leadGeneration/jsf-leads/list/webinar/AF8YFe4LVVkzAoL8GTTi9Y/dPFWGd7HbwrT3khbgPfKum", "location": "线上", "name": "研发效率提升与可观测体系建设", "start_time": 1672070400 }, { "end_time": 1667923199, "img": "https://static001.infoq.cn/resource/image/2e/7a/2ef2160e5b4a15yy92bc1b84b358yy7a.jpeg", "link": "https://services.google.cn/fb/forms/1108next22/?channel=infoq", "location": "线上", "name": "Google Cloud Next 中文精选课", "start_time": 1667836800 }, { "end_time": 1667577599, "img": "https://static001.infoq.cn/resource/image/87/cd/873bf0e3965bba90601449aea4f4c3cd.jpg", "link": "https://www.huaweicloud.com/lab/data/luntan.html", "location": "杭州", "name": "华为云数智高峰论坛", "start_time": 1667491200 }, { "end_time": 1667491200, "img": "https://static001.infoq.cn/resource/image/9b/53/9bb2b6dd90f2083d809e7d2b67cbb453.jpg", "link": "https://www.bagevent.com/event/sales/kdxhshikk7fixlcaf4rg9uqxf6qmwr34", "location": "线上", "name": "RTE2022 第八届实时互联网大会", "start_time": 1667232000 }, { "end_time": 1667577600, "img": "https://static001.infoq.cn/resource/image/c9/97/c9abc8ef3298c095f13689741f428497.jpg", "link": "https://openanolis.cn/summitMeeting", "location": "杭州云栖小镇国际会展", "name": "2022云栖大会龙蜥操作系统峰会", "start_time": 1667404800 }, { "end_time": 1669737600, "img": "https://static001.infoq.cn/resource/image/01/3f/01dc280a768yy70f2aaa009c08f75d3f.jpeg", "link": "https://datayi.cn/w/GR4vgygo", "location": "线上", "name": "2022中国企业敏捷实践调研全面启动", "start_time": 1666281600 }, { "end_time": 1666799999, "img": "https://static001.infoq.cn/resource/image/5c/81/5c889ef6c2dfbf94c4f48b8e3f47e581.jpg", "link": "https://static001.geekbang.org/resource/image/37/cb/376438fe7496af1596dcd1e1bc6d8dcb.png", "location": "视频号", "name": "YMatrix 5.0 新品发布会", "start_time": 1666713600 }, { "end_time": 1665503999, "img": "https://static001.infoq.cn/resource/image/7d/7f/7d02690f3b83682719f69f36a47ba87f.png", "link": "https://home.netease.im/promotion/innovation2022", "location": "线上", "name": "网易云信 Innovation 2022 开发者大赛", "start_time": 1663776000 }, { "end_time": 1664035199, "img": "https://static001.infoq.cn/resource/image/40/93/4031be87ff4495608a4e8c72ed233c93.jpg", "link": "https://survey.aliyun.com/apps/zhiliao/-NasGL9m3", "location": "广州", "name": "OceanBase Meetup 广州站", "start_time": 1663516800 }, { "end_time": 1664035199, "img": "https://static001.infoq.cn/resource/image/45/74/4553ac06ae495382409c09489eb0e974.png", "link": "https://summitasia2022infoq.starrocks.com", "location": "线上", "name": "StarRocks Summit Asia 2022", "start_time": 1663948800 }, { "end_time": 1662047999, "img": "https://static001.infoq.cn/resource/image/b0/f4/b0b22daef32a0affa031f7d013057af4.png", "link": "https://datayi.cn/w/a9B3Z7wR", "location": "线上", "name": "企业服务行业研发管理解决方案发布会", "start_time": 1661961600 }, { "end_time": 1661270400, "img": "https://static001.infoq.cn/resource/image/5e/9e/5e86555d83a40c531919ec39e38b279e.jpg", "link": "https://opensource.alibaba.com/collection/osweek2022#asoc", "location": "线上", "name": "阿里巴巴开源开放周", "start_time": 1661097600 }, { "end_time": 1660665599, "img": "https://static001.infoq.cn/resource/image/69/bb/6921feca8052910ca9e0da42d9ca00bb.jpg", "link": "https://app.jingsocial.com/microFrontend/leadGeneration/jsf-leads/list/seminar/NStgRxTWXMFTfBRockKi54/n4C9K5FnnnJUyxTcH3LejM", "location": "北京", "name": "SUSECON 北京 2022", "start_time": 1660579200 }, { "end_time": 1663257599, "img": "https://static001.infoq.cn/resource/image/b9/54/b9de045280504b10eb7299d274df6154.jpg", "link": "https://www.infoq.cn/form/?id=1198&utm_source=gw&sign=iq_6319b6f8c9040", "location": "线上", "name": "Kyligence 2022 秋季线上论坛", "start_time": 1663171200 }, { "end_time": 1660406399, "img": "https://static001.infoq.cn/resource/image/95/99/955562245afddc63235e2054c619fe99.jpg", "link": "https://www.taosdata.com/events/tdcon2022?utm_source=infoq&utm_medium=event&utm_campaign=tdcon2022", "location": "北京", "name": "TDengine 开发者大会", "start_time": 1660320000 }, { "end_time": 1658937599, "img": "https://static001.infoq.cn/resource/image/b0/49/b0f8a4ce0e4b55048b07b9fa48c96e49.jpg", "link": "https://xie.infoq.cn/article/c6dcbcac25f3778008dc0a81d", "location": "线上", "name": "小红书技术对话", "start_time": 1658851200 }, { "end_time": 1658937599, "img": "https://static001.infoq.cn/resource/image/37/yy/37168630c2882711c56e74225ee5f2yy.jpg", "link": "https://mp.weixin.qq.com/s/i7rec4dYdAXI375saiZeQg", "location": "北京", "name": "2022开放原子全球开源峰会龙蜥专场", "start_time": 1658851200 }, { "end_time": 1658591999, "img": "https://static001.infoq.cn/resource/image/17/07/17ff78e333920ee4d44a7737c0779507.jpg", "link": "https://conf.juejin.cn/xdc2022/?utm_source=infoq#s2", "location": "线上", "name": "稀土开发者大会", "start_time": 1658419200 }, { "end_time": 1664380800, "img": "https://static001.infoq.cn/resource/image/2c/ff/2cc082a108813bf4cfe277c7da4a7aff.png", "link": "https://www.infoq.cn/event/?id=235", "location": "线上", "name": "飞桨黑客马拉松第三期", "start_time": 1656864000 }, { "end_time": 1656518399, "img": "https://static001.infoq.cn/resource/image/0b/0e/0b47b188aff652f7bc8b0bb8a33b940e.jpg", "link": "https://tapdata.net/tapdata-LDP-launch-event.html", "location": "线上", "name": "Tapdata LDP 产品发布暨开源说明会", "start_time": 1656432000 }, { "end_time": 1657382399, "img": "https://static001.infoq.cn/resource/image/52/22/52e48d297c5d725ec03e825ddcb81e22.png", "link": "https://mp.weixin.qq.com/s/pPafW2_wMprRh_RW1vlGWw", "location": "北京", "name": "ITCP联盟“架构”主题开放性大会", "start_time": 1657296000 }, { "end_time": 1656863999, "img": "https://static001.infoq.cn/resource/image/2y/7c/2yyaf4ba00f96b57c94ab8eb525bcc7c.jpg", "link": "https://xie.infoq.cn/article/be47a5e41e3b5c75ead7f1c00", "location": "线上", "name": "火山引擎开发者社区征文大赛", "start_time": 1654790400 }, { "end_time": 1656086399, "img": "https://static001.infoq.cn/resource/image/50/89/50a44a422e8b885c7e7f6f06ddc48389.jpg", "link": "https://www.infoq.cn/form/?id=1044&utm_source=hddby&sign=iq_62a80575c32cb", "location": "线上", "name": "面向体验的视频云-火山引擎增长沙龙", "start_time": 1656000000 }, { "end_time": 1655913599, "img": "https://static001.infoq.cn/resource/image/15/b9/15d8102b3f97e099a82d1010585338b9.jpg", "link": "https://www.infoq.cn/event/?id=227&utm_source=hd&sign=iq_62a0dbf6289a4", "location": "线上", "name": "2022 OpenCloudOS 社区开放日", "start_time": 1655827200 }, { "end_time": 1656604799, "img": "https://static001.infoq.cn/resource/image/fb/47/fbaae22a09682f707c102e2aa1311847.png", "link": "https://gitee.com/activity/9th/", "location": "线上", "name": "Gitee 9 周年，以有源之水，引万物生长", "start_time": 1653580800 }, { "end_time": 1653062399, "img": "https://static001.infoq.cn/resource/image/e0/ca/e0beyy709dd8b4d54cbb69103276e3ca.png", "link": "https://www.infoq.cn/event/?id=226&utm_source=hddby&sign=iq_627e147087158", "location": "北京", "name": "WAVE SUMMIT 2022 深度学习开发者峰会", "start_time": 1652976000 }, { "end_time": 1650038400, "img": "https://static001.infoq.cn/resource/image/2d/86/2db870b36b19f82218b537402b93ae86.jpg", "link": "https://dive.infoq.cn/2021/beijing", "location": "线上", "name": "DIVE 全球基础软件创新大会", "start_time": 1649952000 }, { "end_time": 1655567999, "img": "https://static001.infoq.cn/resource/image/39/c0/3929995e7d2032ca8bc280da55718fc0.jpg", "link": "https://www.infoq.cn/minibook/O8BNoAf7iKAGqKzmMJ2w", "location": "线上", "name": "超低延时直播白皮书下载", "start_time": 1654963200 }, { "end_time": 1649347199, "img": "https://static001.infoq.cn/resource/image/97/ff/97d4301eda24df6cf451d6d69a886cff.jpg", "link": "https://live.infoq.cn/room/1369", "location": "线上", "name": "《数据 Cool 谈》第4期", "start_time": 1649260800 }, { "end_time": 1649260799, "img": "https://static001.infoq.cn/resource/image/94/d2/94c1c28c2d0511f6e83d6df7d90d22d2.jpg", "link": "https://live.infoq.cn/room/1374", "location": "线上", "name": "头号云话题：进击的开源操作系统", "start_time": 1649174400 }, { "end_time": 1656086399, "img": "https://static001.infoq.cn/resource/image/26/c6/261e08b762b70634ce4ea332eebc11c6.jpg", "link": "https://www.infoq.cn/event/?id=220&utm_source=hd&sign=iq_6220daec95bce", "location": "北京", "name": "云数据库的架构设计与技术演进专场", "start_time": 1656000000 }, { "end_time": 1656086399, "img": "https://static001.infoq.cn/resource/image/30/92/30855fdd90ef86706c70bed295be0b92.jpg", "link": "https://www.infoq.cn/event/?id=219&utm_source=hd&sign=iq_6220d3a70dc6d", "location": "北京", "name": "企业云原生成本优化最佳实践专场", "start_time": 1656000000 }, { "end_time": 1641657600, "img": "https://static001.infoq.cn/resource/image/f8/88/f874b150cfc8e234fb382c222d6cfc88.png", "link": "https://developer.aliyun.com/special/ffa2021/live", "location": "线上", "name": "Flink Forward Asia 2021 峰会", "start_time": 1641571200 }, { "end_time": 1641484799, "img": "https://static001.infoq.cn/resource/image/a6/2b/a64c9b98dde31fe3850f93744720eb2b.jpg", "link": "https://live.infoq.cn/room/1267", "location": "线上", "name": "DC2021 分布式数据库开发者大会", "start_time": 1641398400 }, { "end_time": 1647964799, "img": "https://static001.infoq.cn/resource/image/24/b7/243eeb97ae454057f1c9560e7ce73ab7.jpg", "link": "https://www.infoq.cn/form/?id=881", "location": "线上", "name": "【参与有奖】Red Hat IDC 业务价值白皮书", "start_time": 1647273600 }, { "end_time": 1626624000, "img": "https://static001.infoq.cn/resource/image/8d/bd/8d43034b3b26906f8f0c68e100c574bd.jpg", "link": "https://www.infoq.cn/event/?id=184", "location": "线上", "name": "平安科技高校挑战赛", "start_time": 1623254400 }, { "end_time": 1576166399, "img": "https://static001.infoq.cn/resource/image/28/ae/283ae28141e197c05a69d7144011b1ae.jpg", "link": "https://www.geekbang.org/page/aws2019.html?utm_source=infoq&utm_medium=specialtopic", "location": "上海", "name": "亚马逊云科技技术专区", "start_time": 1576080000 }, { "end_time": 1574697599, "img": "https://static001.infoq.cn/resource/image/43/6d/43ac3623d95c0c0d2e98c04756c4d46d.jpg", "link": "https://www.infoq.cn/zones/talkingdata/2019/t11/", "location": "北京", "name": "T11数据智能峰会", "start_time": 1574611200 }, { "end_time": 1592668799, "img": "https://static001.infoq.cn/resource/image/77/56/77983803e1e48c2b822cb8fc00790d56.jpg", "link": "https://www.infoq.cn/zones/huawei/shengteng/", "location": "杭州", "name": "华为昇腾技术沙龙", "start_time": 1592582400 }, { "end_time": 1564761599, "img": "https://static001.infoq.cn/resource/image/c4/a7/c45a97169699bfd58a85b89292bfb0a7.jpg", "link": "https://www.infoq.cn/zones/intel/2019/ai.html?utm_campaign=2019_q4_dcg_ai&utm_source=infoq&utm_medium=display_banner&utm_content=ai", "location": "线上", "name": "Intel AI", "start_time": 1564675200 }, { "end_time": 1567871999, "img": "https://static001.infoq.cn/resource/image/0f/e5/0f7eeb8c8e828a1153bccd81be8fa7e5.jpg", "link": "https://www.infoq.cn/zones/huawei/cloudnativedays.html", "location": "南京", "name": "华为云 MeetUp", "start_time": 1567785600 }, { "end_time": 1543075199, "img": "https://static001.infoq.cn/resource/image/36/42/361cyy0d3345298ee32d91b3ab4e9442.jpg", "link": "https://www.geekbang.org/page/thunderchain.html", "location": "上海", "name": "迅雷链技术专区", "start_time": 1542988800 }, { "end_time": 1544889599, "img": "https://static001.infoq.cn/resource/image/73/ed/735d47d1144e550cce1d635bb48481ed.jpg", "link": "https://cloud.tencent.com/developer/conf/2018-beijing?channel=tbzt", "location": "北京", "name": "云+社区开发者大会", "start_time": 1544803200 }, { "end_time": 1611331199, "img": "https://static001.infoq.cn/resource/image/71/28/713c4225c9b2b6f76febd8b391631a28.jpg", "link": "https://www.infoq.cn/zones/huawei/cloud/", "location": "北京", "name": "华为云5G+X联创营", "start_time": 1611244800 }, { "end_time": 1607788799, "img": "https://static001.infoq.cn/resource/image/77/56/77983803e1e48c2b822cb8fc00790d56.jpg", "link": "https://www.infoq.cn/zones/huawei/devrun.html", "location": "珠海", "name": "华为云 DevRun 专区", "start_time": 1607702400 }, { "end_time": 1599148799, "img": "https://static001.infoq.cn/resource/image/46/ec/46fbffbf4fb530315eaa169ae7e5e6ec.jpg", "link": "https://www.infoq.cn/zones/microsoft/?utm_source=guanwang&utm_medium=0831", "location": "线上", "name": "微软技术专区", "start_time": 1599062400 }, { "end_time": 1639756799, "img": "https://static001.infoq.cn/resource/image/96/d8/964639b427776db5368ff2a173a043d8.jpg", "link": "https://www.infoq.cn/zones/baidu-salon/", "location": "北京", "name": "百度技术沙龙", "start_time": 1639670400 }, { "end_time": 1645977599, "img": "https://static001.infoq.cn/resource/image/3f/05/3fecb9e2d446abf609e9cf711e61db05.jpg", "link": "https://www.infoq.cn/zones/TechReview2021/", "location": "线上", "name": "2021 InfoQ 年度技术盘点与展望", "start_time": 1643040000 }, { "end_time": 1644076799, "img": "https://static001.infoq.cn/resource/image/72/7c/7294e60ebf8e5ae85d5730b5aaa7e27c.png", "link": "https://www.infoq.cn/event/?id=217&utm_source=gb&sign=iq_61f0b62007217", "location": "线上", "name": "2021中国数字经济灯塔奖获奖名单公布", "start_time": 1643040000 }, { "end_time": 1650988799, "img": "https://static001.infoq.cn/resource/image/30/92/30855fdd90ef86706c70bed295be0b92.jpg", "link": "https://www.infoq.cn/event/?id=219&utm_source=hd&sign=iq_6220d3a70dc6d", "location": "北京", "name": "企业云原生成本优化最佳实践专场·", "start_time": 1650902400 }, { "end_time": 1648742399, "img": "https://static001.infoq.cn/resource/image/3c/70/3cd7a69b044df8d2280da05344992a70.png", "link": "https://b.geekbang.org/con/dtds/202203/online?utm_source=infoq&utm_medium=event&utm_campaign=dtdslive&gk_source=2022031401infoqevent", "location": "线上", "name": "DTDS 全球数字人才发展线上峰会", "start_time": 1648656000 }, { "end_time": 1650988799, "img": "https://static001.infoq.cn/resource/image/26/c6/261e08b762b70634ce4ea332eebc11c6.jpg", "link": "https://www.infoq.cn/event/?id=220&utm_source=hd&sign=iq_6220daec95bce", "location": "北京", "name": "云数据库的架构设计与技术演进专场·", "start_time": 1650902400 }, { "end_time": 1650124799, "img": "https://static001.infoq.cn/resource/image/2d/86/2db870b36b19f82218b537402b93ae86.jpg", "link": "https://dive.infoq.cn/2021/beijing", "location": "线上", "name": "DIVE 全球基础软件创新大会", "start_time": 1649952000 }, { "end_time": 1651161599, "img": "https://static001.infoq.cn/resource/image/2d/46/2dc27869cccf20ff71e38a745e45fe46.jpeg", "link": "https://www.guance.com/productrelease", "location": "线上", "name": "专为IT工程师定制｜2022观测云产品发布会", "start_time": 1651075200 }, { "end_time": 1653148799, "img": "https://static001.infoq.cn/resource/image/8d/af/8daa9543de9668d8136db9d0526567af.jpg", "link": "https://app.ma.scrmtech.com/meetings/MeetingPc/Detail?pf_uid=27715_2095&id=58292&source=2&pf_type=3&channel_id=28603&channel_name=InfoQ&tag_id=e227c32eddab16f3", "location": "线上", "name": "TDengine用户案例专场", "start_time": 1653062400 }, { "end_time": 1656604799, "img": "https://static001.infoq.cn/resource/image/3e/c1/3eaa69eb89184ab16531402e2b4aa5c1.jpg", "link": "https://www.infoq.cn/minibook/gSVX7kVxVd1CIgfDSLD9?id=973&utm_source=gw&sign=iq_628470f6ed9ec", "location": "线上", "name": "2021年中国分布式数据库市场报告", "start_time": 1653235200 }, { "end_time": 1656518400, "img": "https://static001.infoq.cn/resource/image/bf/51/bf18022yy2616cd9d55dfbcaa4b1c951.jpg", "link": "https://www.infoq.cn/minibook/zH4ITB3UzibuYg3UTxyB", "location": "线上", "name": "腾讯云原生路线图手册", "start_time": 1652716800 }, { "end_time": 1657900799, "img": "https://static001.infoq.cn/resource/image/c1/2e/c17aab59632dc41aaa5aeaa0429ce82e.jpg", "link": "https://www.infoq.cn/event/?id=232&utm_source=gw&sign=iq_62c29c5f81751", "location": "深圳", "name": "腾讯自研业务大规模云原生实践专场", "start_time": 1657814400 }, { "end_time": 1658937599, "img": "https://static001.infoq.cn/resource/image/d9/d0/d952785ef4fe64yy390ac0fa8c93f6d0.png", "link": "https://mp.weixin.qq.com/s/gNlVUcSNfRh6OwhwKP-2Dw", "location": "线上", "name": "研发效能黄金三角及需求与敏捷协作领域实践", "start_time": 1658851200 }, { "end_time": 1658851199, "img": "https://static001.infoq.cn/resource/image/6c/5d/6cd448341929757b0904d9ac0da0705d.jpg", "link": "https://live.infoq.cn/room/1526", "location": "线上", "name": "分布式数据库在金融核心场景落地实践", "start_time": 1658764800 }, { "end_time": 1659283199, "img": "https://static001.infoq.cn/resource/image/46/ab/46d2c7e473b216df11592719f9f348ab.jpg", "link": "https://www.infoq.cn/event/?id=236", "location": "广州", "name": "腾讯自研业务大规模云原生实践专场", "start_time": 1659196800 }, { "end_time": 1658505599, "img": "https://static001.infoq.cn/resource/image/71/34/714e4b9387daf584544cbc202038fe34.png", "link": "https://www.aliyun.com/page-source/tianchi/promotion/RocketMQ_Summit2022", "location": "线上", "name": "RocketMQ Summit 全球开发者峰会", "start_time": 1658332800 }, { "end_time": 1657987199, "img": "https://static001.infoq.cn/resource/image/12/e4/128752cb4b0fe89564d9ed26737b9de4.jpg", "link": "https://www.infoq.cn/event/?id=233&utm_source=gw&sign=iq_62c28993c5b5d", "location": "深圳", "name": "企业云上管理与治理专场", "start_time": 1657900800 }, { "end_time": 1657814399, "img": "https://static001.infoq.cn/resource/image/4f/fa/4fb53f9599990098cc3590b3ea7c0afa.jpg", "link": "https://live.infoq.cn/room/1520", "location": "线上", "name": "新知·音视频技术公开课第五期", "start_time": 1657728000 }, { "end_time": 1660838399, "img": "https://static001.infoq.cn/resource/image/08/bf/080d6253808cb41011be166578yy4fbf.jpg", "link": "https://www.infoq.cn/event/?id=219", "location": "北京", "name": "企业云原生成本优化最佳实践专场", "start_time": 1660752000 }, { "end_time": 1660924799, "img": "https://static001.infoq.cn/resource/image/8a/53/8a46a5a1bc2fa426f6929673238b5a53.jpg", "link": "https://www.infoq.cn/event/?id=239", "location": "线上", "name": "腾讯云大数据峰会暨Techo TVP开发者峰会", "start_time": 1660838400 }, { "end_time": 1660147199, "img": "https://static001.infoq.cn/resource/image/0e/62/0eb00cbf52abb2186601f9e69e5d5c62.jpg", "link": "https://www.infoq.cn/event/?id=237", "location": "深圳", "name": "2022 OceanBase 年度发布会", "start_time": 1660060800 }, { "end_time": 1660233599, "img": "https://static001.infoq.cn/resource/image/9f/6f/9f328460d64d729e48370f21d054976f.jpg", "link": "https://www.infoq.cn/event/?id=238", "location": "深圳", "name": "2022阿里云飞天科技峰会", "start_time": 1660147200 }, { "end_time": 1660233599, "img": "https://static001.infoq.cn/resource/image/90/8e/90f1b4260fd376ea48edef3962dafe8e.png", "link": "https://xie.infoq.cn/article/01cd0581e3d7bae0f75027d0a", "location": "线上", "name": "ONES 研发管理大师课", "start_time": 1659974400 }, { "end_time": 1667145599, "img": "https://static001.infoq.cn/resource/image/yy/2e/yy63948fb2c360db81223155ffc7302e.jpg", "link": "https://www.infoq.cn/event/?id=246", "location": "北京", "name": "腾讯云原生最佳实践专场", "start_time": 1667059200 }, { "end_time": 1664207999, "img": "https://static001.infoq.cn/resource/image/6a/e5/6aeba4feyye7285e5a9230ed3eea3ae5.jpg", "link": "https://live.infoq.cn/room/1578", "location": "线上", "name": "自研数据库技术破局与最佳实践", "start_time": 1664121600 }, { "end_time": 1667923199, "img": "https://static001.infoq.cn/resource/image/df/5a/dff2c1276bac4b4f487ca0182a08d85a.jpg", "link": "https://live.infoq.cn/room/1605", "location": "线上", "name": "自研数据库技术破局与最佳实践", "start_time": 1667836800 }, { "end_time": 1669478399, "img": "https://static001.infoq.cn/resource/image/fb/18/fbf1f9cc7ec6ca06a3c17e182b6fb018.jpg", "link": "https://www.infoq.cn/event/?id=250", "location": "上海", "name": "实时音视频技术在抖音上的深度磨砺专场", "start_time": 1669392000 }, { "end_time": 1669391999, "img": "https://static001.infoq.cn/resource/image/83/e8/83235dc18c07yy5b20bb1d67f4d525e8.jpg", "link": "https://www.infoq.cn/event/?id=249&utm_source=hsss&sign=iq_637336d57aa28", "location": "上海", "name": "腾讯自研业务大规模云原生实践专场", "start_time": 1669305600 }, { "end_time": 1668959999, "img": "https://static001.infoq.cn/resource/image/a0/cf/a0bf8117a778385df8744669ca0d67cf.jpg", "link": "https://www.infoq.cn/form/?id=1277&utm_source=web&utm_medium=hd&sign=iq_636a1a0e02ff2", "location": "线上", "name": "Pulsar Summit Asia 2022", "start_time": 1668787200 }, { "end_time": 1669391999, "img": "https://static001.infoq.cn/resource/image/22/97/22f0606ea24861e118b3edyyc6d31c97.jpg", "link": "https://www.infoq.cn/form/?id=1278&utm_source=gw&sign=iq_636e197618943", "location": "上海·宏安瑞士大酒店", "name": "QCon上海-网易智企专场", "start_time": 1669305600 }, { "end_time": 1669478400, "img": "https://static001.infoq.cn/resource/image/58/67/586759c3451398bca513265fe8301467.jpg", "link": "https://developer.aliyun.com/special/ffa2022", "location": "线上", "name": "Flink Forward Asia 2022", "start_time": 1669392000 }, { "end_time": 1666886399, "img": "https://static001.infoq.cn/resource/image/29/0e/294354bf979912e69cda609a4441f70e.jpg", "link": "https://www.infoq.cn/event/?id=245", "location": "线上", "name": "数字化转型第一步，“低代码”搭建更靠谱！", "start_time": 1666800000 }, { "end_time": 1667145599, "img": "https://static001.infoq.cn/resource/image/d6/da/d6441fa0109e3efafb44eb60974066da.jpg", "link": "https://www.infoq.cn/event/?id=244", "location": "线上直播", "name": "云无界，端无边 OGeek 技术峰会", "start_time": 1666972800 }, { "end_time": 1669910399, "img": "https://static001.infoq.cn/resource/image/70/4b/70577884a67ee322204149b85a024b4b.jpg", "link": "https://live.infoq.cn/room/1641", "location": "线上", "name": "DBTalk 第五期", "start_time": 1669824000 }, { "end_time": 1671638399, "img": "https://static001.infoq.cn/resource/image/dd/12/ddc5793176743b217528e2a26c5eaf12.jpg", "link": "https://live.infoq.cn/room/1653", "location": "线上", "name": "高科制造类项目研发如何软硬协同、敏捷提效", "start_time": 1671552000 }, { "end_time": 1670774399, "img": "https://static001.infoq.cn/resource/image/6a/a7/6af9a7694971c586048a004b124345a7.jpg", "link": "https://www.infoq.cn/minibook/RuveM045ptfTNKwfeq6c?id=1326&utm_source=gw&sign=iq_638957ce4f3dd", "location": "线上", "name": "腾讯技术实践精选集", "start_time": 1669910400 }, { "end_time": 1672329599, "img": "https://static001.infoq.cn/resource/image/0d/54/0dc4c07fd11f4e94385e50afdb543354.jpg", "link": "https://live.infoq.cn/room/1672", "location": "线上", "name": "DBTalk第六期", "start_time": 1671984000 }, { "end_time": 1675612799, "img": "https://static001.infoq.cn/resource/image/19/05/19407a3b3bb0c5d681a69c82f712ac05.png", "link": "https://www.infoq.cn/event/?id=255&utm_source=gw&sign=iq_63d7671a45d19", "location": "线上", "name": "QCon北京-现代数据栈建设及思考", "start_time": 1675008000 }, { "end_time": 1675612799, "img": "https://static001.infoq.cn/resource/image/b5/4d/b52de3a6006366d6fc220c16b5ac784d.jpg", "link": "https://www.infoq.cn/event/?id=254&utm_source=gwg&sign=iq_63d60f639f272", "location": "线上", "name": "QCon北京-腾讯云专场@董家冠", "start_time": 1675008000 }, { "end_time": 1672156799, "img": "https://static001.infoq.cn/resource/image/52/b4/52f1e97e4c21c87186e5e1d3yyedbdb4.jpg", "link": "https://www.infoq.cn/event/?id=253", "location": "线上", "name": "2022 云智峰会·智算峰会", "start_time": 1672070400 }, { "end_time": 1671984000, "img": "https://static001.infoq.cn/resource/image/1b/2f/1b7eaa4097c90b3fe8657227a600172f.jpg", "link": "https://www.infoq.cn/form/?id=1328&utm_source=activity&sign=iq_6389a90914ca9", "location": "北京、线上", "name": "百度百舸·AI 异构计算平台", "start_time": 1669910400 }, { "end_time": 1680191999, "img": "https://static001.infoq.cn/resource/image/3b/58/3bf1e0124aa9e80df9f4b274aae5db58.jpg", "link": "https://www.infoq.cn/event/?id=261", "location": "线上", "name": "腾讯云直播课安全篇", "start_time": 1680105600 }, { "end_time": 1678550399, "img": "https://static001.infoq.cn/resource/image/b9/87/b9c76bd730d5bbd2f326cafbf1bb5787.jpg", "link": "https://www.infoq.cn/event/?id=248", "location": "北京", "name": "OGeek 小布沙龙", "start_time": 1678464000 }, { "end_time": 1679673599, "img": "https://static001.infoq.cn/resource/image/31/dd/317230139b0dbe53e09115a73ec572dd.jpg", "link": "https://www.infoq.cn/event/?id=259&utm_source=hddby&sign=iq_640fd4ea03e7f", "location": "北京", "name": "阿里云瑶池数据库峰会 • 北京", "start_time": 1679587200 }, { "end_time": 1679759999, "img": "https://static001.infoq.cn/resource/image/3a/66/3a6aac32e153f0037132bb067a0ccd66.jpg", "link": "https://www.oceanbase.com/devcon2023?utm_source=infoq&utm_medium=pc&utm_campaign=dc2023&utm_term=homebanner", "location": "北京", "name": "OceanBase开发者大会 · 2023", "start_time": 1679673600 }, { "end_time": 1682006399, "img": "https://static001.infoq.cn/resource/image/d2/e3/d223bc5b37623329d614ce98408899e3.png", "link": "https://xie.infoq.cn/article/a2bb8290ef4e9ba436e04dbdc", "location": "线上", "name": "《中国企业软件研发管理白皮书》", "start_time": 1681920000 }, { "end_time": 1687881600, "img": "https://static001.infoq.cn/resource/image/1a/41/1a797cc09be8f79a58041b5c944be641.jpg", "link": "https://summit.awsevents.cn/2023/form.html?source=A1Jmo+hFUc3p2QU//2KWmu/axcY5zRDRzaPxMrZTjIU=", "location": "上海", "name": "2023亚马逊云科技中国峰会··", "start_time": 1687795200 }, { "end_time": 1687881600, "img": "https://static001.infoq.cn/resource/image/1a/41/1a797cc09be8f79a58041b5c944be641.jpg", "link": "https://summit.awsevents.cn/2023/form.html?source=A1Jmo+hFUc3p2QU//2KWmu/axcY5zRDRzaPxMrZTjIU=", "location": "上海", "name": "2023亚马逊云科技中国峰会", "start_time": 1687795200 }, { "end_time": 1687017599, "img": "https://static001.infoq.cn/resource/image/55/7c/55cd06b294721f7e8759579f2eff707c.jpg", "link": "https://static001.geekbang.org/resource/image/9c/3c/9c24570a3cdaea6f9f1db8b28a3a493c.jpg", "location": "线上", "name": "《再谈数据架构》第一期", "start_time": 1686931200 }, { "end_time": 1689091199, "img": "https://static001.infoq.cn/resource/image/2a/d0/2a465b70eaeea5aba10c4e6c3bbb73d0.png", "link": "https://static001.geekbang.org/resource/image/54/cc/54d7f1049cdf37e4ae1835ece16bbacc.jpg", "location": "线上", "name": "数据引擎，One Size Fits All真的能实现么", "start_time": 1689004800 }, { "end_time": 1690300799, "img": "https://static001.infoq.cn/resource/image/fb/2b/fbfba6f97a6e6c90218d35394daf912b.jpg", "link": "https://static001.geekbang.org/resource/image/f2/8a/f2b2a8cac897dc23a95830ba87999a8a.jpg", "location": "线上", "name": "云器科技产品发布会", "start_time": 1690214400 }, { "end_time": 1690819199, "img": "https://static001.infoq.cn/resource/image/63/21/63afb05ba699b993ae1899b6ffbfb921.png", "link": "https://live.infoq.cn/room/1838", "location": "线上", "name": "阿里云智能飞天发布会", "start_time": 1690732800 }, { "end_time": 1685289599, "img": "https://static001.infoq.cn/resource/image/7f/2c/7f66b1f3f7aa26acea890d216b57122c.jpg", "link": "https://dev.amazoncloud.cn/activity/activityDetail?id=645a17374b2abb2a75068e76&catagoryName=communityDay&sc_medium=owned&sc_campaign=Communityday&sc_channel=InfoQ1", "location": "深圳", "name": "亚马逊云科技 Community Day 深圳站", "start_time": 1685203200 }, { "end_time": 1693929599, "img": "https://static001.infoq.cn/resource/image/9d/13/9d6b0bac0dba84097c51a836ef023a13.jpg", "link": "https://cloud.baidu.com/summit/AIcloudsummit_2023/index.html?track=infoQ", "location": "北京", "name": "2023百度云智大会", "start_time": 1693843200 }, { "end_time": 1695311999, "img": "https://static001.infoq.cn/resource/image/e2/be/e2c9728001413b208e0903eee022c0be.png", "link": "https://www.infoq.cn/form/?id=1763&utm_source=hd&sign=iq_64e85d7f35e01", "location": "上海", "name": "Akamai World Tour 2023 上海站", "start_time": 1695225600 }, { "end_time": 1694102400, "img": "https://static001.infoq.cn/resource/image/1a/c5/1a8da5aef7d757e6cc72ff8300f919c5.jpg", "link": "https://qdrl.qq.com/T9OAntJY", "location": "深圳", "name": "2023腾讯全球数字生态大会", "start_time": 1694016000 }, { "end_time": 1694534399, "img": "https://static001.infoq.cn/resource/image/52/0d/52e53bbd519a86yy9201ab96622c1f0d.png", "link": "https://live.infoq.cn/room/1864?utm_source=infoq-web&utm_medium=hdbanner", "location": "线上", "name": "提质增效,数智升级,泛娱企业的研发进化之路", "start_time": 1694448000 }, { "end_time": 1698335999, "img": "https://static001.infoq.cn/resource/image/b4/9e/b41e3ebb3b835f8745b5bcfc8fb3469e.jpg", "link": "https://www.infoq.cn/event/?id=274&utm_source=123&sign=iq_652e5af976d54", "location": "线上", "name": "如何基于AIGC 技术快速开发应用，助力企业", "start_time": 1698249600 }, { "end_time": 1701273600, "img": "https://static001.infoq.cn/resource/image/96/e1/96c324dd2e3904e8c88e8e9c9760c4e1.jpg", "link": "https://cloud.baidu.com/product/wenxinworkshop?track=InfoQ", "location": "线上", "name": "百度文心大模型4.0", "start_time": 1697817600 }, { "end_time": 1701359999, "img": "https://static001.infoq.cn/resource/image/4a/84/4a99774d470387882bc7808d7yy78384.png", "link": "https://www.infoq.cn/minibook/JrzLW0G9ZMZlppDXmWWi?id=1885&utm_source=gw&sign=iq_653b7ed70c95a", "location": "线上", "name": "分布式云行业实践指南", "start_time": 1698595200 }, { "end_time": 1698163199, "img": "https://static001.infoq.cn/resource/image/76/5a/76fe09bdc8c6b013128bb4bc282e2f5a.jpg", "link": "https://ct.ctrip.com/activitysetupapp/mkt/index/sigfh?popup=close&type=dynamic", "location": "线上", "name": "B+携程商旅伙伴大会", "start_time": 1698076800 }, { "end_time": 1703952000, "img": "https://static001.infoq.cn/resource/image/bc/3c/bc3yy75739ef34b57f50f3d48c8f0e3c.png", "link": "https://www.infoq.cn/event/?id=275", "location": "线上", "name": "2023年OceanBase发布会", "start_time": 1699200000 }, { "end_time": 1704038399, "img": "https://static001.infoq.cn/resource/image/c4/3a/c4e71f32aa47ebe48ffe35e421066e3a.png", "link": "https://xie.infoq.cn/article/65f0fe58bfe5a2bd828a234b5", "location": "线上", "name": "技术人的 2023 总结｜有奖征文大赛来袭！", "start_time": 1701360000 }, { "end_time": 1703087999, "img": "https://static001.infoq.cn/resource/image/be/9f/bea0ee740a82595085eb0b2bbec4519f.jpg", "link": "https://cloud.baidu.com/summit/aicomputing_2023/index.html?track=mkt_InfoQ", "location": "线下", "name": "2023百度云智大会•智算大会", "start_time": 1701273600 }, { "end_time": 1703001600, "img": "https://static001.infoq.cn/resource/image/08/e1/08e97acc6b8d066ffc0c021cc4995ae1.jpg", "link": "https://www.infoq.cn/form/?id=1953&utm_source=wz01&sign=iq_65698a46f3a5d", "location": "线下", "name": "2023百度云智大会•智算大会开发者沙龙", "start_time": 1701360000 }, { "end_time": 1702828800, "img": "https://static001.infoq.cn/resource/image/4c/02/4ce5e4234d3bdab6ce7f765cc5e9d502.jpg", "link": "https://openanolis.cn/openanolisconference", "location": "北京", "name": "2023 龙蜥操作系统大会", "start_time": 1702742400 }, { "end_time": 1703088000, "img": "https://static001.infoq.cn/resource/image/4f/e5/4f037904e16c000b17e3f2e9d73ba3e5.png", "link": "https://www.infoq.cn/form/?id=1979&utm_source=1&sign=iq_657961be4a74fhttps://www.infoq.cn/form/?id=1979&utm_source=1&sign=iq_657961be4a74f", "location": "线上", "name": "百度 Comate公开课", "start_time": 1702396800 }, { "end_time": 1703952000, "img": "https://static001.infoq.cn/resource/image/62/54/62668b5abed7ac0c7b0afcc1a3b1c254.jpg", "link": "https://www.huodongxing.com/event/2730575053800?td=2144095838345", "location": "北京", "name": "FFA大会&闭门会", "start_time": 1700668800 }, { "end_time": 1702137599, "img": "https://static001.infoq.cn/resource/image/46/04/46b43918ed8ff11713bcf357e23b6504.jpg", "link": "https://flink-forward.org.cn", "location": "北京", "name": "Flink Forward 2023", "start_time": 1701964800 } ], "notice": { "content": "重磅揭晓｜56 个项目上榜 2023 年度数字化践行者力量榜", "end_time": 1704038399, "link": "https://www.infoq.cn/zones/digitalpioneerawards2023/", "start_time": 1703829600, "time_now": 1703839824 }, "top": [ { "alias": " industrynews", "child_list": [ { "id": 1164, "name": "英特尔", "alias": "", "type": 1 }, { "id": 19, "name": "Google", "alias": "google", "type": 1 }, { "id": 20, "name": "微软", "alias": "Microsoft", "type": 1 }, { "id": 26, "name": "亚马逊云科技", "alias": "AWS", "type": 1 }, { "id": 54, "name": "百度", "alias": "baidu", "type": 1 }, { "id": 61, "name": "阿里巴巴", "alias": "alibaba", "type": 1 }, { "id": 78, "name": "腾讯", "alias": "tencent", "type": 1 }, { "id": 145, "name": "华为", "alias": "huawei", "type": 1 }, { "id": 1167, "name": "字节跳动", "alias": "", "type": 1 }, { "id": 1215, "name": "用友", "alias": "", "type": 1 } ], "icon": "", "id": 147, "link": "", "name": "企业动态", "type": 1 }, { "alias": "", "child_list": [ { "id": 160, "name": "医疗", "alias": "medical", "type": 1 }, { "id": 1169, "name": "银行", "alias": "", "type": 1 }, { "id": 1170, "name": "证券", "alias": "", "type": 1 }, { "id": 1171, "name": "保险", "alias": "", "type": 1 }, { "id": 1172, "name": "汽车", "alias": "", "type": 1 }, { "id": 1173, "name": "工业", "alias": "", "type": 1 } ], "icon": "", "id": 1168, "link": "", "name": "行业深度", "type": 1 }, { "alias": "", "child_list": [ { "id": 141, "name": "小程序", "alias": "applets", "type": 1 }, { "id": 1209, "name": "Android/iOS", "alias": "", "type": 1 }, { "id": 1210, "name": "架构/框架", "alias": "", "type": 1 }, { "id": 1211, "name": "工程化", "alias": "", "type": 1 }, { "id": 1212, "name": "跨端开发", "alias": "", "type": 1 }, { "id": 1213, "name": "性能优化", "alias": "", "type": 1 }, { "id": 1214, "name": "音视频（前端）", "alias": "", "type": 1 } ], "icon": "", "id": 1208, "link": "", "name": "大前端", "type": 1 }, { "alias": "", "child_list": [ { "id": 29, "name": "数据库", "alias": "Database", "type": 1 }, { "id": 32, "name": "编程语言", "alias": "programing-languages", "type": 1 }, { "id": 81, "name": "区块链", "alias": "Blockchain", "type": 1 }, { "id": 1175, "name": "中间件", "alias": "", "type": 1 }, { "id": 1176, "name": "操作系统", "alias": "", "type": 1 }, { "id": 1177, "name": "5G/IoT", "alias": "", "type": 1 }, { "id": 1178, "name": "音视频（后端）", "alias": "", "type": 1 } ], "icon": "", "id": 1174, "link": "", "name": "后端", "type": 1 }, { "alias": "architecture", "child_list": [ { "id": 12, "name": "SOA", "alias": "soa", "type": 1 }, { "id": 70, "name": "微服务", "alias": "microservice", "type": 1 }, { "id": 1179, "name": "业务架构", "alias": "", "type": 1 }, { "id": 1180, "name": "框架", "alias": "", "type": 1 }, { "id": 1181, "name": "领域驱动设计", "alias": "", "type": 1 }, { "id": 1182, "name": "多云/混合云", "alias": "", "type": 1 } ], "icon": "", "id": 8, "link": "", "name": "架构", "type": 1 }, { "alias": "AI", "child_list": [ { "id": 125, "name": "自动驾驶", "alias": "Autopilot", "type": 1 }, { "id": 127, "name": "计算机视觉", "alias": " computing-vision", "type": 1 }, { "id": 1183, "name": "生成式 AI", "alias": "", "type": 1 }, { "id": 1184, "name": "AI 工程化", "alias": "", "type": 1 }, { "id": 1185, "name": "机器学习/深度学习", "alias": "", "type": 1 }, { "id": 1186, "name": "自然语言处理", "alias": "", "type": 1 }, { "id": 1187, "name": "芯片", "alias": "", "type": 1 } ], "icon": "", "id": 31, "link": "", "name": "AI", "type": 1 }, { "alias": "bigdata", "child_list": [ { "id": 1188, "name": "DataOps", "alias": "", "type": 1 }, { "id": 1189, "name": "数据集成", "alias": "", "type": 1 }, { "id": 1190, "name": "DataMesh", "alias": "", "type": 1 }, { "id": 1191, "name": "数据编织", "alias": "", "type": 1 }, { "id": 1192, "name": "数据湖仓", "alias": "", "type": 1 }, { "id": 1193, "name": "在离线混部", "alias": "", "type": 1 }, { "id": 1194, "name": "实时计算", "alias": "", "type": 1 } ], "icon": "", "id": 15, "link": "", "name": "大数据", "type": 1 }, { "alias": "", "child_list": [ { "id": 21, "name": "安全", "alias": "Security", "type": 1 }, { "id": 120, "name": "AIOps", "alias": "AIOps", "type": 1 }, { "id": 1165, "name": "BizDevOps", "alias": "BizDevOps", "type": 1 }, { "id": 1196, "name": "DevOps & 平台工程", "alias": "", "type": 1 }, { "id": 1197, "name": "FinOps", "alias": "", "type": 1 }, { "id": 1198, "name": "研发效能", "alias": "", "type": 1 }, { "id": 1199, "name": "可观测", "alias": "", "type": 1 } ], "icon": "", "id": 1195, "link": "", "name": "软件工程", "type": 1 }, { "alias": "cloud-computing", "child_list": [ { "id": 106, "name": "云原生", "alias": "CloudNative", "type": 1 }, { "id": 119, "name": "Serverless", "alias": "Serverless", "type": 1 }, { "id": 1200, "name": "服务革新", "alias": "", "type": 1 }, { "id": 1201, "name": "云安全", "alias": "", "type": 1 }, { "id": 1202, "name": "云端开发", "alias": "", "type": 1 }, { "id": 1203, "name": "芯片与网络", "alias": "", "type": 1 }, { "id": 1216, "name": "低代码", "alias": "", "type": 1 } ], "icon": "", "id": 11, "link": "", "name": "云计算", "type": 1 }, { "alias": "", "child_list": [ { "id": 17, "name": "开源", "alias": "opensource", "type": 1 }, { "id": 1131, "name": "技术选型", "alias": "technology-selection", "type": 1 }, { "id": 1160, "name": "数字化转型", "alias": "", "type": 1 }, { "id": 1205, "name": "数字人才培养", "alias": "", "type": 1 }, { "id": 1206, "name": "团队搭建", "alias": "", "type": 1 }, { "id": 1207, "name": "技术出海", "alias": "", "type": 1 } ], "icon": "", "id": 1204, "link": "", "name": "管理/文化", "type": 1 }, { "child_list": [ ], "icon": "", "id": 0, "link": "https://www.infoq.cn/theme/207", "name": "AI 编程助手", "type": 2 }, { "child_list": [ ], "icon": "https://static001.infoq.cn/resource/image/90/73/906da8288da450f8732be47ad0120073.png", "id": 0, "link": "https://www.infoq.cn/zones/business/?utm_source=infoq-web&utm_medium=header_graybar&utm_term=0126", "name": "企业会员", "type": 3 }, { "child_list": [ ], "icon": "https://static001.infoq.cn/resource/image/52/fa/5248de588832bcf63b5a9570caa2b0fa.png", "id": 0, "link": "https://www.infoq.cn/article/CHFMEwL6dcXxxpqx7Avc?utm_source=infoq&utm_medium=header_graybar", "name": "访谈录报名", "type": 3 } ] }, "error": { }, "extra": { "cost": 0.0107395, "request-id": "dc4fbc8ee7414158948b1e6cbe9c1287@3@xie" } }

{ "code": 0, "data": { "id": 153, "name": "mapreduce", "sub_count": 0, "cite_count": 47, "is_sub": false, "relation_list": [ ] }, "error": { }, "extra": { "cost": 0.011949794, "request-id": "569e6b38d8ba43f9a321b98cb2f394ca@3@xie" } }

{ "code": 0, "data": { "author_list": [ { "uid": 1078842, "nickname": "后台技术汇", "avatar": "https://static001.geekbang.org/account/avatar/00/10/76/3a/8a839b9b.jpg", "active": 1, "is_early": 3, "ucode": "881B046DA8EEA9", "uri": "/u/bryant/", "author_type": 1, "vip": 0, "mood": "InfoQ签约作者、腾讯云社区优秀创作者", "intro": "", "is_sub": false }, { "uid": 2437020, "nickname": "三掌柜", "avatar": "https://static001.geekbang.org/account/avatar/00/25/2f/9c/5be8d796.jpg", "active": 1, "is_early": 3, "ucode": "17BE7513EE0682", "uri": "/profile/17BE7513EE0682/", "author_type": 1, "vip": 0, "mood": "某某某技术有限责任公司架构师", "intro": "", "is_sub": false }, { "uid": 3620241, "nickname": "于仔学技术", "avatar": "https://static001.geekbang.org/account/avatar/00/37/3d/91/741963ca.jpg", "active": 1, "is_early": 0, "ucode": "D7D8D623835A7F", "uri": "/profile/D7D8D623835A7F/", "author_type": 1, "vip": 0, "mood": "行动就在当下。", "intro": "", "is_sub": false }, { "uid": 1057792, "nickname": "架构精进之路", "avatar": "https://static001.geekbang.org/account/avatar/00/10/24/00/26431acf.jpg", "active": 1, "is_early": 3, "ucode": "D43D575DAC4243", "uri": "/u/flyer01126/", "author_type": 2, "vip": 0, "mood": "🏆 InfoQ写作平台-签约作者 🏆", "intro": "", "is_sub": false }, { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0, "mood": "提供全面深入的云计算技术干货", "intro": "", "is_sub": false } ], "label_list": [ { "id": 34432, "name": "我和极客时间的故事" }, { "id": 8, "name": "Python" }, { "id": 211, "name": "开源" }, { "id": 3, "name": "Java" }, { "id": 264, "name": "个人成长" }, { "id": 238, "name": "架构" }, { "id": 201, "name": "读书笔记" }, { "id": 221, "name": "学习方法" }, { "id": 232, "name": "程序员" }, { "id": 97, "name": "redis" }, { "id": 29545, "name": "热门推荐" }, { "id": 18167, "name": "热门活动" }, { "id": 30432, "name": "CodeWhisperer" }, { "id": 19388, "name": "亚马逊云科技" }, { "id": 192, "name": "创业" } ], "month_author": { "list": [ ], "title": "" }, "square_list": [ { "id": 13, "index_desc": "Q&A", "index_icon": "https://static001.infoq.cn/resource/image/88/9e/88ab2e218d41b9360ccdb95a8bac779e.png", "index_name": "玩转社区", "type": 2, "url": "" }, { "id": 8, "index_desc": "精选文章", "index_icon": "https://static001.infoq.cn/resource/image/3b/01/3bc62710e82ccbec4f1ce6cf53524001.png", "index_name": "优质合集", "type": 2, "url": "" }, { "id": 14, "index_desc": "行业技术资讯", "index_icon": "https://static001.infoq.cn/resource/image/6c/c1/6ca78714488964cdab94a50c04e033c1.png", "index_name": "企业号社区", "type": 2, "url": "" } ] }, "error": { }, "extra": { "cost": 0.005660267, "request-id": "9d13c404be8641b883fbdd5880365e94@3@xie" } }

{ "code": 0, "data": { "list": [ { "id": 195745, "aid": "594838807456329730", "aidint": 594838807456329700, "type": 1, "views": 0, "uuid": "7057235bc2e6c7d3eebe1cc9e", "status": 1, "publish_time": 1666620161143, "ctime": 1666620161252, "utime": 1666620161252, "vid": "", "source": 2, "sub_status": 0, "ip_address": "北京", "score": 1666620161143, "article_title": "MapReduce作业生命周期", "article_sharetitle": "MapReduce作业生命周期", "article_subtitle": "", "article_summary": "MapReduce作业作为一种分布式应用程序，可直接运行在Hadoop资源管理系统YARN之上（MapReduce On YARN）。每个MapReduce应用程序由一个MRAppMaster以及一系列MapTask和ReduceTask构成，它们通过ResourceManager获得资源，并由NodeManager启动运行。", "article_cover": "https://static001.geekbang.org/infoq/16/16ade6bf7562e17e3f6aedbfd0e01283.jpeg", "article_cover_point": "", "author": [ { "uid": 1334958, "nickname": "穿过生命散发芬芳", "avatar": "https://static001.geekbang.org/account/avatar/00/14/5e/ae/9b57b3b4.jpg", "active": 1, "is_early": 3, "ucode": "24E2E30D6BC39A", "uri": "/profile/24E2E30D6BC39A/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" }, { "id": 24058, "name": "10月月更" } ], "out_id": null, "word_count": 966, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MapReduce作业作为一种分布式应用程序，可直接运行在Hadoop资源管理系统YARN之上（MapReduce On YARN）。每个MapReduce应用程序由一个MRAppMaster以及一系列MapTask和ReduceTask构成，它们通过ResourceManager获得资源，并由NodeManager启动运行。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/af/afb99c723d523a4ca8df69634d2cf1a2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当用户向YARN中提交一个MapReduce应用程序后，YARN将分两个阶段运行该应用程序：第一个阶段是由ResourceManager启动MRAppMaster；第二个阶段是由MRAppMaster创建应用程序，为它申请资源，并监控它的整个运行过程，直到运行成功。如图YARN的工作流程分为以下几个步骤：","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/73/73a4f800d4478f2a345711709d07e854.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1）用户向YARN集群提交应用程序，该应用程序包括以下配置信息：MRAppMaster所在jar包、启动MRAppMaster的命令及其资源需求（CPU、内存等）、用户程序jar包等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 167285, "aid": "594797420382642176", "aidint": 594797420382642200, "type": 1, "views": 0, "uuid": "f40f637ec067337601ffcb8c3", "status": 1, "publish_time": 1656515894205, "ctime": 1656515895443, "utime": 1656737477060, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1656515894205, "article_title": "在Hadoop环境里面统计西游记文章的词组（hdfs实验）", "article_sharetitle": "在Hadoop环境里面统计西游记文章的词组（hdfs实验）", "article_subtitle": "", "article_summary": "在Hadoop的环境里面我们测试一个文件，对西游记这个书籍里面的所有词语，进行统计排序。hadoop包括两个核心组成：HDFS：分布式文件系统，存储海量的数据MapReduce：并行处理框架，实现任务分解和调度。", "article_cover": "https://static001.geekbang.org/infoq/9f/9f79347afebc4259d61717f2ab089eee.gif", "article_cover_point": "", "author": [ { "uid": 2535339, "nickname": "王小王-123", "avatar": "https://static001.geekbang.org/account/avatar/00/26/af/ab/ac687112.jpg", "active": 1, "is_early": 0, "ucode": "FD6BDBBCDC7C4F", "uri": "/u/wxw-123/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" }, { "id": 21689, "name": "hadoop统计词频" }, { "id": 21690, "name": "hadoop项目" }, { "id": 21710, "name": "文本统计分析" } ], "out_id": null, "word_count": 1429, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"在Hadoop的环境里面我们测试一个文件，对西游记这个书籍里面的所有词语，进行统计排序。hadoop包括两个核心组成：HDFS：分布式文件系统，存储海量的数据MapReduce：并行处理框架，实现任务分解和调度。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"hadoop运行机制","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"MapReduce工作原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/45/4509bc5403494ad422adaf7269bfb548.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"首先我们准备四个文件，在自己Windows里面已经写好的，或者是从其他地方copy过来的（涉及一些参数和路径的修改）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"点击下载这四个参考文件前面两文件是实现我们的文章词组的分割处理和排序处理（Python文件），后面的一个是执行文件，最后一个是西游记里面的文章，这个里面已经利用pycharm的jieba 库做好的文字的处理，只有词组和空格，没有标点符号和其他的东西。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/af/af91b16f55f7ecd6491bf3f05bce231e.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"准备：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"首先打开虚拟机，启动Hadoop","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/88/88113fb7d0d811cae1c05a9ec8449627.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这样就启动成功了，有的时候我们很容易忽略这个，我的这台机器配置了，开机自启动，所以打开虚拟机就会自己启动，但是如果没有的话我们就需要利用下面的代码","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"bash"},"content":[{"type":"text","text":"start-all.sh\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之后等待启动OK，再去查看即可","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"实验过程","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"思路：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 156287, "aid": "594775694752346120", "aidint": 594775694752346100, "type": 1, "views": 0, "uuid": "c27ca6260e22cc3f42ad1d696", "status": 1, "publish_time": 1651211786292, "ctime": 1651211786400, "utime": 1651211786400, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1651211786292, "article_title": "开发一个不需要重写成Hive QL的大数据SQL引擎", "article_sharetitle": "开发一个不需要重写成Hive QL的大数据SQL引擎", "article_subtitle": "", "article_summary": "开发一款能支持标准数据库SQL的大数据仓库引擎，让那些在Oracle上运行良好的SQL可以直接运行在Hadoop上，而不需要重写成Hive QL。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 95, "name": "sql" }, { "id": 143, "name": "大数据" }, { "id": 153, "name": "mapreduce" }, { "id": 19877, "name": "SQL引擎" }, { "id": 19878, "name": "大数据仓库" } ], "out_id": null, "word_count": 4137, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区《","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/349763?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=other&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"从零开发大数据SQL引擎","attrs":{}}]},{"type":"text","text":"》，作者：JavaEdge 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"学习大数据技术的核心原理，掌握一些高效的思考和思维方式，构建自己的技术知识体系。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"明白了原理，有时甚至不需要学习，顺着原理就可以推导出各种实现细节。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"各种知识表象看杂乱无章，若只是学习繁杂知识点，固然自己的知识面是有限的，并且遇到问题的应变能力也很难提高。所以有些高手看起来似乎无所不知，不论谈论起什么技术，都能头头是道，其实并不是他们学习、掌握了所有技术，而是他们是在谈到这个问题时，才开始进行推导，并迅速得出结论。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"高手不一定要很资深、经验丰富，把握住技术的核心本质，掌握快速分析推导的能力，能迅速将自己的知识技能推到陌生领域，就是高手。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本系列专注大数据开发需要关注的问题及解决方案。跳出繁杂知识表象，掌握核心原理和思维方式，进而融会贯通各种技术，再通过各种实践训练，成为终极高手。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"大数据仓库Hive","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作为一个成功的大数据仓库，它将SQL语句转换成MapReduce执行过程，并把大数据应用的门槛下降到普通数据分析师和工程师就可以很快上手的地步。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 154573, "aid": "594773228921331715", "aidint": 594773228921331700, "type": 1, "views": 0, "uuid": "ada38ee0e1ce90143ff3ded0c", "status": 1, "publish_time": 1650609776769, "ctime": 1650609776875, "utime": 1650609776875, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1650609776769, "article_title": "带你了解极具弹性的Spark架构的原理", "article_sharetitle": "带你了解极具弹性的Spark架构的原理", "article_subtitle": "", "article_summary": "相比MapReduce僵化的Map与Reduce分阶段计算相比，Spark的计算框架更加富有弹性和灵活性，运行性能更佳。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 151, "name": "spark" }, { "id": 153, "name": "mapreduce" }, { "id": 1229, "name": "内存" }, { "id": 4462, "name": "RDD" }, { "id": 19599, "name": "Spark架构" } ], "out_id": null, "word_count": 2474, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区《","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/348639?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=other&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"Spark架构原理","attrs":{}}]},{"type":"text","text":"》，作者：JavaEdge。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"相比MapReduce僵化的Map与Reduce分阶段计算相比，Spark的计算框架更加富有弹性和灵活性，运行性能更佳。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1 Spark的计算阶段","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MapReduce一个应用一次只运行一个map和一个reduce","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Spark可根据应用的复杂度，分割成更多的计算阶段（stage），组成一个有向无环图DAG，Spark任务调度器可根据DAG的依赖关系执行计算阶段","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"逻辑回归机器学习性能Spark比MapReduce快100多倍。因为某些机器学习算法可能需要进行大量迭代计算，产生数万个计算阶段，这些计算阶段在一个应用中处理完成，而不是像MapReduce那样需要启动数万个应用，因此运行效率极高。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DAG，有向无环图，不同阶段的依赖关系是有向的，计算过程只能沿依赖关系方向执行，被依赖的阶段执行完成前，依赖的阶段不能开始执行。该依赖关系不能有环形依赖，否则就死循环。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"典型的Spark运行DAG的不同阶段：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/resource/image/c8/db/c8cf515c664b478e51058565e0d4a8db.png","alt":null,"title":"","style":[{"key":"width","value":"50%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个应用被切分成3个阶段，阶段3需要依赖阶段1、2，阶段1、2互不依赖。Spark执行调度时，先执行阶段1、2，完成后，再执行阶段3。对应Spark伪代码：","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"rddB = rddA.groupBy(key)\nrddD = rddC.map(func)\nrddF = rddD.union(rddE)\nrddG = rddB.join(rddF)\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以Spark作业调度执行的核心是DAG，整个应用被切分成数个阶段，每个阶段的依赖关系也很清楚。根据每个阶段要处理的数据量生成任务集合（TaskSet），每个任务都分配一个任务进程去处理，Spark就实现了大数据的分布式计算。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 153755, "aid": "594772175459348480", "aidint": 594772175459348500, "type": 1, "views": 0, "uuid": "dab9e1de813ff82832ff93d75", "status": 1, "publish_time": 1650352583976, "ctime": 1650352584082, "utime": 1650352584082, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1650352583976, "article_title": "不care工具，在大数据平台中Hive能自动处理SQL", "article_sharetitle": "不care工具，在大数据平台中Hive能自动处理SQL", "article_subtitle": "", "article_summary": "有没有更简单的办法，可以直接将SQL运行在大数据平台？", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 95, "name": "sql" }, { "id": 143, "name": "大数据" }, { "id": 153, "name": "mapreduce" }, { "id": 155, "name": "hive" }, { "id": 326, "name": "数据分析" } ], "out_id": null, "word_count": 2434, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区《","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/348195?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=other&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"Hive执行原理","attrs":{}}]},{"type":"text","text":"》，作者： JavaEdge 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MapReduce简化了大数据编程的难度，使得大数据计算不再是高不可攀的技术圣殿，普通工程师也能使用MapReduce开发大数据程序。但是对于经常需要进行大数据计算的人，比如从事研究商业智能（BI）的数据分析师来说，他们通常使用SQL进行大数据分析和统计，MapReduce编程还是有一定的门槛。而且如果每次统计和分析都开发相应的MapReduce程序，成本也确实太高了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有没有更简单的办法，可以直接将SQL运行在大数据平台？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"先看如何用MapReduce实现SQL数据分析。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"MapReduce实现SQL的原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"常见的一条SQL分析语句，MapReduce如何编程实现？","attrs":{}}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":" SELECT pageid, age, count(1) FROM pv_users GROUP BY pageid, age;","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"统计分析语句，统计不同年龄用户访问不同网页的兴趣偏好，具体数据输入和执行结果：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/resource/image/0a/37/0ade10e49216575962e071d6fe9a7137.jpg","alt":"img","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"左边，要分析的数据表","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"右边，分析结果","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"把左表相同的行求和，即得右表，类似WordCount计算。该SQL的MapReduce的计算过程，按MapReduce编程模型","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"map函数的输入K和V，主要看V","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"V就是左表中每行的数据，如<1, 25>","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"map函数的输出就是以输入的V作为K，V统一设为1","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"比如<<1, 25>, 1>","attrs":{}}]}]}],"attrs":{}}]}", "is_fill_info": 0 }, { "id": 152092, "aid": "603776826641678343", "aidint": 603776826641678300, "type": 1, "views": 0, "uuid": "b794067e12dfb9ef87ce63827", "status": 1, "publish_time": 1649730496075, "ctime": 1649730496181, "utime": 1649730496181, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1649730496075, "article_title": "坐实大数据资源调度框架之王，Yarn为何这么牛", "article_sharetitle": "坐实大数据资源调度框架之王，Yarn为何这么牛", "article_subtitle": "", "article_summary": "Yarn的出现伴随着Hadoop的发展，使Hadoop从一个单一的大数据计算引擎，成为大数据的代名词。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 143, "name": "大数据" }, { "id": 150, "name": "hadoop" }, { "id": 153, "name": "mapreduce" }, { "id": 813, "name": "YARN" }, { "id": 19260, "name": "资源调度框架" } ], "out_id": null, "word_count": 2431, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区《","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/345416?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=other&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"Yarn为何能坐实资源调度框架之王？","attrs":{}}]},{"type":"text","text":"》，作者： JavaEdge。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop主要组成：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分布式文件系统HDFS","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分布式计算框架MapReduce","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分布式集群资源调度框架Yarn","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Yarn的出现伴随着Hadoop的发展，使Hadoop从一个单一的大数据计算引擎，成为一个集存储、计算、资源管理为一体的完整大数据平台，进而发展出自己的生态体系，成为大数据的代名词。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在MapReduce应用程序的启动过程中，最重要的就是把MapReduce程序分发到大数据集群的服务器，Hadoop 1中，这个过程主要是通过TaskTracker和JobTracker通信完成。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"方案的缺点","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"服务器集群资源调度管理和MapReduce执行过程耦合在一起，如果想在当前集群中运行其他计算任务，比如Spark或者Storm，就无法统一使用集群中的资源。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop早期，大数据技术就只有Hadoop，这缺点不明显。但随大数据发展，各种新计算框架出现，我们不可能为每种计算框架部署一个服务器集群，而且就算能部署新集群，数据还是在原来集群的HDFS上。所以需要把MapReduce的资源管理和计算框架分开，这也是Hadoop2最主要变化：将Yarn从MapReduce中分离出来，成为一个独立的资源调度框架。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 150470, "aid": "603774122393776136", "aidint": 603774122393776100, "type": 1, "views": 0, "uuid": "556b3b52f8fe66d71c570572e", "status": 1, "publish_time": 1649070279147, "ctime": 1649070279217, "utime": 1649070279217, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1649070279147, "article_title": "MapReduce 学习思考", "article_sharetitle": "MapReduce 学习思考", "article_subtitle": "", "article_summary": " MapReduce是我最近知晓的一个编程模型，简单学习了一下，发现它的设计思路和之前Rob Pike演讲的go语言并发模式还有一点异曲同工之妙，所以在此整理归纳一下。", "article_cover": "https://static001.geekbang.org/infoq/26/2669016f6f4c8e8832a9c16990449d16.jpeg", "article_cover_point": "", "author": [ { "uid": 1157417, "nickname": "en", "avatar": "https://static001.geekbang.org/account/avatar/00/11/a9/29/dad1f942.jpg", "active": 1, "is_early": 0, "ucode": "C14EDEED543AC8", "uri": "/u/ycstudy/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" } ], "out_id": null, "word_count": 1078, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"前言","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" MapReduce是我最近知晓的一个编程模型，简单学习了一下，发现它的设计思路和之前Rob Pike演讲的go语言并发模式还有一点异曲同工之妙，所以在此整理归纳一下。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一.MapReduce 简介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MapReduce 是 Google 提出的一个软件架构，用于大规模数据集（大于1TB）的并行运算。简而言之，就是将任务切分成很小的任务然后一个一个区的执行最后汇总。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4e8b907416f22d492caf61f75c41909a.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MapReduce 通常由三个步骤组成","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"map操作：每个worker节点对本地数据应用map函数，并将输出写入临时存储。主节点负责文件的分配。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"shuffle：需要将map阶段的worker节点根据生成的k/v数据重新分配，保证相同key的数据位于reduce 阶段的同一个工作节点上。（这一步视实际场景可以省略，如计算一个无重复文件中每个单词的长度，就没有必要shuffle）","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"reduce操作：reduce阶段主要负责并行的将接收到的k/v数据进行处理","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"需要注意的是如果是单线程的实现MapReduce并不会比不用MapReduce快，MapReduce的收益通常在多处理器硬件上的多线程实现中看到。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"二.Example","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.1 问题","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"让我们来解决这样一个问题--当前我有上千个文件，每个文件都有上万个单词，我需要统计所有文件中的单词出现的总次数，请问我们应该怎么办？","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.2 思考","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"结合一下MapReduce的思维，我们来解决一下这个问题。","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5d/5d6ab32eb68e60bdf2243aa99ce85dc4.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"numberedlist","attrs":{"start":1,"normalizeStart":1},"content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":1,"align":null,"origin":null},"content":[{"type":"text","text":"将数千个文件均分到map阶段的节点上，map阶段的作用是计算分给自己的这些文件中每个单词出现的总次数，这一步并行的进行统计","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null},"content":[{"type":"text","text":"shuffle阶段用于保证map阶段的数据相同的单词结果一定到相同的reduce节点上","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":3,"align":null,"origin":null},"content":[{"type":"text","text":"reduce节点进行最后的汇总处理","attrs":{}}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这便是一个简单的MapReduce的模式应用。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"三.对比总结","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"可以看到MapReduce的核心思路是将一个大的任务拆分成了很多个小的任务，之后的任务依赖于之前任务的输出。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我在学习MapReduce编程模型的过程中脑海里不断浮现的是我之前听过的一个讲座--","attrs":{}},{"type":"link","attrs":{"href":"https://vimeo.com/49718712","title":"","type":null},"content":[{"type":"text","text":"Concurrency Is Not Parallelism","attrs":{}}]}]}]}", "is_fill_info": 0 }, { "id": 147431, "aid": "603769766947180547", "aidint": 603769766947180500, "type": 1, "views": 0, "uuid": "2d6608561bd615bed0d89a246", "status": 1, "publish_time": 1648006936644, "ctime": 1648006936735, "utime": 1648006936735, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1648006936644, "article_title": "大数据培训hive和mapreduce的区别", "article_sharetitle": "大数据培训hive和mapreduce的区别", "article_subtitle": "", "article_summary": "考虑到维护的成本的增大,慢慢的开始准备将MR的作业,逐渐的移植到一些脚本平台上去，hive成了我们的首选_大数据培训。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2851561, "nickname": "@零度", "avatar": "https://static001.geekbang.org/account/avatar/00/2b/82/e9/73d547f0.jpg", "active": 1, "is_early": 0, "ucode": "328E29C1F5D312", "uri": "/profile/328E29C1F5D312/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" }, { "id": 155, "name": "hive" }, { "id": 15894, "name": "大数据开发" } ], "out_id": null, "word_count": 827, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.hive本身只是在hadoop map reduce 或者spark 计算引擎上的封装，应用场景自然更局限，不可能满足所有需求。有些场景是不能用hive来实现，就需要map reduce或者spark rdd编程来实现。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.结构复杂的日志文件，首先要经过ETL处理（使用mapreduce），得到的数据再有hive处理比较合适。直接让hive处理结构复杂的数据估计很难处理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"小结：业务比较复杂的，还是必须写mapreduce才能实现。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/94/948a9e0d1ef4ad4914d9d01c66cfac8b.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"随着工作的数据内容越来越多,越来越复杂,对应的调整也越来越多,越来越复杂.纯使用MR方式整个流程就比较复杂,如果需要修改某个部分,那首先需要修改代码中的逻辑,然后把代码打包上传到某个可访问路径上(一般就是hdfs),然后在调度平台内执行.如果改动较大的情况,可能还会需要在测试环境中多次调试. 总之就是会花比较多的时间在非业务逻辑改动的工作上.","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"考虑到维护的成本的增大,慢慢的开始准备将MR的作业,逐渐的移植到一些脚本平台上去，hive成了我们的首选_","attrs":{}},{"type":"link","attrs":{"href":"http://www.atguigu.com","title":"大数据培训","type":null},"content":[{"type":"text","text":"大数据培训","attrs":{}}]},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"mr和hive比较","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"1. 运算资源消耗","attrs":{}}]}]}", "is_fill_info": 0 }, { "id": 146812, "aid": "594760838637289476", "aidint": 594760838637289500, "type": 1, "views": 0, "uuid": "ae6b07e43a5ba219b0ddf5aa7", "status": 1, "publish_time": 1647584804013, "ctime": 1647584804157, "utime": 1647838157576, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1647584804013, "article_title": "Map-Reduce 思想在 ABAP 编程中的一个实际应用案例", "article_sharetitle": "Map-Reduce 思想在 ABAP 编程中的一个实际应用案例", "article_subtitle": "", "article_summary": "Map-Reduce 是一种编程模型和相关实现，用于在集群上使用并行分布式算法，生成和处理大规模数据集。ABAP 编程语言从语言层面支持对大规模数据的 REDUCE 操作。本文分享了笔者工作中使用Map-Reduce思路处理大规模数据集的一个实际案例，希望能给大家带来启发。", "article_cover": "https://static001.geekbang.org/infoq/60/608778d210ce6b4290568a810dbb25c3.png", "article_cover_point": "", "author": [ { "uid": 1027196, "nickname": "Jerry Wang", "avatar": "https://static001.geekbang.org/account/avatar/00/0f/ac/7c/059daba9.jpg", "active": 1, "is_early": 3, "ucode": "045F61DD1CD93A", "uri": "/u/jerrywang/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" }, { "id": 10520, "name": "abap" }, { "id": 15098, "name": "CRM系统" }, { "id": 18063, "name": "企业级应用" }, { "id": 18147, "name": "3月月更" } ], "out_id": null, "word_count": 3052, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ABAP 是一门企业级应用编程语言，其 740 版本于 2013 年发布，增添了许多新的语法和关键字：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7bc5a5b7d77370c454d20fd562b66ff2.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中一个亮点就是新引入的 REDUCE 关键字。这个关键字的作用和在大规模数据集并行计算领域里广泛使用的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"Map-Reduce","attrs":{}}],"attrs":{}},{"type":"text","text":" 编程模型中的 Reduce 操作类似，可以按照字面意思理解为","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"归约","attrs":{}}],"attrs":{}},{"type":"text","text":" 。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"什么是 Map-Reduce 思想？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Map-Reduce 是一种编程模型和相关实现，用于在集群上使用并行分布式算法，生成和处理大规模数据集。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"一个 Map-Reduce 程序由一个 Map 过程和一个 Reduce 方法组成。Map 过程负责执行过滤和排序，例如将学生按名字排序到队列中，每个名称由一个队列维护。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Reduce 方法负责执行汇总操作，例如计算学生的数量。Map-Reduce 系统通过编排分布式服务器，来并行运行各种任务，管理系统各个部分之间的所有通信和数据传输，以及提供数据冗余，实现容错机制。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下图是 Map Reduce 框架的工作步骤，统计一个海量输入数据集(比如大于 1TB )中的单词出现次数。工作步骤包含 Splitting, Mapping, Shuffling, Reducing 以得到最后的输出结果。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ba/ba2e5b6769c22bf316ece6bb763cd7a0.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 139656, "aid": "603744617398366214", "aidint": 603744617398366200, "type": 1, "views": 0, "uuid": "6754a904feac7855787123c5b", "status": 1, "publish_time": 1641866910090, "ctime": 1641866910180, "utime": 1641866926696, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1641866910090, "article_title": "Hadoop中mapreduce作业日志是如何生成的", "article_sharetitle": "Hadoop中mapreduce作业日志是如何生成的", "article_subtitle": "", "article_summary": "摘要：本篇博客介绍了hadoop中mapreduce类型的作业日志是如何生成的。主要介绍日志生成的几个关键过程，不涉及过多细节性的内容。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 150, "name": "hadoop" }, { "id": 152, "name": "hdfs" }, { "id": 153, "name": "mapreduce" }, { "id": 813, "name": "YARN" }, { "id": 17330, "name": "作业日志" } ], "out_id": null, "word_count": 2164, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"摘要：本篇博客介绍了hadoop中mapreduce类型的作业日志是如何生成的。主要介绍日志生成的几个关键过程，不涉及过多细节性的内容。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/309682?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=ei&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"《hadoop中mapreduce作业日志是如何生成的》","attrs":{}}]},{"type":"text","text":"，作者：mxg。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我们知道hadoop分为三大块：HDFS，Yarn，Mapreduce。其中mapreduce相关的核心代码都在hadoop-mapreduce-project子工程中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/978edc38389e5aef6ece804b01e9b917.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其中比较重要的功能模块有：MRAppMaster， JobHistory，以及mapreduceClient，分别对应上面的app，hs和jobclient。当然还有一些公共的工具类这里不再细表。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"MRAppMaster","attrs":{}},{"type":"text","text":"：作为一个yarn application运行的第一个Container，用于为所属的mapreduce job申请task资源，并且监控task的运行状态。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意这里引入了名词：yarn application和mapreduce job，其实是同一个事物两种不同层面下的叫法。Yarn里面运行的所有的应用都称之为application，而job是一个mapreduce类型的application在mapreduce框架下的叫法，在其他计算框架下可能又有别的叫法，总之一点，无论在计算框架侧怎么叫，在yarn这里都是yarn application。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]}]}", "is_fill_info": 0 }, { "id": 138000, "aid": "603739767925280768", "aidint": 603739767925280800, "type": 1, "views": 0, "uuid": "c15ecd0e6189244d8d6478184", "status": 1, "publish_time": 1640682956678, "ctime": 1640682956760, "utime": 1640682956760, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1640682956678, "article_title": "Hadoop学习过程中遇到的错误及解决方法", "article_sharetitle": "Hadoop学习过程中遇到的错误及解决方法", "article_subtitle": "", "article_summary": "在自己学习搭建并使用Hadoop集群的时候，遇到的一些错误及解决方法。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2867710, "nickname": "阿丞", "avatar": "https://static001.geekbang.org/account/avatar/00/2b/c1/fe/44f2f511.jpg", "active": 1, "is_early": 0, "ucode": "64331B410D36A8", "uri": "/profile/64331B410D36A8/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 150, "name": "hadoop" }, { "id": 152, "name": "hdfs" }, { "id": 153, "name": "mapreduce" }, { "id": 813, "name": "YARN" } ], "out_id": null, "word_count": 4431, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.ERROR: Unable to write in /opt/module/hadoop-3.1.3/logs. Aborting.","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"错误","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"ERROR: Unable to write in /opt/module/hadoop-3.1.3/logs. Aborting.\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"解决方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"sudo chmod -R 777 /opt/module/hadoop-3.1.3/logs\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"2.ERROR: JAVA_HOME is not set and could not be found.","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"错误","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"[lzc@hadoop02 sbin]$ start-dfs.sh \nStarting namenodes on [hadoop02]\nStarting datanodes\nhadoop04: ERROR: JAVA_HOME is not set and could not be found.\nhadoop03: ERROR: JAVA_HOME is not set and could not be found.\nStarting secondary namenodes [hadoop04]\nhadoop04: ERROR: JAVA_HOME is not set and could not be found.\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"解决方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"#在hadoop-env.sh文件中添加JAVA_HOME路径\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.hdfs Couldn't preview the file.","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"错误","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"hdfs Couldn't preview the file.\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"解决方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"#在windows系统中修改hosts文件，添加Hadoop集群的ip和主机名\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.测试","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"HDFS","attrs":{}},{"type":"text","text":"和","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Yarn","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"错误","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"Container exited with a non-zero exit code 1. Error file: prelaunch.err.\nLast 4096 bytes of prelaunch.err :\nLast 4096 bytes of stderr :\nError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster\n\nPlease check whether your etc/hadoop/mapred-site.xml contains the below configuration:\n<property>\n<name>yarn.app.mapreduce.am.env</name>\n<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>\n</property>\n<property>\n<name>mapreduce.map.env</name>\n<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>\n</property>\n<property>\n<name>mapreduce.reduce.env</name>\n<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>\n</property>\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"解决方式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"#按照提示在mapred-site.xml文件中添加\n\n<property>\n <name>yarn.app.mapreduce.am.env</name>\n <value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>\n</property>\n<property>\n <name>mapreduce.map.env</name>\n <value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>\n</property>\n<property>\n <name>mapreduce.reduce.env</name>\n <value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>\n</property>\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"5.集群崩溃处理办法","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"#1.杀死进程\nsbin/stop-dfs.sh\n\n#2.删除每个集群上的data和logs\nrm -rf data/ logs/\n\n#3.格式化\nhdfs namenode -format\n\n#4.启动集群\nsbin/start-dfs.sh\n\n#问题：删除的data和logs岂不是丢失了\n","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.配置历史服务器有问题（历史服务器启动不了）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"错误","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"[lzc@hadoop02 hadoop-3.1.3]$ bin/mapred mapred --daemon start historyserver\nERROR: mapred is not COMMAND nor fully qualified CLASSNAME.\nUsage: mapred [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]\n or mapred [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]\n where CLASSNAME is a user-provided Java class\n\n OPTIONS is none or any of:\n\n--config dir Hadoop config directory\n--debug turn on shell script debug mode\n--help usage information\n\n SUBCOMMAND is one of:\n\n\n Admin Commands:\n\nframeworkuploader mapreduce framework upload\nhsadmin job history server admin interface\n\n Client Commands:\n\narchive create a Hadoop archive\narchive-logs combine aggregated logs into hadoop archives\nclasspath prints the class path needed for running mapreduce\n subcommands\ndistcp copy file or directories recursively\nenvvars display computed Hadoop environment variables\njob manipulate MapReduce jobs\nminicluster CLI MiniCluster\npipes run a Pipes job\nqueue get information regarding JobQueues\nsampler sampler\nstreaming launch a mapreduce streaming job\nversion print the version\n\n Daemon Commands:\n\nhistoryserver run job history servers as a standalone daemon\n\nSUBCOMMAND may print help when invoked w/o parameters or with -h.\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看hadoop/sbin目录，发现只有一个mr-jobhistory-daemon.sh文件，尝试启动这个文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"sbin/mr-jobhistory-daemon.sh start historyserver\n\n#情况\n[lzc@hadoop02 hadoop-3.1.3]$ sbin/mr-jobhistory-daemon.sh start historyserver\nWARNING: Use of this script to start the MR JobHistory daemon is deprecated.\nWARNING: Attempting to execute replacement \"mapred --daemon start\" instead.\n[lzc@hadoop02 hadoop-3.1.3]$ jps\n19009 DataNode\n2546 Jps\n1174 NodeManager\n18878 NameNode\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"发现历史服务器没有启动起来，查看mapred-site.xml文件配置信息","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"sh"},"content":[{"type":"text","text":"<configuration>\n\n <property>\n <name>mapreduce.framework.name</name>\n <value>yarn</value>\n </property>\n\n<property>\n <name>yarn.app.mapreduce.am.env</name>\n <value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>\n</property>\n<property>\n <name>mapreduce.map.env</name>\n <value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>\n</property>\n<property>\n <name>mapreduce.reduce.env</name>\n <value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>\n</property>\n\n\n<property>\n <name>mapreduce.jobhistory.address</name>\n <value>hadoop102:10020</value>\n</property>\n\n\n<property>\n <name>mapreduce.jobhistory.webapp.address</name>\n <value>hadoop102:19888</value>\n</property>\n\n</configuration>\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attr...(15.2 KB)", "is_fill_info": 0 }, { "id": 104890, "aid": "603703573279014920", "aidint": 603703573279014900, "type": 1, "views": 0, "uuid": "93b45b49df2b8f215619ed234", "status": 1, "publish_time": 1631846373160, "ctime": 1631846373259, "utime": 1632020136379, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1631846373160, "article_title": "MapReduce 示例：减少 Hadoop MapReduce 中的侧连接", "article_sharetitle": "MapReduce 示例：减少 Hadoop MapReduce 中的侧连接", "article_subtitle": "", "article_summary": "摘要：在排序和reducer 阶段，reduce 侧连接过程会产生巨大的网络I/O 流量，在这个阶段，相同键的值被聚集在一起。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 150, "name": "hadoop" }, { "id": 153, "name": "mapreduce" }, { "id": 1478, "name": "排序" }, { "id": 14476, "name": "侧连接" }, { "id": 14477, "name": "联接" } ], "out_id": null, "word_count": 4862, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"摘要：在排序和reducer 阶段，reduce 侧连接过程会产生巨大的网络I/O 流量，在这个阶段，相同键的值被聚集在一起。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/300550?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=ei&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"《MapReduce 示例：减少 Hadoop MapReduce 中的侧连接》","attrs":{}}]},{"type":"text","text":"，作者：DonglianLin。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在这篇博客中，将使用 MapReduce 示例向您解释如何在 Hadoop MapReduce 中执行缩减侧连接。在这里，我假设您已经熟悉 MapReduce 框架并知道如何编写基本的 MapReduce 程序。本博客中讨论的主题如下：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"什么是加入？","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"MapReduce 中的连接","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"什么是 Reduce 侧连接？","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"减少侧连接的 MapReduce 示例","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"结论","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"什么是联接？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"join操作用于基于外键将两个或多个数据库表合并。通常，公司在其数据库中为客户和交易记录维护单独的表。而且，很多时候这些公司需要使用这些单独表格中的数据生成分析报告。因此，他们使用公共列（外键）（如客户 ID 等）对这些单独的表执行连接操作，以生成组合表。然后，他们分析这个组合表以获得所需的分析报告。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"MapReduce 中的连接","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"就像 SQL join 一样，我们也可以在 MapReduce 中对不同的数据集进行 join 操作。MapReduce 中有两种类型的连接操作：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Map Side Join：","attrs":{}},{"type":"text","text":"顾名思义，join操作是在map阶段本身进行的。因此，在 map side join 中，mapper 执行 join 并且每个 map 的输入都必须根据键进行分区和排序。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"减少副加入：","attrs":{}},{"type":"text","text":"顾名思义，在减少侧加入，减速是负责执行连接操作。由于排序和改组阶段将具有相同键的值发送到同一个 reducer，因此它比 map side join 相对简单和容易实现，因此，默认情况下，数据是为我们组织的。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现在，让我们详细了解reduce side join。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"什么是减少侧连接？","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c2/c23a87b595f3dc9f0324151ff1c58fb1.png","alt":"","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如前所述，reduce side join是在reducer 阶段执行join 操作的过程。基本上，reduce side join以下列方式发生：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Mapper 根据公共列或连接键读取要组合的输入数据。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"映射器处理输入并向输入添加标签以区分属于不同来源或数据集或数据库的输入。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"映射器输出中间键值对，其中键只是连接键。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在排序和改组阶段之后，会为减速器生成一个键和值列表。 ","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"现在，reducer 将列表中存在的值与键连接起来，以给出最终的聚合输出。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"减少边连接的 MapReduce 示例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"假设我有两个单独的运动场数据集：","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"cust_details：","attrs":{}},{"type":"text","text":" 它包含客户的详细信息。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"transaction_details：","attrs":{}},{"type":"text","text":" 包含客户的交易记录。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 102057, "aid": "603696159397236743", "aidint": 603696159397236700, "type": 1, "views": 0, "uuid": "40c558eef0bcbeb825cd0727d", "status": 1, "publish_time": 1630036343403, "ctime": 1630036343508, "utime": 1630036343508, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1630036343403, "article_title": "Spark必读！总有一些Spark知识点你需要知道", "article_sharetitle": "Spark必读！总有一些Spark知识点你需要知道", "article_subtitle": "", "article_summary": "一文梳理 Spark 必知必会知识点既可以快速对 Spark 有个简单清晰的认知；也可以回顾 Spark 基础概念和原理；还可以进行知识点查缺补漏。", "article_cover": "https://static001.geekbang.org/infoq/6f/6fa13c501de472b158bd5b0119774fc2.png", "article_cover_point": "", "author": [ { "uid": 2345790, "nickname": "Qunar技术沙龙", "avatar": "https://static001.geekbang.org/account/avatar/00/23/cb/3e/60604d56.jpg", "active": 1, "is_early": 0, "ucode": "CA15029C90B3C1", "uri": "/profile/CA15029C90B3C1/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 151, "name": "spark" }, { "id": 152, "name": "hdfs" }, { "id": 153, "name": "mapreduce" }, { "id": 3096, "name": "Web UI" }, { "id": 11240, "name": "数据倾斜" } ], "out_id": null, "word_count": 5619, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"作者介绍","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"周明，去哪网算法开发工程师。2018年加入去哪儿网，主要从事推荐算法相关工作。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"这篇文章可以带给你什么","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"不太了解 Spark ：","attrs":{}},{"type":"text","text":"可以快速对 Spark 有个简单且清晰的认知，同时知道Spark可以用来做什么，对于经常处理大数据的同学可以思考如何运用到自己的工作中；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"刚开始写 Spark ：","attrs":{}},{"type":"text","text":"一起来回顾 Spark 基础概念和原理，避免新手那些常见的坑，培养性能优化意识，知道如何做性能调优；","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Spark 老司机：","attrs":{}},{"type":"text","text":"一起来回顾 Spark 的三三两两，知识点查缺补漏。","attrs":{}}]}]}],"attrs":{}},{"type":"horizontalrule","attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、Spark 的由来","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7f/7f6ce89c001436b4f1a6463ca912cd7d.webp","alt":"图片","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"花一些篇幅来讲讲大数据技术的发展史，可以帮助我们更好理解 Spark 诞生的历史背景。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天我们常说的大数据技术，其实起源于 Google 在 2004 年前后发表的三篇论文，也就是我们经常听到的“三驾马车”，分别是分布式文件系统 GFS 、大数据分布式计算框架 MapReduce 和 NoSQL 数据库系统 BigTable 。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2006年，Lucene 开源项目的创始人 Doug Cutting 基于论文原理开发了 Hadoop ，主要包括 Hadoop 分布式文件系统 HDFS 和大数据计算引擎 MapReduce ，Hadoop 一经发布便引起轰动，Yahoo 、百度和阿里巴巴等知名互联网公司逐渐使用 Hadoop 进行大数据存储与计算。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 101117, "aid": "594686521845288960", "aidint": 594686521845289000, "type": 1, "views": 0, "uuid": "146dd9cc69f9461a9bfdd4da3", "status": 1, "publish_time": 1629441055981, "ctime": 1629441056076, "utime": 1629441056076, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1629441055981, "article_title": "云小课｜MRS基础原理之ClickHouse组件介绍", "article_sharetitle": "云小课｜MRS基础原理之ClickHouse组件介绍", "article_subtitle": "", "article_summary": "阅识风云是华为云信息大咖，擅长将复杂信息多元化呈现，其出品的一张图(云图说)、深入浅出的博文(云小课)或短视频(云视厅)总有一款能让您快速上手华为云。更多精彩内容请单击此处。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2065305, "nickname": "华为云开发者联盟", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/83/99/ce751136.jpg", "active": 1, "is_early": 0, "ucode": "0A99CCF397BE5D", "uri": "/profile/0A99CCF397BE5D/", "author_type": 5, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" }, { "id": 211, "name": "开源" }, { "id": 1358, "name": "Clickhouse" }, { "id": 13781, "name": "EI企业智能" }, { "id": 13782, "name": "列式数据库" } ], "out_id": null, "word_count": 2929, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/29/2970c9b87f23cafcdc6592a439d77640.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"阅识风云是华为云信息大咖，擅长将复杂信息多元化呈现，其出品的一张图(云图说)、深入浅出的博文(云小课)或短视频(云视厅)总有一款能让您快速上手华为云。更多精彩内容请单击","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/194909?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=other&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"此处","attrs":{}}]},{"type":"text","text":"。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"摘要：在2016年开源的高性能、开源联机分析列式数据库管理系统。其数据压缩比高，基于多核并行计算、向量化执行与SIMD，在性能上表现卓越。当前ClickHouse被广泛的应用于互联网广告、App和Web流量、电信、金融、物联网等众多领域，非常适用于商业智能化应用场景，在国内外有大量的应用和实践。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文分享自华为云社区","attrs":{}},{"type":"link","attrs":{"href":"https://bbs.huaweicloud.com/blogs/278872?utm_source=infoq&utm_medium=bbs-ex&utm_campaign=other&utm_content=content","title":"","type":null},"content":[{"type":"text","text":"《使用scipy.linalg在Python中使用线性系统》","attrs":{}}]},{"type":"text","text":"，作者： Hi,EI。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse是一款开源的面向联机分析处理的列式数据库，其独立于Hadoop大数据体系，最核心的特点是极致压缩率和极速查询性能。同时，ClickHouse支持SQL查询，且查询性能好，特别是基于大宽表的聚合分析查询性能非常优异，比其他分析型数据库速度快一个数量级。当前ClickHouse被广泛的应用于互联网广告、App和Web流量、电信、金融、物联网等众多领域，非常适用于商业智能化应用场景，在国内外有大量的应用和实践。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/20/20bae0156db44ad3923c1bec10c67739.jpeg","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"ClickHouse关键特性介绍","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"完备的DBMS功能","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClickHouse拥有完备的数据库管理功能，具备一个DBMS（Database Management System，数据库管理系统）基本的功能，如下所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DDL （数据定义语言）：可以动态地创建、修改或删除数据库、表和视图，而无须重启服务。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"DML（数据操作语言）：可以动态查询、插入、修改或删除数据。","attrs":{}}]}]}", "is_fill_info": 0 }, { "id": 93288, "aid": "603680680585584643", "aidint": 603680680585584600, "type": 1, "views": 0, "uuid": "9a8c90ae297f80e5f33b7232c", "status": 1, "publish_time": 1626257336635, "ctime": 1626257336708, "utime": 1626257507536, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1626257336635, "article_title": "MapReduce排序以及序列化实践", "article_sharetitle": "MapReduce排序以及序列化实践", "article_subtitle": "", "article_summary": "序列化（Serialization）是指把结构化对象转化为字节流。", "article_cover": "", "article_cover_point": "", "author": [ { "uid": 2304155, "nickname": "五分钟学大数据", "avatar": "https://static001.geekbang.org/account/avatar/00/23/28/9b/6a6fad37.jpg", "active": 1, "is_early": 3, "ucode": "8E88B105371A1B", "uri": "/u/yuanmore/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" }, { "id": 11965, "name": "7月日更" } ], "out_id": null, "word_count": 2503, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"序列化（Serialization）是指把结构化对象转化为字节流。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"反序列化（Deserialization）是序列化的逆过程。把字节流转为结构化对象。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当要在进程间传递对象或持久化对象的时候，就需要序列化对象成字节流反之当要将接收到或从磁盘读取的字节流转换为对象，就要进行反序列化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Java 的序列化（Serializable）是一个重量级序列化框架，一个对象被序列化后，会附带很多额外的信息（各种校验信息，header，继承体系…），不便于在网络中高效传输；所以，hadoop 自己开发了一套序列化机制（Writable），精简，高效。不用像 java 对象类一样传输多层的父子关系，需要哪个属性就传输哪个属性值，大大的减少网络传输的开销。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Writable是Hadoop的序列化格式，hadoop定义了这样一个Writable接口。一个类要支持可序列化只需实现这个接口即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 90482, "aid": "603674978306154503", "aidint": 603674978306154500, "type": 1, "views": 0, "uuid": "ae23ef1e68593b221980921f2", "status": 1, "publish_time": 1624865178566, "ctime": 1624865178637, "utime": 1624865178637, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1624865178566, "article_title": "腾讯云EMR基于YARN针对云原生容器化的优化与实践", "article_sharetitle": "腾讯云EMR基于YARN针对云原生容器化的优化与实践", "article_subtitle": "", "article_summary": "HADOOP资源调度器YARN在容器环境中能多有效率？", "article_cover": "https://static001.geekbang.org/infoq/24/243acf13475ad61cc92837d92b08b722.jpeg", "article_cover_point": "", "author": [ { "uid": 2042867, "nickname": "腾讯云大数据", "avatar": "https://static001.geekbang.org/account/avatar/00/1f/2b/f3/77333d74.jpg", "active": 1, "is_early": 2, "ucode": "8260DA10E9A4F3", "uri": "/profile/8260DA10E9A4F3/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 153, "name": "mapreduce" } ], "out_id": null, "word_count": 3568, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"导语 | 传统HADOOP生态系统使用YARN管理/调度计算资源，该系统⼀般具有明显的资源使⽤周期。实时计算集群资源消耗主要在⽩天，而数据报表型业务则安排在离线计算集群中。离在线业务分开部署的首要问题就是资源使用率低，消耗成本⾼。随着业务的增⻓和突发的报表计算需求，为了解决为离线集群预留资源，腾讯云EMR团队和容器团队联合推出Hadoop Yarn on Kubernetes Pod，以提⾼容器资源使用率，降低资源成本，将闲时容器集群CPU使⽤率提升数倍之多。本文主要介绍HADOOP资源调度器YARN在容器环境中的优化与实践。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一、Hadoop Yarn on Kubernetes Pod 混合部署模式","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hadoop Yarn on Kubernetes Pod 方案提供弹性扩缩容和离在线混合部署两项功能。弹性扩缩容主要聚焦于如何利⽤云原生资源，快速扩容资源以补充算力。离在线混合部署模式的目的是为了充分使用在线集群的空闲资源，尽可能减少为离线集群预留空闲资源的频次。","attrs":{}}]}]}", "is_fill_info": 0 }, { "id": 84430, "aid": "594651132882296840", "aidint": 594651132882296800, "type": 1, "views": 0, "uuid": "6ee00897afbfb7fdf6ee4b829", "status": 1, "publish_time": 1620801172437, "ctime": 1620801172524, "utime": 1620801172524, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1620801172437, "article_title": "MapReduce排序以及序列化", "article_sharetitle": "MapReduce排序以及序列化", "article_subtitle": "", "article_summary": "序列化（Serialization）是指把结构化对象转化为字节流。", "article_cover": "https://static001.geekbang.org/infoq/29/2942095ecc9aefda5e10f385e171ddcd.jpeg", "article_cover_point": "", "author": [ { "uid": 2304155, "nickname": "五分钟学大数据", "avatar": "https://static001.geekbang.org/account/avatar/00/23/28/9b/6a6fad37.jpg", "active": 1, "is_early": 3, "ucode": "8E88B105371A1B", "uri": "/u/yuanmore/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 143, "name": "大数据" }, { "id": 150, "name": "hadoop" }, { "id": 153, "name": "mapreduce" } ], "out_id": null, "word_count": 2764, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#FF7021","name":"orange"}},{"type":"strong","attrs":{}}],"text":"序列化","attrs":{}},{"type":"text","text":"（Serialization）是指把结构化对象转化为字节流。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"color","attrs":{"color":"#FF7021","name":"orange"}},{"type":"strong","attrs":{}}],"text":"反序列化","attrs":{}},{"type":"text","text":"（Deserialization）是序列化的逆过程。把字节流转为结构化对象。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"当要在进程间传递对象或持久化对象的时候，就需要序列化对象成字节流反之当要将接收到或从磁盘读取的字节流转换为对象，就要进行反序列化。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Java 的序列化（Serializable）是一个重量级序列化框架，一个对象被序列化后，会附带很多额外的信息（各种校验信息，header，继承体系…），不便于在网络中高效传输；所以，hadoop 自己开发了一套序列化机制（Writable），精简，高效。不用像 java 对象类一样传输多层的父子关系，需要哪个属性就传输哪个属性值，大大的减少网络传输的开销。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Writable是Hadoop的序列化格式，hadoop定义了这样一个Writable接口。一个类要支持可序列化只需实现这个接口即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", "is_fill_info": 0 }, { "id": 77482, "aid": "594633452581949448", "aidint": 594633452581949400, "type": 1, "views": 0, "uuid": "2006c36923ecdf055817b0968", "status": 1, "publish_time": 1616484692859, "ctime": 1616484692946, "utime": 1616484692946, "vid": "", "source": 2, "sub_status": 0, "ip_address": "", "score": 1616484692859, "article_title": "MapReduce的运行机制详解", "article_sharetitle": "MapReduce的运行机制详解", "article_subtitle": "", "article_summary": "整个Map阶段流程大体如上图所示。", "article_cover": "https://static001.geekbang.org/infoq/1e/1ed4815548718a63507fb8e744dd215d.jpeg", "article_cover_point": "", "author": [ { "uid": 2304155, "nickname": "五分钟学大数据", "avatar": "https://static001.geekbang.org/account/avatar/00/23/28/9b/6a6fad37.jpg", "active": 1, "is_early": 3, "ucode": "8E88B105371A1B", "uri": "/u/yuanmore/", "author_type": 1, "vip": 0 } ], "translator": null, "planner": null, "topic": null, "is_collect": false, "no_author": "", "is_promotion": false, "share_pic": "", "sub_type": 0, "comment_user": 0, "comment_count": 0, "speaker_title": "", "speaker_introduce": "", "video_status": 0, "duration": "", "total_duration": "", "label": [ { "id": 143, "name": "大数据" }, { "id": 153, "name": "mapreduce" }, { "id": 8441, "name": "3月日更" } ], "out_id": null, "word_count": 1702, "front_type": 1, "content_short": "{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"MapTask 工作机制","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/af/af303fe3511b7aa1d7ada54866743af8.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e0/e03278cf1aa3cfb50c41802f57ef92db.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整个Map阶段流程大体如上图所示。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"简单概述：inputFile通过split被逻辑切分为多个split文件，通过Record按行读取内容给map（用户自己实现的）进行处理，数据被map处理结束之后交给OutputCollector收集器，对其结果key进行分区（默认使用hash分区），然后写入buffer，每个map task都有一个内存缓冲区，存储着map的输出结果，当缓冲区快满的时候需要将缓冲区的数据以一个临时文件的方式存放到磁盘，当整个map task结束后再对磁盘中这个map task产生的所有临时文件做合并，生成最终的正式输出文件，然后等待reduce task来拉数据","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"详细步骤","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1) 读取数据组件 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"InputFormat","attrs":{}},{"type":"text","text":" (默认 TextInputFormat) 会通过 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"getSplits","attrs":{}}],"attrs":{}},{"type":"text","text":" 方法对输入目录中文件进行逻辑切片规划得到 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"block","attrs":{}}],"attrs":{}},{"type":"text","text":", 有多少个 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"block","attrs":{}}],"attrs":{}},{"type":"text","text":"就对应启动多少个 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"MapTask","attrs":{}}],"attrs":{}},{"type":"text","text":".","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2) 将输入文件切分为 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"block","attrs":{}}],"attrs":{}},{"type":"text","text":" 之后, 由 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"RecordReader","attrs":{}}],"attrs":{}},{"type":"text","text":" 对象 (默认是LineRecordReader) 进行","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"读取","attrs":{}},{"type":"text","text":", 以 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"\\n","attrs":{}}],"attrs":{}},{"type":"text","text":" 作为分隔符, 读取一行数据, 返回 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"<key，value>","attrs":{}}],"attrs":{}},{"type":"text","text":". Key 表示每行首字符偏移值, Value 表示这一行文本内容","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3) 读取 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"block","attrs":{}}],"attrs":{}},{"type":"text","text":" 返回 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"<key,value>","attrs":{}}],"attrs":{}},{"type":"text","text":", ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"进入用户自己继承的 Mapper 类中","attrs":{}},{"type":"text","text":"，执行用户重写的 map 函数, RecordReader 读取一行这里调用一次","attrs":{}}]}],"attrs":{}},{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4) Mapper 逻辑结束之后, 将 Mapper 的每条结果通过 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"context.write","attrs":{}}],"attrs":{}},{"type":"text","text":" 进行collect数据收集. 在 collect 中, 会先对其进行分区处理，默认使用 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"HashPartitioner","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":" MapReduce 提供 Partitioner 接口, 它的作用就是根据 Key 或 Value 及 Reducer 的数量来决定当前的这对输出数据最终应该交由哪个 Reduce task 处理, 默认对 Key Hash 后再以 Reducer 数量取模. 默认的取模方式只是为了平均 Reducer 的处理能力, 如果用户自己对 Partitioner 有需求, 可以订制并设置到 Job 上","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"5) 接下来, 会将数据写入内存, 内存中这片区域叫做环形缓冲区, 缓冲区的作用是批量收集 Mapper 结果, 减少磁盘 IO 的影响. 我们的 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Key/Value 对以及 Partition 的结果都会被写入缓冲区","attrs":{}},{"type":"text","text":". 当然, 写入之前，Key 与 Value 值都会被序列化成字节数组","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":" 环形缓冲区其实是一个数组, 数组中存放着 Key, Value 的序列化数据和 Key, Value 的元数据信息, 包括 Partition, Key 的起始位置, Value 的起始位置以及 Value 的长度. 环形结构是一个抽象概念\n 缓冲区是有大小限制, 默认是 100MB. 当 Mapper 的输出结果很多时, 就可能会撑爆内存, 所以需要在一定条件下将缓冲区中的数据临时写入磁盘, 然后重新利用这块缓冲区. 这个从内存往磁盘写数据的过程被称为 Spill, 中文可译为溢写. 这个溢写是由单独线程来完成, 不影响往缓冲区写 Mapper 结果的线程. 溢写线程启动时不应该阻止 Mapper 的结果输出, 所以整个缓冲区有个溢写的比例 spill.percent. 这个比例默认是 0.8, 也就是当缓冲区的数据已经达到阈值 buffer size * spill percent = 100MB * 0.8 = 80MB, 溢写线程启动, 锁定这 80MB 的内存, 执行溢写过程. Mapper 的输出结果还可以往剩下的 20MB 内存中写, 互不影响","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"6) 当溢写线程启动后, 需要","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"对这 80MB 空间内的 Key 做排序 (Sort)","attrs":{}},{"type":"text","text":". 排序是 MapReduce 模型默认的行为, 这里的排序也是对序列化的字节做的排序","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"codeblock","attrs":{"lang":""},"content":[{"type":"text","text":" 如果 Job 设置过 Combiner, 那么现在就是使用 Combiner 的时候了. 将有相同 Key 的 Key/Value 对的 Value 合并在起来, 减少溢写到磁盘的数据量. Combiner 会优化 MapReduce 的中间结果, 所以它在整个模型中会多次使用\n 那哪些场景才能使用 Combiner 呢? 从这里分析, Combiner 的输出是 Reducer 的输入, Combiner 绝不能改变最终的计算结果. Combiner 只应该用于那种 Reduce 的输入 Key/Value 与输出 Key/Value 类型完全一致, 且不影响最终结果的场景. 比如累加, 最大值等. Combiner 的使用一定得慎重, 如果用好, 它对 Job 执行效率有帮助, 反之会影响 Reducer 的最终结果","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7) ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"合并溢写文件","attrs":{}},{"type":"text","text":", 每次溢写会在磁盘上生成一个临时文件 (写之前判断是否有 Combiner), 如果 Mapper 的输出结果真的很大, 有多次这样的溢写发生, 磁盘上相应的就会有多个临时文件存在. 当整个数据处理结束之后开始对磁盘中的临时文件进行 Merge 合并, 因为最终的文件只有一个, 写入磁盘, 并且为这个文件提供了一个索引文件, 以记录每个reduce对应数据的偏移量","attrs":{}}]}],"attrs":{}}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}", (...) }, (...) ], (...) }, (...)}

创作场景

mapreduce

推荐作者

推荐标签