OpenMLDB Weekly Update(2021.8.30-2021.9.6)

OpenMLDB

Summary

本周合并 Pull requests 18 个，新增 Pull requests 9 个，关闭 Issues 10 个，新增 Issues 26 个。总计 94 个文件修改，新增 1502 行代码，删除 7764 行代码。发布 Release 版本 v0.2.3。

Merged Pull Requests

• docs: add the demo link in readme#305 

• docs: add a new logo#364 

• refactor: refact AppendEntries in log replicator#327 

• fix: sql and ns client desc result#341 

• docs: add compile doc#346 

• docs: new shilds#355 merged 

• docs: added new shields#348 

• fix: importer fix and doc#328 

• build(deps): bump hive-exec from 3.0.0 to 3.1.1 in /java/openmldb-import#307 

• update the blog link#299 

• refactor: remove unused scripts#343 

• fix: revert spark and hadoop version#344 

• build(deps-dev): bump junit from 4.11 to 4.13.1 in /java/openmldb-import#308 

• feat: update spark and hadoop version#332 

• feat: update openmldb version#310 

• feat: add offline update for changelog#326 

• build(hybridse): prepare version to 0.2.3#325 

• feat: build nearline tablet framework#246 

Open Pull Requests

• feat: add java common lib#347 

• feat: support column query with the same name in window skew optimization#349

• feat: enhance plan optimization for group and filter#350 

• feat: add batchjob module#359 

• feat: add task manager module#361 

• ci: openmldb java deploy workflow#366 

• feat: add openmldb jmh#367 

• test: modify test case#368 

• update readme_add ai pipeline#370

Close Issues

• docs: Add a new OpenMLDB logo#363 

• bug: sql client `desc` result#331 

• Add compile doc#345 

• 在readme里添加知乎、oschina等媒体链接#298 

• Support reading and write tables in hive metastore catalog#202 

• Remove unused scripts#337 

• Update Spark version to 3.1.2#333 

• TestWindowUnion failed: 20 did not equal 10 (TestWindowUnion.scala:58)#122 

• Update the verison of openmldb-jdbc, openmldb-native and openmldb-import before auto release#324 

• Build NearLineTablet framework#218

Open Issues

• feat: support and validate query table with multiple-levels path#365 

• feat: refactor of sql query router#362 

• Add TaskManager service to submit OpenMLDB Batch jobs#360 

• ut: so many unit test have memory leak, we should use sanitizer to check#358 

• bug: cicd build failed but report ok#357 

• Multiple columns with the same name can't execute when last join and over window#356 

• bug: cicd python test#354 

• bug: create index can't set ttl with no units#353 

• Add OpenMLDB docs#352 

• Add module for OpenMLDB Batch to run custom SQL and submit by TaskManager#351 

• Add java common lib#342 

• Replace `rtidb` to `openmldb` in some files#340 

• Enable the unit test of TestWindowUnion#339 

• Test and register the iceberg tables without using default Spark catalog#338 

• Enable UnsafeRowOpt by default and resolve the related issues#336 

• Enable WindowSkewOpt by default and resolve the running issues#335 

• Add the unit test for physical nodes like GroupByAgg for offline#334 

• Add deployment script#330 

• Reduce the size of docker image#329 

• Automatic packaging openmldb-import after tag creation#323 

• Add `exit` command for sql_client#322 

• Not able to find out SQL syntax docs from readme.#321 

• feat: support distributed query on BatchMode under some restrictions#318 

• feat: engine plan optimization for where and group with the same partition#317 

• feat: support integration test for java/python sdk and offline batch#316 

• enable python style check via pylint#315 

Contributors

• aceforeverd(teapot@aceforeverd.com)

• dependabot[bot](49699333+dependabot[bot]@http://users.noreply.github.com)

• dl239(dl239@126.com)

• HuangWei(huangwei@apache.org)

• imotai(codego.me@gmail.com)
  

• Lucifer(63491234+ashish-patwal@users.noreply.github.com)

• luyuxiao211(79981940+luyuxiao211@users.noreply.github.com)

• Rohan Sharma(rhnsharma5113@gmail.com)

• tobe(tobeg3oogle@gmail.com)

• Wang ZeKai(1290561498@qq.com)

• wuyou10206(zw1020688@163.com)

• xuman2019(52193163+xuman2019@users.noreply.github.com)

Highlights

本周发布 Release v0.2.3 版本，在功能特性、性能稳定性上都在稳步提升。在线存储引擎数据导入工具支持批量导入（Bulk load），在线计算引擎批处理模式（Batch mode）支持带参数的 SQL 查询，离线存储引擎支持数仓元数据服务（Hive metastore）以及数据湖存储格式（Apache Iceberg），数据分析能力支持 Trino 计算集群，离线计算引擎支持全局排序节点（SortByNode）。修复大量性能和功能相关的 Bug，如端到端离线计算引擎测试时使用相同 SQL 导致的错误，在线存储引擎添加索引使用 desc 来展示 TTL 值有误等等。SQL 语法也有拓展支持 nvl、nvl2 等函数，支持”&”、“|”、“^”、“~“等位操作函数，支持 Between 谓词逻辑等。详细 Release 日志请参考 Release OpenMLDB v0.2.3 · 4paradigm/OpenMLDB 。

OpenMLDB 整体架构也在升级重构中，全新的近实时存储服务 Nearline Tablet 和批处理任务管理服务 TaskManager 也在初步设计和实现中，新增模块在 OpenMLDB 的易用性和可拓展性上有进一步增强，相信在不久的 v0.3.0 正式版中可以与开发者见面。

欢迎更多开发者关注和参与OpenMLDB开源项目。