OpenMLDB Weekly Update(2021.9.5-2021.9.12)

OpenMLDB

Summary

本周合并 Pull requests 11 个，新增 Pull requests 9 个，关闭 Issues 14 个，新增 Issues 12 个。总计 57 个文件修改，新增 3002 行代码，删除 539 行代码。

Merged Pull Requests

• feat: refine the description of benchmark in readme#405

• feat: update the benchmark images for readme#404

• feat: remove junit dependencies and covert java cases as scala#396

• feat: bump hadoop-common to 2.8.5 to avoid vulnerability#388

• fix: fix a bug in union with instance not in window#381

• feat: add batchjob module#359

• feat: add task manager module#361

• feat: enhance plan optimization for group and filter#350

• feat: support column query with the same name in window skew optimization#349

• feat: add java common lib#347

• test: modify test case#368

Open Pull Requests

• feat: bump junit from 4.11 to 4.13.1 in /java/openmldb-batchjob#382

• WIP : create message table#383

• feat: add batch, batchjob and taskmanager as java submodules#386

• fix: fix some code implementation in window skew optimization#392

• feat: support aggregation over the whole table#393

• feat: add integration test#395

• feat: support insert multiple rows into a table using a single SQL insert statement#399

• feat: add kubernetes java dependencies for taskmanager#400

• fix: python test and cicd#401

Close Issues

• Remove junit since of the EPL license for OpenMLDB Batch#390

• Avoid vulnerability by upgrading hadoop-common dependency#387

• rtidb磁盘占用#389

• feat: engine plan optimization for where and group with the same partition#317

• 是否支持批量插入#177

• 字段含义#378

• 表使用大小#373

• Fix the issue to enable window skew optimization for window union case#374

• Enable WindowSkewOpt by default and resolve the running issues#335

• Add module for OpenMLDB Batch to run custom SQL and submit by TaskManager#351

• Add TaskManager service to submit OpenMLDB Batch jobs#360

• scripts: package java sdk set cmake type to release#372

• Multiple columns with the same name can't execute when last join and over window#356

• Add java common lib#342

Open Issues

• feat: improve error message system.#406

• feat: support insert multiple rows into a table using a single SQL insert statement.#398

• feat: api server support parameterized query#397

• Add engine test on performance insensitive mode#394

• Bug: SQL INSERT Statement with multi rows does not work as expected#391

• Make openmldb-batchjob and openmldb-taskmanager as submodules of openmldb-parent#385

• Support AWS S3 for offline data lake storage#384

• Support creating database API for NearlineTablet#380

• Support hive metastore for NearlineTablet#379

• Support submit and manage Flink jobs for TaskManager#376

• Support submit and manage Kubernetes jobs for TaskManager#375

• scripts: package java sdk set cmake type to release#371

Contributors

• dl239 (dl239@126.com)

• tobe (tobeg3oogle@gmail.com)

• Wang ZeKai (1290561498@qq.com)

• wuyou10206 (zw1020688@163.com)

Highlights

本周新的三个 Java 模块，分别是 openmldb-common、openmldb-batchjob 和 openmldb-taskmanager。通过可重用的 Java 模块抽象，可以快速实现新的子模块。BatchJob 模块和 TaskManager 实现了第一个最小功能集版本，提供了基础的批处理任务管理功能。

本周通过拓展 SQL 语法解析器和物理计划优化器，实现了在线执行引擎的全表分组聚合和过滤功能，此功能难点在于需要在 SQL 语法解析阶段就区分 SQL 中的 UDF 和 UDAF/UDTF 函数，并最终生成对应的逻辑计划和优化后的物理计划。

欢迎更多开发者关注和参与OpenMLDB开源项目。