OpenMLDB Weekly Update(2021.9.12-2021.9.19)

OpenMLDB

Summary 

本周合并 Pull requests 12 个，新增 Pull requests 4 个，关闭 Issues 6 个，新增 Issues 18 个。总计 353 个文件修改，新增 36056 行代码，删除 879 行代码。

Merged Pull Requests

• feat: add integration test cicd#434 

• feat: add batchjob as java submodules#386 

• feat: add kubernetes java dependencies for taskmanager#400

• fix: fix count in some yaml cases#436 

• feat: add a new optimization for expanding data in window skew optimization#424 

• feat: support insert multiple rows into a table using a single SQL insert statement#399 

• feat: support aggregation over the whole table#393 

• ci: openmldb java deploy workflow#366

• test: rm DataSyncReplicaCluster test#415 

• feat: reconfiguration window skew optimization#414 

• feat: add integration test#395 

• feat: bump junit from 4.11 to 4.13.1 in /java/openmldb-batchjob#382 

Open Pull Requests

• build(deps): bump snakeyaml from 1.17 to 1.26 in /test/batch-test/openmldb-batch-test#416 

• build(deps): bump httpclient from 4.5.2 to 4.5.13 in /test/integration-test/openmldb-test-java/openmldb-test-common#417 

• feat: support in predicate#423 

• feat: reorganize error code and use check_status and check_true#435 

Close Issues

• Make openmldb-batchjob and openmldb-taskmanager as submodules of openmldb-parent#385 

• Support submit and manage Kubernetes jobs for TaskManager#375 

• Bug: SQL INSERT Statement with multi rows does not work as expected#391 

• studio.4.2.0安装的rtidb启动coredump#278 

• Support General aggegrate function over table `COUNT`, `MAX`, `MIN`, `SUM`#219 

• feat: support integration test for java/python sdk and offline batch#316 

Open Issues

• Add feature extraction tools like detecting data skew#433 

• feat: try run benchmark on GitHub workflow, compare & upload test results#432 

• feat: refactor error/warning log in hybridse#430 

• refactor yaml sql test case#427 

• feat: improve cli and make the console output more clean and clear#426 

• RFC: Redesign some interfaces of SQLClusterRouter#425 

• Create memtable when creating procedure#422 

• Sync metadata to hive metastore when creating iceberg table#421 

• Load data from iceberg to memtable#420 

• Get index from sql&procdure#419 

• Create message table and sync data to nearline tablet#418 

• Add optimization passes for native LastJoin#413 

• Enable optimization fo window parallel computation by default#412 

• Package OpenMLDB Spark distribution for release#411 

• Support table aggregation functions for Batch mode#410 

• Support passing Spark parameters for TaskManager#409 

• Refine the parameters from TaskManager API to support more job status#408 

• Integrated TaskManager API with OpenMLDB CLI tool#407 

Contributors

• aceforeverd (teapot@aceforeverd.com)

• Chen22 (jingchen2222@gmail.com)

• dl239 (dl239@126.com)

• Kanekanekane (1290561498@qq.com)

• Kanekanekane (kane@KanedeMacBook-Pro.local)

• tobe (tobeg3oogle@gmail.com)

• Wang ZeKai (1290561498@qq.com)

• wuyou10206 (zw1020688@163.com)

Highlights

本周新增加了 IntegrationTest 集成测试模块，并集成到 CICD 流程，整体代码修改较多，主要是新增了大量 SQL 测试用例。TaskManager 模块新增 Kubernetes 依赖，支持多种计算集群后端的任务管理。BatchJob 模块加入 Java 项目子模块中，纳入完整的 CICD 集成流程。本周项目正式通过 2021 年信通院可信开源项目评审，并在代码上升级 Hadoop-common 依赖版本解决潜在风险问题，修复项目 License 依赖风险问题。

欢迎更多开发者关注和参与OpenMLDB开源项目。