第八周作业

第一题，为 Spark SQL 添加一条自定义命令

• SHOW VERSION

• 显示当前 Spark 版本和 Java 版本

答：

实现过程如下，

1. 通过修改 SqlBase.g4 文件添加自定义命令，需要在原生 SqlBase.g4 文件中添加下面 4 处

1)

statement

: query

| SHOW VERSION #showVersion

2)

nonReserved

//--DEFAULT-NON-RESERVED-START

: ADD

| VERSION

3)

ansiNonReserved

//--ANSI-NON-RESERVED-START

: ADD

| VERSION

4)

//============================

// Start of the keywords list

//============================

//--SPARK-KEYWORD-LIST-START

VERSION: 'VERSION';

1. 执行 spark-catalyst maven module 的 anltr4:anltr4 goal

2. 修改 SparkSqlParser.scala 文件, 重写 visitShowVersion 方法

override def visitShowVersion(ctx: ShowVersionContext): LogicalPlan = withOrigin(ctx) {

ShowVersionCommand()

}

1. 在 commands.scala 类中增加实现类 ShowVersionCommand，代码如下，

case class ShowVersionCommand() extends LeafRunnableCommand { override val output: Seq[Attribute] = Seq(AttributeReference("version", StringType)()) override def run(sparkSession: SparkSession): Seq[Row] = { val sparkVersion = sparkSession.version val javaVersion = System.getProperty("java.version") val output = "Spark Version : %s, Java Version: %s".format(sparkVersion, javaVersion) Seq(Row(output)) } }

1. 在 spark 源码根目录下执行命令"./build/sbt package -DskipTests -Phive -Phive-thriftserver"

2. 将 SPARK_HOME 环境变量设置成 spark 源码根目录，然后执行./bin/spark-sql，进入 spark-sql 控制台之后，执行 show version 命令，过程如下图，

第二题，构建 SQL 满足如下要求通过 set spark.sql.planChangeLog.level=WARN;查看

1. 构建一条 SQL，同时 apply 下面三条优化规则：

CombineFilters

CollapseProject

BooleanSimplification

2. 构建一条 SQL，同时 apply 下面五条优化规则：

ConstantFolding

PushDownPredicates

ReplaceDistinctWithAggregate

ReplaceExceptWithAntiJoin

FoldablePropagation

答：

第一条 sql 如下，

SELECT a11, (a2 + 1) AS a21 FROM ( SELECT (a1 + 1) AS a11, a2 FROM t1 WHERE a1 > 10 ) WHERE a11 > 1 AND 1 = 1;

日志：

22/05/07 23:07:31 WARN PlanChangeLogger: Batch Union has no effect.22/05/07 23:07:31 WARN PlanChangeLogger: Batch OptimizeLimitZero has no effect.22/05/07 23:07:31 WARN PlanChangeLogger: Batch LocalRelation early has no effect.22/05/07 23:07:31 WARN PlanChangeLogger: Batch Pullup Correlated Expressions has no effect.22/05/07 23:07:31 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:07:31 WARN PlanChangeLogger: Batch Replace Operators has no effect.22/05/07 23:07:31 WARN PlanChangeLogger: Batch Aggregate has no effect.22/05/07 23:07:31 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushDownPredicates ===Project [a11#21, (a2#24 + 1) AS a21#22] Project [a11#21, (a2#24 + 1) AS a21#22]!+- Filter ((a11#21 > 1) AND (1 = 1)) +- Project [(a1#23 + 1) AS a11#21, a2#24]! +- Project [(a1#23 + 1) AS a11#21, a2#24] +- Filter ((a1#23 > 10) AND (((a1#23 + 1) > 1) AND (1= 1)))! +- Filter (a1#23 > 10) +- Relation default.t1[a1#23,a2#24] parquet! +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:31 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.CollapseProject ===!Project [a11#21, (a2#24 + 1) AS a21#22] Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]!+- Project [(a1#23 + 1) AS a11#21, a2#24] +- Filter ((a1#23 > 10) AND (((a1#23 + 1) > 1) AND (1 = 1)))! +- Filter ((a1#23 > 10) AND (((a1#23 + 1) > 1) AND (1 = 1))) +- Relation default.t1[a1#23,a2#24] parquet! +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:31 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.ConstantFolding ===Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22] Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]!+- Filter ((a1#23 > 10) AND (((a1#23 + 1) > 1) AND (1 = 1))) +- Filter ((a1#23 > 10) AND (((a1#23 + 1) > 1) AND true))+- Relation default.t1[a1#23,a2#24] parquet +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:31 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.BooleanSimplification ===Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22] Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]!+- Filter ((a1#23 > 10) AND (((a1#23 + 1) > 1) AND true)) +- Filter ((a1#23 > 10) AND ((a1#23 + 1) > 1))+- Relation default.t1[a1#23,a2#24] parquet +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:32 WARN PlanChangeLogger:=== Result of Batch Operator Optimization before Inferring Filters ===!Project [a11#21, (a2#24 + 1) AS a21#22] Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]!+- Filter ((a11#21 > 1) AND (1 = 1)) +- Filter ((a1#23 > 10) AND ((a1#23 + 1) > 1))! +- Project [(a1#23 + 1) AS a11#21, a2#24] +- Relation default.t1[a1#23,a2#24] parquet! +- Filter (a1#23 > 10)! +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:32 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints ===Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22] Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

!+- Filter ((a1#23 > 10) AND ((a1#23 + 1) > 1)) +- Filter (isnotnull(a1#23) AND ((a1#23 > 10) AND ((a1#23 + 1) > 1)))+- Relation default.t1[a1#23,a2#24] parquet +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:32 WARN PlanChangeLogger:=== Result of Batch Infer Filters ===Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22] Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

!+- Filter ((a1#23 > 10) AND ((a1#23 + 1) > 1)) +- Filter (isnotnull(a1#23) AND ((a1#23 > 10) AND ((a1#23 + 1) > 1)))+- Relation default.t1[a1#23,a2#24] parquet +- Relation default.t1[a1#23,a2#24] parquet

22/05/07 23:07:32 WARN PlanChangeLogger: Batch Operator Optimization after Inferring Filters has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Push extra predicate through join has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Early Filter and Projection Push-Down has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Update CTE Relation Stats has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Join Reorder has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Eliminate Sorts has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Decimal Optimizations has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Distinct Aggregate Rewrite has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Object Expressions Optimization has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch LocalRelation has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Check Cartesian Products has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch RewriteSubquery has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch NormalizeFloatingNumbers has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch ReplaceUpdateFieldsExpression has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Optimize Metadata Only Query has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch PartitionPruning has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Pushdown Filters from PartitionPruning has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Cleanup filters that cannot be pushed down has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch Extract Python UDFs has no effect.22/05/07 23:07:32 WARN PlanChangeLogger: Batch User Provided Optimizers has no effect.22/05/07 23:07:32 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 212Total time: 0.1281794 secondsTotal number of effective runs: 6Total time of effective runs: 0.0363412 seconds

22/05/07 23:07:32 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions ===Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

         Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

+- Filter ((isnotnull(a1#23) AND (a1#23 > 10)) AND ((a1#23 + 1) > 1))

         +- Filter ((isnotnull(a1#23) AND (a1#23 > 10)) AND ((a1#23 + 1) > 1))

! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a1#23), (a1#23 > 10), ((a1#23 + 1) > 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a1), GreaterThan(a1,10)], ReadSchema: struct<a1:int,a2:int> +- ColumnarToRow!

               +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a1#23), (a

1#23 > 10), ((a1#23 + 1) > 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a1), GreaterThan(a1,10)], ReadSchema: structa1:int,a2:int

22/05/07 23:07:32 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.CollapseCodegenStages ===!Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

            *(1) Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

!+- Filter ((isnotnull(a1#23) AND (a1#23 > 10)) AND ((a1#23 + 1) > 1))

            +- *(1) Filter ((isnotnull(a1#23) AND (a1#23 > 10)) AND ((a1#23 + 1) > 1))

! +- ColumnarToRow

               +- *(1) ColumnarToRow   +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a1#23), (a1#23 > 10),

((a1#23 + 1) > 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a1), GreaterThan(a1,10)], ReadSchema: struct<a1:int,a2:int> +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a1#23),(a1#23 > 10), ((a1#23 + 1) > 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a1), GreaterThan(a1,10)], ReadSchema: structa1:int,a2:int

22/05/07 23:07:32 WARN PlanChangeLogger:=== Result of Batch Preparations ===!Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

         *(1) Project [(a1#23 + 1) AS a11#21, (a2#24 + 1) AS a21#22]

!+- Filter ((isnotnull(a1#23) AND (a1#23 > 10)) AND ((a1#23 + 1) > 1))

         +- *(1) Filter ((isnotnull(a1#23) AND (a1#23 > 10)) AND ((a1#23 + 1) > 1))

! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a1#23), (a1#23 > 10), ((a1#23 + 1) > 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a1), GreaterThan(a1,10)], ReadSchema: struct<a1:int,a2:int> +- *(1) ColumnarToRow!

               +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a1#23), (a

1#23 > 10), ((a1#23 + 1) > 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a1), GreaterThan(a1,10)], ReadSchema: structa1:int,a2:int

22/05/07 23:07:32 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:07:32 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 1.3E-5 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:07:32 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:07:32 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 1.36E-5 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:07:32 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:07:32 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 1.23E-5 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:07:32 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:07:32 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 1.62E-5 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

Time taken: 2.35 seconds

第二条 sql 如下

CREATE TABLE t2(b1 INT, b2 INT) USING parquet; SELECT DISTINCT a1, a2, 'custom' a3 FROM ( SELECT * FROM t1 WHERE a2 = 10 AND 1 = 1 ) WHERE a1 > 5 AND 1 = 1 EXCEPT SELECT b1, b2, 1.0 b3 FROM t2 WHERE b2 = 10 ;

日志：

spark-sql> CREATE TABLE t2(b1 INT, b2 INT) USING parquet;22/05/07 23:19:34 WARN PlanChangeLogger: Batch Substitution has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Disable Hints has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Hints has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Simple Sanity Check has no effect.22/05/07 23:19:34 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.analysis.ResolveSessionCatalog ===!'CreateTableStatement [t2], [StructField(b1,IntegerType,true), StructField(b2,IntegerType,true)], parquet, false, false 'CreateTable default.t2, ErrorIfExists

22/05/07 23:19:34 WARN PlanChangeLogger:=== Result of Batch Resolution ===!'CreateTableStatement [t2], [StructField(b1,IntegerType,true), StructField(b2,IntegerType,true)], parquet, false, false 'CreateTable default.t2, ErrorIfExists

22/05/07 23:19:34 WARN PlanChangeLogger: Batch Remove TempResolvedColumn has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Apply Char Padding has no effect.22/05/07 23:19:34 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.datasources.DataSourceAnalysis ===!'CreateTable default.t2, ErrorIfExists CreateDataSourceTableCommand default.t2, false

22/05/07 23:19:34 WARN PlanChangeLogger:=== Result of Batch Post-Hoc Resolution ===!'CreateTable default.t2, ErrorIfExists CreateDataSourceTableCommand default.t2, false

22/05/07 23:19:34 WARN PlanChangeLogger: Batch Remove Unresolved Hints has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Nondeterministic has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch UDF has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch UpdateNullability has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Cleanup has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch HandleAnalysisOnlyCommand has no effect.22/05/07 23:19:34 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 133Total time: 0.0084025 secondsTotal number of effective runs: 2Total time of effective runs: 0.003528 seconds

22/05/07 23:19:34 WARN PlanChangeLogger: Batch Eliminate Distinct has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Finish Analysis has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Union has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch OptimizeLimitZero has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch LocalRelation early has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Pullup Correlated Expressions has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Replace Operators has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Aggregate has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Operator Optimization before Inferring Filters has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Infer Filters has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Operator Optimization after Inferring Filters has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Push extra predicate through join has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Early Filter and Projection Push-Down has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Update CTE Relation Stats has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Join Reorder has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Eliminate Sorts has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Decimal Optimizations has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Distinct Aggregate Rewrite has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Object Expressions Optimization has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch LocalRelation has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Check Cartesian Products has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch RewriteSubquery has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch NormalizeFloatingNumbers has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch ReplaceUpdateFieldsExpression has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Optimize Metadata Only Query has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch PartitionPruning has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Pushdown Filters from PartitionPruning has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Cleanup filters that cannot be pushed down has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch Extract Python UDFs has no effect.22/05/07 23:19:34 WARN PlanChangeLogger: Batch User Provided Optimizers has no effect.22/05/07 23:19:34 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 165Total time: 0.0016627 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:34 WARN PlanChangeLogger: Batch Preparations has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Substitution has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Disable Hints has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Hints has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Simple Sanity Check has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Resolution has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Remove TempResolvedColumn has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Apply Char Padding has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Post-Hoc Resolution has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Remove Unresolved Hints has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Nondeterministic has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch UDF has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch UpdateNullability has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Cleanup has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch HandleAnalysisOnlyCommand has no effect.22/05/07 23:19:35 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 80Total time: 0.0014476 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:35 WARN PlanChangeLogger: Batch Eliminate Distinct has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Finish Analysis has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Union has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch OptimizeLimitZero has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch LocalRelation early has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Pullup Correlated Expressions has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Replace Operators has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Aggregate has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Operator Optimization before Inferring Filters has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Infer Filters has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Operator Optimization after Inferring Filters has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Push extra predicate through join has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Early Filter and Projection Push-Down has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Update CTE Relation Stats has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Join Reorder has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Eliminate Sorts has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Decimal Optimizations has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Distinct Aggregate Rewrite has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Object Expressions Optimization has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch LocalRelation has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Check Cartesian Products has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch RewriteSubquery has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch NormalizeFloatingNumbers has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch ReplaceUpdateFieldsExpression has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Optimize Metadata Only Query has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch PartitionPruning has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Pushdown Filters from PartitionPruning has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Cleanup filters that cannot be pushed down has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Extract Python UDFs has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch User Provided Optimizers has no effect.22/05/07 23:19:35 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 165Total time: 0.0013135 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:35 WARN PlanChangeLogger: Batch Preparations has no effect.Time taken: 0.797 secondsspark-sql> SELECT DISTINCT a1, a2, 'custom' a3> FROM (> SELECT * FROM t1 WHERE a2 = 10 AND 1 = 1> ) WHERE a1 > 5 AND 1 = 1> EXCEPT SELECT b1, b2, 1.0 b3 FROM t2 WHERE b2 = 10 ;22/05/07 23:19:35 WARN PlanChangeLogger: Batch Substitution has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Disable Hints has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Hints has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Simple Sanity Check has no effect.22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations ==='Except false 'Except false:- 'Distinct :- 'Distinct: +- 'Project ['a1, 'a2, custom AS a3#27] : +- 'Project ['a1, 'a2, custom AS a3#27]: +- 'Filter (('a1 > 5) AND (1 = 1)) : +- 'Filter (('a1 > 5) AND (1 = 1)): +- 'SubqueryAlias __auto_generated_subquery_name : +- 'SubqueryAlias __auto_generated_subquery_name: +- 'Project [] : +- 'Project []: +- 'Filter (('a2 = 10) AND (1 = 1)) : +- 'Filter (('a2 = 10) AND (1 = 1))!: +- 'UnresolvedRelation [t1], [], false : +- 'SubqueryAlias spark_catalog.default.t1!+- 'Project ['b1, 'b2, 1.0 AS b3#28] : +- 'UnresolvedCatalogRelation default.t1, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, [], false! +- 'Filter ('b2 = 10) +- 'Project ['b1, 'b2, 1.0 AS b3#28]! +- 'UnresolvedRelation [t2], [], false +- 'Filter ('b2 = 10)! +- 'SubqueryAlias spark_catalog.default.t2! +- 'UnresolvedCatalogRelation default.t2, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, [], false

22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.datasources.FindDataSourceTable ==='Except false 'Except false:- 'Distinct :- 'Distinct: +- 'Project ['a1, 'a2, custom AS a3#27] : +- 'Project ['a1, 'a2, custom AS a3#27]: +- 'Filter (('a1 > 5) AND (1 = 1)) : +- 'Filter (('a1 > 5) AND (1 = 1)): +- 'SubqueryAlias __auto_generated_subquery_name : +- 'SubqueryAlias __auto_generated_subquery_name: +- 'Project [] : +- 'Project []: +- 'Filter (('a2 = 10) AND (1 = 1)) : +- 'Filter (('a2 = 10) AND (1 = 1))!: +- 'SubqueryAlias spark_catalog.default.t1 : +- SubqueryAlias spark_catalog.default.t1!: +- 'UnresolvedCatalogRelation default.t1, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, [], false : +- Relation default.t1[a1#23,a2#24] parquet+- 'Project ['b1, 'b2, 1.0 AS b3#28] +- 'Project ['b1, 'b2, 1.0 AS b3#28]+- 'Filter ('b2 = 10) +- 'Filter ('b2 = 10)! +- 'SubqueryAlias spark_catalog.default.t2 +- SubqueryAlias spark_catalog.default.t2! +- 'UnresolvedCatalogRelation default.t2, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, [], false +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences ==='Except false 'Except false!:- 'Distinct :- Distinct!: +- 'Project ['a1, 'a2, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]!: +- 'Filter (('a1 > 5) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- 'SubqueryAlias __auto_generated_subquery_name : +- SubqueryAlias __auto_generated_subquery_name!: +- 'Project [*] : +- Project [a1#23, a2#24]!: +- 'Filter (('a2 = 10) AND (1 = 1)) : +- Filter ((a2#24 = 10) AND (1 = 1)): +- SubqueryAlias spark_catalog.default.t1 : +- SubqueryAlias spark_catalog.default.t1: +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet!+- 'Project ['b1, 'b2, 1.0 AS b3#28] +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- 'Filter ('b2 = 10) +- Filter (b2#30 = 10)+- SubqueryAlias spark_catalog.default.t2 +- SubqueryAlias spark_catalog.default.t2+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.analysis.TypeCoercionBase$WidenSetOperationTypes ===!'Except false Except false!:- Distinct :- Project [a1#23, a2#24, a3#27]!: +- Project [a1#23, a2#24, custom AS a3#27] : +- Distinct!: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Project [a1#23, a2#24, custom AS a3#27]!: +- SubqueryAlias __auto_generated_subquery_name : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- Project [a1#23, a2#24] : +- SubqueryAlias __auto_generated_subquery_name!: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Project [a1#23, a2#24]!: +- SubqueryAlias spark_catalog.default.t1 : +- Filter ((a2#24 = 10) AND (1 = 1))!: +- Relation default.t1[a1#23,a2#24] parquet : +- SubqueryAlias spark_catalog.default.t1!+- Project [b1#29, b2#30, 1.0 AS b3#28] : +- Relation default.t1[a1#23,a2#24] parquet! +- Filter (b2#30 = 10) +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- SubqueryAlias spark_catalog.default.t2 +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Relation default.t2[b1#29,b2#30] parquet +- Filter (b2#30 = 10)! +- SubqueryAlias spark_catalog.default.t2! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Result of Batch Resolution ===!'Except false Except false!:- 'Distinct :- Project [a1#23, a2#24, a3#27]!: +- 'Project ['a1, 'a2, custom AS a3#27] : +- Distinct!: +- 'Filter (('a1 > 5) AND (1 = 1)) : +- Project [a1#23, a2#24, custom AS a3#27]!: +- 'SubqueryAlias __auto_generated_subquery_name : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- 'Project [*] : +- SubqueryAlias __auto_generated_subquery_name!: +- 'Filter (('a2 = 10) AND (1 = 1)) : +- Project [a1#23, a2#24]!: +- 'UnresolvedRelation [t1], [], false : +- Filter ((a2#24 = 10) AND (1 = 1))!+- 'Project ['b1, 'b2, 1.0 AS b3#28] : +- SubqueryAlias spark_catalog.default.t1! +- 'Filter ('b2 = 10) : +- Relation default.t1[a1#23,a2#24] parquet! +- 'UnresolvedRelation [t2], [], false +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Filter (b2#30 = 10)! +- SubqueryAlias spark_catalog.default.t2! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger: Batch Remove TempResolvedColumn has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Apply Char Padding has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Post-Hoc Resolution has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Remove Unresolved Hints has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Nondeterministic has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch UDF has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch UpdateNullability has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.analysis.CleanupAliases ===Except false Except false:- Project [a1#23, a2#24, a3#27] :- Project [a1#23, a2#24, a3#27]: +- Distinct : +- Distinct: +- Project [a1#23, a2#24, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1)): +- SubqueryAlias __auto_generated_subquery_name : +- SubqueryAlias __auto_generated_subquery_name: +- Project [a1#23, a2#24] : +- Project [a1#23, a2#24]: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Filter ((a2#24 = 10) AND (1 = 1)): +- SubqueryAlias spark_catalog.default.t1 : +- SubqueryAlias spark_catalog.default.t1: +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]+- Project [b1#29, b2#30, 1.0 AS b3#28] +- Project [b1#29, b2#30, 1.0 AS b3#28]+- Filter (b2#30 = 10) +- Filter (b2#30 = 10)+- SubqueryAlias spark_catalog.default.t2 +- SubqueryAlias spark_catalog.default.t2+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Result of Batch Cleanup ===Except false Except false:- Project [a1#23, a2#24, a3#27] :- Project [a1#23, a2#24, a3#27]: +- Distinct : +- Distinct: +- Project [a1#23, a2#24, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1)): +- SubqueryAlias __auto_generated_subquery_name : +- SubqueryAlias __auto_generated_subquery_name: +- Project [a1#23, a2#24] : +- Project [a1#23, a2#24]: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Filter ((a2#24 = 10) AND (1 = 1)): +- SubqueryAlias spark_catalog.default.t1 : +- SubqueryAlias spark_catalog.default.t1: +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]+- Project [b1#29, b2#30, 1.0 AS b3#28] +- Project [b1#29, b2#30, 1.0 AS b3#28]+- Filter (b2#30 = 10) +- Filter (b2#30 = 10)+- SubqueryAlias spark_catalog.default.t2 +- SubqueryAlias spark_catalog.default.t2+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger: Batch HandleAnalysisOnlyCommand has no effect.22/05/07 23:19:35 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 187Total time: 0.1352776 secondsTotal number of effective runs: 5Total time of effective runs: 0.1161735 seconds

22/05/07 23:19:35 WARN PlanChangeLogger: Batch Eliminate Distinct has no effect.22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases ===Except false Except false:- Project [a1#23, a2#24, a3#27] :- Project [a1#23, a2#24, a3#27]: +- Distinct : +- Distinct: +- Project [a1#23, a2#24, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- SubqueryAlias __auto_generated_subquery_name : +- Project [a1#23, a2#24]!: +- Project [a1#23, a2#24] : +- Filter ((a2#24 = 10) AND (1 = 1))!: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Relation default.t1[a1#23,a2#24] parquet!: +- SubqueryAlias spark_catalog.default.t1 +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]!: +- Relation default.t1[a1#23,a2#24] parquet +- Project [b1#29, b2#30, 1.0 AS b3#28]!+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Filter (b2#30 = 10)! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Relation default.t2[b1#29,b2#30] parquet! +- Filter (b2#30 = 10)! +- SubqueryAlias spark_catalog.default.t2! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Result of Batch Finish Analysis ===Except false Except false:- Project [a1#23, a2#24, a3#27] :- Project [a1#23, a2#24, a3#27]: +- Distinct : +- Distinct: +- Project [a1#23, a2#24, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- SubqueryAlias __auto_generated_subquery_name : +- Project [a1#23, a2#24]!: +- Project [a1#23, a2#24] : +- Filter ((a2#24 = 10) AND (1 = 1))!: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Relation default.t1[a1#23,a2#24] parquet!: +- SubqueryAlias spark_catalog.default.t1 +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]!: +- Relation default.t1[a1#23,a2#24] parquet +- Project [b1#29, b2#30, 1.0 AS b3#28]!+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Filter (b2#30 = 10)! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Relation default.t2[b1#29,b2#30] parquet! +- Filter (b2#30 = 10)! +- SubqueryAlias spark_catalog.default.t2! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators ===Except false Except false!:- Project [a1#23, a2#24, a3#27] :- Distinct!: +- Distinct : +- Project [a1#23, a2#24, custom AS a3#27]!: +- Project [a1#23, a2#24, custom AS a3#27] : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a2#24 = 10) AND (1 = 1))!: +- Project [a1#23, a2#24] : +- Relation default.t1[a1#23,a2#24] parquet!: +- Filter ((a2#24 = 10) AND (1 = 1)) +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]!: +- Relation default.t1[a1#23,a2#24] parquet +- Project [b1#29, b2#30, 1.0 AS b3#28]!+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Filter (b2#30 = 10)! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Relation default.t2[b1#29,b2#30] parquet! +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger:=== Result of Batch Union ===Except false Except false!:- Project [a1#23, a2#24, a3#27] :- Distinct!: +- Distinct : +- Project [a1#23, a2#24, custom AS a3#27]!: +- Project [a1#23, a2#24, custom AS a3#27] : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a2#24 = 10) AND (1 = 1))!: +- Project [a1#23, a2#24] : +- Relation default.t1[a1#23,a2#24] parquet!: +- Filter ((a2#24 = 10) AND (1 = 1)) +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]!: +- Relation default.t1[a1#23,a2#24] parquet +- Project [b1#29, b2#30, 1.0 AS b3#28]!+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Filter (b2#30 = 10)! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Relation default.t2[b1#29,b2#30] parquet! +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:35 WARN PlanChangeLogger: Batch OptimizeLimitZero has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch LocalRelation early has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Pullup Correlated Expressions has no effect.22/05/07 23:19:35 WARN PlanChangeLogger: Batch Subquery has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.ReplaceExceptWithAntiJoin ===!Except false Distinct!:- Distinct +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31))!: +- Project [a1#23, a2#24, custom AS a3#27] :- Distinct!: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Project [a1#23, a2#24, custom AS a3#27]!: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- Relation default.t1[a1#23,a2#24] parquet : +- Filter ((a2#24 = 10) AND (1 = 1))!+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] : +- Relation default.t1[a1#23,a2#24] parquet! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- Filter (b2#30 = 10) +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Relation default.t2[b1#29,b2#30] parquet +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.ReplaceDistinctWithAggregate ===!Distinct Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]+- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31)) +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31))! :- Distinct :- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]: +- Project [a1#23, a2#24, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1)): +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Filter ((a2#24 = 10) AND (1 = 1)): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]+- Project [b1#29, b2#30, 1.0 AS b3#28] +- Project [b1#29, b2#30, 1.0 AS b3#28]+- Filter (b2#30 = 10) +- Filter (b2#30 = 10)+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch Replace Operators ===!Except false Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]!:- Distinct +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31))!: +- Project [a1#23, a2#24, custom AS a3#27] :- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]!: +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Project [a1#23, a2#24, custom AS a3#27]!: +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Filter ((a1#23 > 5) AND (1 = 1))!: +- Relation default.t1[a1#23,a2#24] parquet : +- Filter ((a2#24 = 10) AND (1 = 1))!+- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] : +- Relation default.t1[a1#23,a2#24] parquet! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- Filter (b2#30 = 10) +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Relation default.t2[b1#29,b2#30] parquet +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger: Batch Aggregate has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushDownPredicates ===Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]+- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31)) +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31)):- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] :- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]: +- Project [a1#23, a2#24, custom AS a3#27] : +- Project [a1#23, a2#24, custom AS a3#27]! : +- Filter ((a1#23 > 5) AND (1 = 1)) : +- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5))! : +- Filter ((a2#24 = 10) AND (1 = 1)) : +- Relation default.t1[a1#23,a2#24] parquet! : +- Relation default.t1[a1#23,a2#24] parquet +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Filter (b2#30 = 10)! +- Filter (b2#30 = 10) +- Relation default.t2[b1#29,b2#30] parquet! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin ===Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]!+- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31)) +- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]! :- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] +- Project [a1#23, a2#24, custom AS a3#27]! : +- Project [a1#23, a2#24, custom AS a3#27] +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (custom <=> b3#31))! : +- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5)) :- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5))! : +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet! +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Filter (b2#30 = 10) +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.CollapseProject ===Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]+- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] +- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27]+- Project [a1#23, a2#24, custom AS a3#27] +- Project [a1#23, a2#24, custom AS a3#27]+- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (custom <=> b3#31)) +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (custom <=> b3#31)):- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5)) :- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5)): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet! +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31] +- Project [b1#29, b2#30, cast(1.0 as string) AS b3#31]! +- Project [b1#29, b2#30, 1.0 AS b3#28] +- Filter (b2#30 = 10)! +- Filter (b2#30 = 10) +- Relation default.t2[b1#29,b2#30] parquet! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.FoldablePropagation ===!Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] +- Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Project [a1#23, a2#24, custom AS a3#27] +- Project [a1#23, a2#24, custom AS a3#27]! +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (custom <=> b3#31)) +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (custom <=> cast(1.0 as string))):- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5)) :- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5)): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project [b1#29, b2#30, cast(1.0 as string) AS b3#31] +- Project [b1#29, b2#30, cast(1.0 as string) AS b3#31]+- Filter (b2#30 = 10) +- Filter (b2#30 = 10)+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.ConstantFolding ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] +- Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Project [a1#23, a2#24, custom AS a3#27] +- Project [a1#23, a2#24, custom AS a3#27]! +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (custom <=> cast(1.0 as string))) +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND false)! :- Filter (((a2#24 = 10) AND (1 = 1)) AND (a1#23 > 5)) :- Filter (((a2#24 = 10) AND true) AND (a1#23 > 5)): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet! +- Project [b1#29, b2#30, cast(1.0 as string) AS b3#31] +- Project [b1#29, b2#30, 1.0 AS b3#31]+- Filter (b2#30 = 10) +- Filter (b2#30 = 10)+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.BooleanSimplification ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] +- Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Project [a1#23, a2#24, custom AS a3#27] +- Project [a1#23, a2#24, custom AS a3#27]! +- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND false) +- Join LeftAnti, false! :- Filter (((a2#24 = 10) AND true) AND (a1#23 > 5)) :- Filter ((a2#24 = 10) AND (a1#23 > 5)): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project [b1#29, b2#30, 1.0 AS b3#31] +- Project [b1#29, b2#30, 1.0 AS b3#31]+- Filter (b2#30 = 10) +- Filter (b2#30 = 10)+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAggregates ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] +- Project [a1#23, a2#24, custom AS a3#27]! +- Project [a1#23, a2#24, custom AS a3#27] +- Join LeftAnti, false! +- Join LeftAnti, false :- Filter ((a2#24 = 10) AND (a1#23 > 5))! :- Filter ((a2#24 = 10) AND (a1#23 > 5)) : +- Relation default.t1[a1#23,a2#24] parquet! : +- Relation default.t1[a1#23,a2#24] parquet +- Project [b1#29, b2#30, 1.0 AS b3#31]! +- Project [b1#29, b2#30, 1.0 AS b3#31] +- Filter (b2#30 = 10)! +- Filter (b2#30 = 10) +- Relation default.t2[b1#29,b2#30] parquet! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.ColumnPruning ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Project [a1#23, a2#24, custom AS a3#27] +- Project [a1#23, a2#24]! +- Join LeftAnti, false +- Project [a1#23, a2#24]! :- Filter ((a2#24 = 10) AND (a1#23 > 5)) +- Join LeftAnti, false! : +- Relation default.t1[a1#23,a2#24] parquet :- Filter ((a2#24 = 10) AND (a1#23 > 5))! +- Project [b1#29, b2#30, 1.0 AS b3#31] : +- Relation default.t1[a1#23,a2#24] parquet! +- Filter (b2#30 = 10) +- Project! +- Relation default.t2[b1#29,b2#30] parquet +- Project! +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.CollapseProject ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Project [a1#23, a2#24] +- Project [a1#23, a2#24]! +- Project [a1#23, a2#24] +- Join LeftAnti, false! +- Join LeftAnti, false :- Filter ((a2#24 = 10) AND (a1#23 > 5))! :- Filter ((a2#24 = 10) AND (a1#23 > 5)) : +- Relation default.t1[a1#23,a2#24] parquet! : +- Relation default.t1[a1#23,a2#24] parquet +- Project! +- Project +- Filter (b2#30 = 10)! +- Project +- Relation default.t2[b1#29,b2#30] parquet! +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.RemoveNoopOperators ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Project [a1#23, a2#24] +- Join LeftAnti, false! +- Join LeftAnti, false :- Filter ((a2#24 = 10) AND (a1#23 > 5))! :- Filter ((a2#24 = 10) AND (a1#23 > 5)) : +- Relation default.t1[a1#23,a2#24] parquet! : +- Relation default.t1[a1#23,a2#24] parquet +- Project! +- Project +- Filter (b2#30 = 10)! +- Filter (b2#30 = 10) +- Relation default.t2[b1#29,b2#30] parquet! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch Operator Optimization before Inferring Filters ===!Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Join LeftAnti, (((a1#23 <=> b1#29) AND (a2#24 <=> b2#30)) AND (a3#27 <=> b3#31)) +- Join LeftAnti, false! :- Aggregate [a1#23, a2#24, a3#27], [a1#23, a2#24, a3#27] :- Filter ((a2#24 = 10) AND (a1#23 > 5))! : +- Project [a1#23, a2#24, custom AS a3#27] : +- Relation default.t1[a1#23,a2#24] parquet! : +- Filter ((a1#23 > 5) AND (1 = 1)) +- Project! : +- Filter ((a2#24 = 10) AND (1 = 1)) +- Filter (b2#30 = 10)! : +- Relation default.t1[a1#23,a2#24] parquet +- Relation default.t2[b1#29,b2#30] parquet! +- Project [b1#29, b2#30, cast(b3#28 as string) AS b3#31]! +- Project [b1#29, b2#30, 1.0 AS b3#28]! +- Filter (b2#30 = 10)! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Join LeftAnti, false +- Join LeftAnti, false! :- Filter ((a2#24 = 10) AND (a1#23 > 5)) :- Filter ((isnotnull(a2#24) AND isnotnull(a1#23)) AND ((a2#24 = 10) AND (a1#23 > 5))): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project +- Project! +- Filter (b2#30 = 10) +- Filter (isnotnull(b2#30) AND (b2#30 = 10))+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch Infer Filters ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]+- Join LeftAnti, false +- Join LeftAnti, false! :- Filter ((a2#24 = 10) AND (a1#23 > 5)) :- Filter ((isnotnull(a2#24) AND isnotnull(a1#23)) AND ((a2#24 = 10) AND (a1#23 > 5))): +- Relation default.t1[a1#23,a2#24] parquet : +- Relation default.t1[a1#23,a2#24] parquet+- Project +- Project! +- Filter (b2#30 = 10) +- Filter (isnotnull(b2#30) AND (b2#30 = 10))+- Relation default.t2[b1#29,b2#30] parquet +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger: Batch Operator Optimization after Inferring Filters has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Push extra predicate through join has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Early Filter and Projection Push-Down has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Update CTE Relation Stats has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Join Reorder has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Eliminate Sorts has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Decimal Optimizations has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Distinct Aggregate Rewrite has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Object Expressions Optimization has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PropagateEmptyRelation ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Join LeftAnti, false +- Filter ((isnotnull(a2#24) AND isnotnull(a1#23)) AND ((a2#24 = 10) AND (a1#23 > 5)))! :- Filter ((isnotnull(a2#24) AND isnotnull(a1#23)) AND ((a2#24 = 10) AND (a1#23 > 5))) +- Relation default.t1[a1#23,a2#24] parquet! : +- Relation default.t1[a1#23,a2#24] parquet! +- Project! +- Filter (isnotnull(b2#30) AND (b2#30 = 10))! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch LocalRelation ===Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27] Aggregate [a1#23, a2#24, custom], [a1#23, a2#24, custom AS a3#27]!+- Join LeftAnti, false +- Filter ((isnotnull(a2#24) AND isnotnull(a1#23)) AND ((a2#24 = 10) AND (a1#23 > 5)))! :- Filter ((isnotnull(a2#24) AND isnotnull(a1#23)) AND ((a2#24 = 10) AND (a1#23 > 5))) +- Relation default.t1[a1#23,a2#24] parquet! : +- Relation default.t1[a1#23,a2#24] parquet! +- Project! +- Filter (isnotnull(b2#30) AND (b2#30 = 10))! +- Relation default.t2[b1#29,b2#30] parquet

22/05/07 23:19:36 WARN PlanChangeLogger: Batch Check Cartesian Products has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch RewriteSubquery has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch NormalizeFloatingNumbers has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch ReplaceUpdateFieldsExpression has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Optimize Metadata Only Query has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch PartitionPruning has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Pushdown Filters from PartitionPruning has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Cleanup filters that cannot be pushed down has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Extract Python UDFs has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch User Provided Optimizers has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 269Total time: 0.1163372 secondsTotal number of effective runs: 16Total time of effective runs: 0.0797053 seconds

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.RemoveRedundantProjects ===HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- Project [a1#23, a2#24]+- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))! +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.exchange.EnsureRequirements ===HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])!+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#62]! +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch AQE Preparations ===HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])!+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#62]! +- Project [a1#23, a2#24]+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters:[IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan ===!HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])AdaptiveSparkPlan isFinalPlan=false!+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[],output=[a1#23, a2#24, a3#27])! +- Project [a1#23, a2#24]+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200),ENSURE_REQUIREMENTS, [id=#62]! +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch Preparations ===!HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])AdaptiveSparkPlan isFinalPlan=false!+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[],output=[a1#23, a2#24, a3#27])! +- Project [a1#23, a2#24]+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200),ENSURE_REQUIREMENTS, [id=#62]! +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger: Batch AQE Query Stage Optimization has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions ===!Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#62]Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#74]+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24= 10)) AND (a1#23 > 5))! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- ColumnarToRow!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.CollapseCodegenStages ===!Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#74]Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]!+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35],functions=[], output=[a1#23, a2#24, custom#35])! +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND(a2#24 = 10)) AND (a1#23 > 5))! +- ColumnarToRow+- *(1) ColumnarToRow+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters:[IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch AQE Post Stage Creation ===!Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#62]Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]!+- HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- *(1) ColumnarToRow!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 1.03E-5 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:36 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 7.2E-6 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:36 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 8.2E-6 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:36 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 6.9E-6 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:36 WARN PlanChangeLogger: Batch Propagate Empty Relations has no effect.22/05/07 23:19:36 WARN PlanChangeLogger: Batch Dynamic Join Selection has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 4Total time: 0.0024112 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:36 WARN PlanChangeLogger: Batch AQE Replanning has no effect.22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.adaptive.CoalesceShufflePartitions ===HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])!+- ShuffleQueryStage 0+- AQEShuffleRead coalesced! +- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]+- ShuffleQueryStage 0! +- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]! +- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) HashAggregate(keys=[a1#23, a2#24, customAS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- *(1) ColumnarToRow+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- *(1) ColumnarToRow!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [],PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch AQE Query Stage Optimization ===HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])!+- ShuffleQueryStage 0+- AQEShuffleRead coalesced! +- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]+- ShuffleQueryStage 0! +- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]! +- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) HashAggregate(keys=[a1#23, a2#24, customAS custom#35], functions=[], output=[a1#23, a2#24, custom#35])! +- *(1) ColumnarToRow+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))! +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- *(1) ColumnarToRow!+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [],PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Applying Rule org.apache.spark.sql.execution.CollapseCodegenStages ===!HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])*(2) HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])+- AQEShuffleRead coalesced+- AQEShuffleRead coalesced+- ShuffleQueryStage 0+- ShuffleQueryStage 0+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]+- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) ColumnarToRow+- *(1) ColumnarToRow+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:36 WARN PlanChangeLogger:=== Result of Batch AQE Post Stage Creation ===!HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])*(2) HashAggregate(keys=[a1#23, a2#24, custom#35], functions=[], output=[a1#23, a2#24, a3#27])+- AQEShuffleRead coalesced+- AQEShuffleRead coalesced+- ShuffleQueryStage 0+- ShuffleQueryStage 0+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]+- Exchange hashpartitioning(a1#23, a2#24, custom#35, 200), ENSURE_REQUIREMENTS, [id=#80]+- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- *(1) HashAggregate(keys=[a1#23, a2#24, custom AS custom#35], functions=[], output=[a1#23, a2#24, custom#35])+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) Filter (((isnotnull(a2#24) AND isnotnull(a1#23)) AND (a2#24 = 10)) AND (a1#23 > 5))+- *(1) ColumnarToRow+- *(1) ColumnarToRow+- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int +- FileScan parquet default.t1[a1#23,a2#24] Batched: true, DataFilters: [isnotnull(a2#24), isnotnull(a1#23), (a2#24 = 10), (a1#23 > 5)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/C:/work/code/spark-3.2.1/bin/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(a2), IsNotNull(a1), EqualTo(a2,10), GreaterThan(a1,5)], ReadSchema: structa1:int,a2:int

22/05/07 23:19:37 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:37 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 1.11E-5 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:37 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:37 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 9.8E-6 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

22/05/07 23:19:37 WARN PlanChangeLogger: Batch CleanExpressions has no effect.22/05/07 23:19:37 WARN PlanChangeLogger:=== Metrics of Executed Rules ===Total number of runs: 1Total time: 7.4E-6 secondsTotal number of effective runs: 0Total time of effective runs: 0.0 seconds

Time taken: 1.693 seconds

第三题：

## 作业三：实现自定义优化规则（静默规则）

1. 第一步：实现自定义规则 (静默规则，通过 set spark.sql.planChangeLog.level=WARN，确认执行到就行)

class MyPushDown(spark: SparkSession) extends Rule[LogicalPlan] {  println("new self defined rule")  def apply(plan: LogicalPlan): LogicalPlan =  {    plan  }}

1. 第二步：创建自己的 Extension 并注入

class MySparkSessionExtension extends (SparkSessionExtensions => Unit) {

override def apply(extensions: SparkSessionExtensions): Unit = {

extensions.injectOptimizerRule { session =>

new MyPushDown(session)

}

}

}

1. 第三步：通过 spark.sql.extensions 提交

bin/spark-sql --jars my.jar --conf spark.sql.extensions=com.chloe.MySparkSessionExtension