diff --git a/README.md b/README.md index 30a1aadaa..3192f67dd 100644 --- a/README.md +++ b/README.md @@ -20,12 +20,12 @@ Click it if your want to know more about our design. 👉🏻[Design](https://gi ## Compatibility -| SeaTunnel Web Version | SeaTunnel Version | Doc | -|-----------------------|------------------------|------------------------------------------------------------------------------| -| 1.0.3-SNAPSHOT | 2.3.8 / 2.3.9-SNAPSHOT | [Docs](https://github.com/apache/seatunnel-web/blob/main/README.md) | -| 1.0.2 | 2.3.8 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.2-release/README.md) | -| 1.0.1 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.1-release/README.md) | -| 1.0.0 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.0-release/README.md) | +| SeaTunnel Web Version | SeaTunnel Version | Doc | +|-----------------------|--------------------------|------------------------------------------------------------------------------| +| 1.0.3-SNAPSHOT | 2.3.10 / 2.3.11-SNAPSHOT | [Docs](https://github.com/apache/seatunnel-web/blob/main/README.md) | +| 1.0.2 | 2.3.8 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.2-release/README.md) | +| 1.0.1 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.1-release/README.md) | +| 1.0.0 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.0-release/README.md) | ## How to start @@ -52,16 +52,16 @@ You have two ways to get the SeaTunnel installer package. Build from source code * Get the source package from https://seatunnel.apache.org/download or https://github.com/apache/seatunnel.git * Please follow [Build SeaTunnel From Source Code](https://seatunnel.apache.org/docs/start-v2/locally/deployment#build-seatunnel-from-source-code) to build the SeaTunnel. * After building, it is necessary to set an environment variable `ST_WEB_BASEDIR_PATH` to represent the location of the data source shade package. A custom class loader will be used to load the data source shade package based on this. For example: `ST_WEB_BASEDIR_PATH=/seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/` -* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.8-bin.tar.gz` -* Run `tar -zxvf apache-seatunnel-2.3.8-bin.tar.gz` to unzip the installer package. -* Run `cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server. +* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.10-bin.tar.gz` +* Run `tar -zxvf apache-seatunnel-2.3.10-bin.tar.gz` to unzip the installer package. +* Run `cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server. * Please confirm that port 5801 is being monitored by the SeaTunnelServer process. ##### 2.1.2 Download installer package and deploy The other way to install SeaTunnel Zeta Engine Server is download the installer package from https://seatunnel.apache.org/download and deploy. * Download and install connector plugin(Some third-party dependency packages will also be automatically downloaded and installed during this process, such as hadoop jar). You can get the step from https://seatunnel.apache.org/docs/start-v2/locally/deployment. -* Run `cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server. +* Run `cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server. #### 2.2 Init database @@ -86,7 +86,7 @@ sh build.sh code 1. Edit `seatunnel-server/seatunnel-app/src/main/resources/application.yml` Fill in the database connection information ![img.png](docs/images/application_config.png) 2. Edit `seatunnel-server/seatunnel-app/src/main/resources/application.yml`, add `jwt.secretKey` value. Eg: https://github.com/apache/seatunnel(Notice that cannot be too short). -3. Copy `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` file to `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` dir. +3. Copy `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` file to `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` dir. 4. Run `seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/SeatunnelApplication.java` If there are no errors reported, the seatunnel web backend service is successfully started. Notice that, you must set `-DSEATUNNEL_HOME=${your_seatunnel_install_path}` like this: ![img.png](docs/images/idea_st_home.png) @@ -127,21 +127,21 @@ You have two ways to get the SeaTunnel installer package. Build from source code ##### 3.1.1 Build from source code * Get the source package from https://seatunnel.apache.org/download or https://github.com/apache/seatunnel.git * Build installer package use maven command `./mvnw -U -T 1C clean install -DskipTests -D"maven.test.skip"=true -D"maven.javadoc.skip"=true -D"checkstyle.skip"=true -D"license.skipAddThirdParty" ` -* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.8-bin.tar.gz` +* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.10-bin.tar.gz` ##### 3.1.2 Download installer package The other way to get SeaTunnel Zeta Engine Server installer package is download the installer package from https://seatunnel.apache.org/download and install plugins online. * Download and install connector plugin(Some third-party dependency packages will also be automatically downloaded and installed during this process, such as hadoop jar). You can get the step from https://seatunnel.apache.org/docs/start-v2/locally/deployment. -* After completing the previous step, you will receive an installation package that can be used to install SeaTunnel Zeta Engine Server on the server. Run `tar -zcvf apache-seatunnel-2.3.8-bin.tar.gz apache-seatunnel-2.3.8` +* After completing the previous step, you will receive an installation package that can be used to install SeaTunnel Zeta Engine Server on the server. Run `tar -zcvf apache-seatunnel-2.3.10-bin.tar.gz apache-seatunnel-2.3.10` ##### 3.1.3 Deploy SeaTunnel Zeta Server -After 3.1.1 or 3.1.2 you can get an installer package `apache-seatunnel-2.3.8-bin.tar.gz`, Then you can copy it to you server node and deploy reference https://seatunnel.apache.org/docs/seatunnel-engine/deployment. +After 3.1.1 or 3.1.2 you can get an installer package `apache-seatunnel-2.3.10-bin.tar.gz`, Then you can copy it to you server node and deploy reference https://seatunnel.apache.org/docs/seatunnel-engine/deployment. ##### 3.1.4 Deploy SeaTunnel Zeta Client In SeaTunnel Web Run Node If you use SeaTunnel Web, you need deploy a SeaTunnel Zeta Client in the SeaTunnel Web run Node. **If you run SeaTunnel Zeta Server and SeaTunnel Web in same node, you can skip this step**. -* Copy `apache-seatunnel-2.3.8-bin.tar.gz` to the SeaTunnel Web node and unzip it **in the same path of SeaTunnel Zeta Server node**. +* Copy `apache-seatunnel-2.3.10-bin.tar.gz` to the SeaTunnel Web node and unzip it **in the same path of SeaTunnel Zeta Server node**. * Set `SEATUNNEL_HOME` to environment variable like SeaTunnel Zeta Server node. * Config `hazelcast-client.yaml` reference https://seatunnel.apache.org/docs/seatunnel-engine/deployment#6-config-seatunnel-engine-client * Run `$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template`, If this job run finished, it indicates successful client deployment. @@ -183,7 +183,7 @@ tar -zxvf apache-seatunnel-web-${project.version}.tar.gz ![image](docs/images/application_config.png) * Copy `$SEATUNNEL_HOME/config/hazelcast-client.yaml` to `apache-seatunnel-web-${project.version}/conf/` -* Copy `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` file to `apache-seatunnel-web-${project.version}/conf/` dir. +* Copy `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` file to `apache-seatunnel-web-${project.version}/conf/` dir. #### 3.6 Start SeaTunnel Web diff --git a/README_CN.md b/README_CN.md index 309cec9f6..5f9388b45 100644 --- a/README_CN.md +++ b/README_CN.md @@ -42,16 +42,16 @@ SeaTunnel是下一代超高性能、分布式、海量数据集成工具。它 * 从 https://seatunnel.apache.org/download 或 https://github.com/apache/seatunnel.git 获取源包 * 请按照 [从源码构建 SeaTunnel 来构建 SeaTunnel](https://seatunnel.apache.org/zh-CN/docs/start-v2/locally/deployment/#从源码构建seatunnel)。 * 在构建之后，需要设置一个环境变量`ST_WEB_BASEDIR_PATH`来表示数据源shade包的位置。将使用自定义类加载器来基于此加载数据源shade包。例如:`ST_WEB_BASEDIR_PATH=/seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/` -然后你可以在`${Your_code_dir}/seatunnel-dist/target`下获取安装包，例如:`apache-seatunnel-2.3.8-bin.tar.gz`。 -* 执行`tar -zxvf apache-seatunnel-2.3.8-bin.tar.gz`解压安装包。 -* 运行`cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。 +然后你可以在`${Your_code_dir}/seatunnel-dist/target`下获取安装包，例如:`apache-seatunnel-2.3.10-bin.tar.gz`。 +* 执行`tar -zxvf apache-seatunnel-2.3.10-bin.tar.gz`解压安装包。 +* 运行`cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。 * SeaTunnel Zeta Engine Server默认占用5801端口，请确认端口5801正在被SeaTunnelServer进程占用。 ##### 2.1.2 下载安装程序包并进行部署 下载安装包并部署SeaTunnel Zeta Engine Server的另一种安装方式是从 https://seatunnel.apache.org/download 下载安装包并部署。 -* 下载并安装连接器插件(一些第三方依赖包也会在此过程中自动下载并安装，如Hadoop jar)。您可以从 https://seatunnel.apache.org/docs/2.3.8/start-v2/locally/deployment 获得该步骤。 -* 运行`cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。 +* 下载并安装连接器插件(一些第三方依赖包也会在此过程中自动下载并安装，如Hadoop jar)。您可以从 https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/deployment 获得该步骤。 +* 运行`cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。 #### 2.2 初始化数据库 1. 编辑 `seatunnel-server/seatunnel-app/src/main/resources/script/seatunnel_server_env.sh` 文件, 填写已安装的数据库 address, port, username, and password. 下面是一个例子: @@ -77,7 +77,7 @@ sh build.sh code ![img.png](docs/images/application_config.png) 2. 编辑 `seatunnel-server/seatunnel-app/src/main/resources/application.yml` 添加 `jwt.secretKey` 值. 例如: `https://github.com/apache/seatunnel` (注意: 不能太短). -3. 复制 `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` 文件 到 `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` 目录. +3. 复制 `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` 文件 到 `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` 目录. 4. 运行 `seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/SeatunnelApplication.java` 如果没有报错，说明seatunnel web后端服务启动成功。注意，你必须设置 `-DSEATUNNEL_HOME=${your_seatunnel_install_path}` 像这样: ![img.png](docs/images/idea_st_home.png) @@ -118,23 +118,23 @@ npm run dev ##### 3.1.1 从源码编译 * 从 https://seatunnel.apache.org/download 或 https://github.com/apache/seatunnel.git 获取源码包 * 使用maven命令编译安装包 `./mvnw -U -T 1C clean install -DskipTests -D"maven.test.skip"=true -D"maven.javadoc.skip"=true -D"checkstyle.skip"=true -D"license.skipAddThirdParty" ` -* 然后您可以在`${您的代码目录}/seatunnel-dist/target`中获得安装包,例如:`apache-seatunnel-2.3.8-bin.tar.gz` +* 然后您可以在`${您的代码目录}/seatunnel-dist/target`中获得安装包,例如:`apache-seatunnel-2.3.10-bin.tar.gz` ##### 3.1.2 下载安装包 获取 SeaTunnel Zeta 引擎服务安装包的另一种方式是从 https://seatunnel.apache.org/download 下载安装包并在线安装插件。 -* 下载并安装连接器插件(这个过程中也会自动下载和安装一些第三方依赖包,比如 hadoop jar)。您可以参考 https://seatunnel.apache.org/docs/2.3.8/start-v2/locally/deployment 获取详细步骤。 -* 完成上述步骤后,您将获得一个可以用于在服务器上安装 SeaTunnel Zeta 引擎服务的安装包。运行 `tar -zcvf apache-seatunnel-2.3.8-bin.tar.gz apache-seatunnel-2.3.8` +* 下载并安装连接器插件(这个过程中也会自动下载和安装一些第三方依赖包,比如 hadoop jar)。您可以参考 https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/deployment 获取详细步骤。 +* 完成上述步骤后,您将获得一个可以用于在服务器上安装 SeaTunnel Zeta 引擎服务的安装包。运行 `tar -zcvf apache-seatunnel-2.3.10-bin.tar.gz apache-seatunnel-2.3.10` ##### 3.1.3 部署 SeaTunnel Zeta 服务端 -在完成 3.1.1 或 3.1.2 后您可以获得一个安装包 `apache-seatunnel-2.3.8-bin.tar.gz`,然后您可以将其复制到服务器节点并参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment 进行部署。 +在完成 3.1.1 或 3.1.2 后您可以获得一个安装包 `apache-seatunnel-2.3.10-bin.tar.gz`,然后您可以将其复制到服务器节点并参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment 进行部署。 ##### 3.1.4 在 SeaTunnel Web 运行节点部署 SeaTunnel Zeta 客户端 如果您使用 SeaTunnel Web,您需要在 SeaTunnel Web 运行节点部署一个 SeaTunnel Zeta 客户端。**如果 SeaTunnel Zeta 服务端和 SeaTunnel Web 在同一节点,您可以跳过此步骤**。 -* 将 `apache-seatunnel-2.3.8-bin.tar.gz` 复制到 SeaTunnel Web 节点并解压,解压**到与 SeaTunnel Zeta 服务端节点相同的路径下**。 +* 将 `apache-seatunnel-2.3.10-bin.tar.gz` 复制到 SeaTunnel Web 节点并解压,解压**到与 SeaTunnel Zeta 服务端节点相同的路径下**。 * 和 SeaTunnel Zeta 服务端节点相同,设置 `SEATUNNEL_HOME` 环境变量。 * 参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment#6-config-seatunnel-engine-client 配置 `hazelcast-client.yaml`。 * 运行 `$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template`,如果该作业运行结束,表示客户端部署成功。 @@ -176,7 +176,7 @@ tar -zxvf apache-seatunnel-web-${project.version}.tar.gz ![image](docs/images/application_config.png) * 复制 `$SEATUNNEL_HOME/config/hazelcast-client.yaml` 到 `apache-seatunnel-web-${project.version}/conf/` -* 复制 `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` 文件到 `apache-seatunnel-web-${project.version}/conf/` 目录. +* 复制 `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` 文件到 `apache-seatunnel-web-${project.version}/conf/` 目录. #### 3.6 运行 SeaTunnel Web diff --git a/pom.xml b/pom.xml index 7be252207..55fb16dc4 100644 --- a/pom.xml +++ b/pom.xml @@ -73,7 +73,7 @@ 33.2.1-jre 3.10.0 4.2.0 - 2.3.8 + 2.3.10 21.5.0.0 42.4.3 9.2.1.jre8 @@ -92,7 +92,7 @@ 2.6.8 5.3.20 5.6.5 - 3.5.3.1 + 3.5.11 1.2.9 2.6.1 1.5.10 @@ -379,6 +379,13 @@ + + com.baomidou + mybatis-plus-bom + ${mybatis-plus-boot-starter.version} + pom + import + com.baomidou mybatis-plus-boot-starter @@ -1264,6 +1271,17 @@ test + + com.baomidou + mybatis-plus-jsqlparser-4.9 + + + + com.mysql + mysql-connector-j + 8.0.33 + + diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/connector/ConnectorCache.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/connector/ConnectorCache.java index df0fc5aae..62068578d 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/connector/ConnectorCache.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/connector/ConnectorCache.java @@ -17,6 +17,7 @@ package org.apache.seatunnel.app.bean.connector; +import org.apache.seatunnel.api.common.PluginIdentifier; import org.apache.seatunnel.api.configuration.util.OptionRule; import org.apache.seatunnel.app.domain.response.connector.ConnectorFeature; import org.apache.seatunnel.app.domain.response.connector.ConnectorInfo; @@ -25,7 +26,6 @@ import org.apache.seatunnel.common.config.Common; import org.apache.seatunnel.common.config.DeployMode; import org.apache.seatunnel.common.constants.PluginType; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; import org.apache.seatunnel.server.common.SeatunnelErrorEnum; import org.apache.seatunnel.server.common.SeatunnelException; diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/env/JobEnvCache.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/env/JobEnvCache.java index c488c0e0f..3de4c5418 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/env/JobEnvCache.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/bean/env/JobEnvCache.java @@ -17,7 +17,7 @@ package org.apache.seatunnel.app.bean.env; import org.apache.seatunnel.api.configuration.util.OptionRule; -import org.apache.seatunnel.api.env.EnvOptionRule; +import org.apache.seatunnel.api.options.EnvOptionRule; import org.apache.seatunnel.app.dynamicforms.AbstractFormOption; import org.apache.seatunnel.app.dynamicforms.FormStructure; import org.apache.seatunnel.app.thirdparty.framework.SeaTunnelOptionRuleWrapper; @@ -37,7 +37,7 @@ public class JobEnvCache { @Getter private final FormStructure envFormStructure; public JobEnvCache() { - OptionRule envOptionRules = EnvOptionRule.getEnvOptionRules(); + OptionRule envOptionRules = new EnvOptionRule().optionRule(); envFormStructure = SeaTunnelOptionRuleWrapper.wrapper( envOptionRules.getOptionalOptions(), diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/dal/entity/TaskDefinition.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/dal/entity/TaskDefinition.java index 41c268309..57a483e66 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/dal/entity/TaskDefinition.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/dal/entity/TaskDefinition.java @@ -117,7 +117,7 @@ public class TaskDefinition { private TimeoutFlag timeoutFlag; /** timeout notify strategy */ - @TableField(updateStrategy = FieldStrategy.IGNORED) + @TableField(updateStrategy = FieldStrategy.NEVER) private TaskTimeoutStrategy timeoutNotifyStrategy; /** task warning time out. unit: minute */ diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/domain/response/connector/ConnectorInfo.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/domain/response/connector/ConnectorInfo.java index 615b2bf0d..2371cf4ce 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/domain/response/connector/ConnectorInfo.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/domain/response/connector/ConnectorInfo.java @@ -17,7 +17,7 @@ package org.apache.seatunnel.app.domain.response.connector; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; +import org.apache.seatunnel.api.common.PluginIdentifier; import lombok.AllArgsConstructor; import lombok.Data; diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/EngineServiceImpl.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/EngineServiceImpl.java index ac64495e0..0e66cce1c 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/EngineServiceImpl.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/EngineServiceImpl.java @@ -37,7 +37,7 @@ public class EngineServiceImpl extends SeatunnelBaseServiceImpl implements IEngi Lists.newArrayList( // new Engine("Spark", "2.4.0"), // new Engine("Flink", "1.13.6"), - new Engine(EngineType.SeaTunnel, "2.3.8"))); + new Engine(EngineType.SeaTunnel, "2.3.10"))); @Override public List listSupportEngines() { diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobDefinitionServiceImpl.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobDefinitionServiceImpl.java index 734434ff2..86b9d6b73 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobDefinitionServiceImpl.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobDefinitionServiceImpl.java @@ -97,7 +97,7 @@ public long createJob(JobReq jobReq) throws CodeGenerateUtils.CodeGenerateExcept .name(DEFAULT_VERSION) .id(uuid) .engineName(EngineType.SeaTunnel) - .engineVersion("2.3.8"); + .engineVersion("2.3.10"); if (BusinessMode.DATA_INTEGRATION.equals(jobReq.getJobType())) { builder.jobMode(JobMode.BATCH); } else if (BusinessMode.DATA_REPLICA.equals(jobReq.getJobType())) { diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobInstanceServiceImpl.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobInstanceServiceImpl.java index d0db526f7..9c21342a0 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobInstanceServiceImpl.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/JobInstanceServiceImpl.java @@ -26,9 +26,9 @@ import org.apache.seatunnel.shade.com.typesafe.config.ConfigValueFactory; import org.apache.seatunnel.shade.com.typesafe.config.ConfigValueType; -import org.apache.seatunnel.api.common.CommonOptions; import org.apache.seatunnel.api.configuration.util.OptionRule; import org.apache.seatunnel.api.env.ParsingMode; +import org.apache.seatunnel.api.options.ConnectorCommonOptions; import org.apache.seatunnel.app.bean.connector.ConnectorCache; import org.apache.seatunnel.app.config.ConnectorDataSourceMapperConfig; import org.apache.seatunnel.app.config.EncryptionConfig; @@ -223,7 +223,7 @@ public String generateJobConfig( if (inputLines.containsKey(pluginId)) { config = addTableName( - CommonOptions.RESULT_TABLE_NAME.key(), + ConnectorCommonOptions.PLUGIN_OUTPUT.key(), inputLines.get(pluginId), config); if (!sourceMap.containsKey(task.getConnectorType())) { @@ -269,14 +269,14 @@ public String generateJobConfig( if (inputLines.containsKey(pluginId)) { config = addTableName( - CommonOptions.RESULT_TABLE_NAME.key(), + ConnectorCommonOptions.PLUGIN_OUTPUT.key(), inputLines.get(pluginId), config); } if (targetLines.containsKey(pluginId)) { config = addTableName( - CommonOptions.SOURCE_TABLE_NAME.key(), + ConnectorCommonOptions.PLUGIN_INPUT.key(), targetLines.get(pluginId), config); } @@ -293,7 +293,7 @@ public String generateJobConfig( if (targetLines.containsKey(pluginId)) { config = addTableName( - CommonOptions.SOURCE_TABLE_NAME.key(), + ConnectorCommonOptions.PLUGIN_INPUT.key(), targetLines.get(pluginId), config); if (!sinkMap.containsKey(task.getConnectorType())) { diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/TableSchemaServiceImpl.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/TableSchemaServiceImpl.java index 6035379cc..159d7569e 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/TableSchemaServiceImpl.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/service/impl/TableSchemaServiceImpl.java @@ -17,6 +17,7 @@ package org.apache.seatunnel.app.service.impl; +import org.apache.seatunnel.api.common.PluginIdentifier; import org.apache.seatunnel.api.table.catalog.DataTypeConvertor; import org.apache.seatunnel.api.table.factory.DataTypeConvertorFactory; import org.apache.seatunnel.api.table.type.SeaTunnelDataType; @@ -33,7 +34,6 @@ import org.apache.seatunnel.common.constants.PluginType; import org.apache.seatunnel.common.utils.FileUtils; import org.apache.seatunnel.datasource.plugin.api.model.TableField; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; import org.apache.seatunnel.plugin.discovery.seatunnel.SeaTunnelSinkPluginDiscovery; import org.apache.seatunnel.server.common.SeatunnelErrorEnum; import org.apache.seatunnel.server.common.SeatunnelException; diff --git a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/framework/PluginDiscoveryUtil.java b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/framework/PluginDiscoveryUtil.java index 4641574b8..bdc4eefa8 100644 --- a/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/framework/PluginDiscoveryUtil.java +++ b/seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/thirdparty/framework/PluginDiscoveryUtil.java @@ -16,6 +16,7 @@ */ package org.apache.seatunnel.app.thirdparty.framework; +import org.apache.seatunnel.api.common.PluginIdentifier; import org.apache.seatunnel.api.configuration.util.OptionRule; import org.apache.seatunnel.api.source.SupportColumnProjection; import org.apache.seatunnel.api.table.factory.Factory; @@ -27,7 +28,6 @@ import org.apache.seatunnel.common.config.Common; import org.apache.seatunnel.common.constants.PluginType; import org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; import org.apache.seatunnel.plugin.discovery.seatunnel.SeaTunnelSinkPluginDiscovery; import lombok.NonNull; diff --git a/seatunnel-web-dist/release-docs/LICENSE b/seatunnel-web-dist/release-docs/LICENSE index 8642ed53d..ce858fc59 100644 --- a/seatunnel-web-dist/release-docs/LICENSE +++ b/seatunnel-web-dist/release-docs/LICENSE @@ -226,21 +226,21 @@ The text of each license is the standard Apache 2.0 license. (Apache License, Version 2.0) Apache Commons Compress (org.apache.commons:commons-compress:1.20 - https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.20) (Apache License, Version 2.0) config (com.typesafe:config:1.3.3 - https://github.com/lightbend/config) (The Apache Software License, Version 2.0) hazelcast (com.hazelcast:hazelcast:5.1 - https://github.com/hazelcast/hazelcast) - (Apache License, Version 2.0) SeaTunnel Common (org.apache.seatunnel:seatunnel-common:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-common) - (Apache License, Version 2.0) SeaTunnel API (org.apache.seatunnel:seatunnel-api:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-api) - (Apache License, Version 2.0) SeaTunnel Config Base (org.apache.seatunnel:seatunnel-config-base:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-config-base) - (Apache License, Version 2.0) SeaTunnel Config Shade (org.apache.seatunnel:seatunnel-config-shade:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-config-shade) - (Apache License, Version 2.0) SeaTunnel Core Starter (org.apache.seatunnel:seatunnel-core-starter:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-core-starter) - (Apache License, Version 2.0) SeaTunnel Core Starter (org.apache.seatunnel:seatunnel-core-starter:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-core-starter) - (Apache License, Version 2.0) SeaTunnel Engine Client (org.apache.seatunnel:seatunnel-engine-client:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-engine-client) - (Apache License, Version 2.0) SeaTunnel Engine Common (org.apache.seatunnel:seatunnel-engine-common:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-engine-common) - (Apache License, Version 2.0) SeaTunnel Engine Core (org.apache.seatunnel:seatunnel-engine-core:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-engine-core) - (Apache License, Version 2.0) SeaTunnel Guava (org.apache.seatunnel:seatunnel-guava:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-guava) - (Apache License, Version 2.0) SeaTunnel Jackson (org.apache.seatunnel:seatunnel-jackson:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-jackson) - (Apache License, Version 2.0) SeaTunnel Plugin Discovery (org.apache.seatunnel:seatunnel-plugin-discovery:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-plugin-discovery) - (Apache License, Version 2.0) SeaTunnel Transform V2 (org.apache.seatunnel:seatunnel-transforms-v2:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-transforms-v2) - (Apache License, Version 2.0) SeaTunnel Serializer Api (org.apache.seatunnel:serializer-api:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/serializer-api) - (Apache License, Version 2.0) SeaTunnel Serializer Protobuf (org.apache.seatunnel:serializer-protobuf:2.3.8 - https://mvnrepository.com/artifact/org.apache.seatunnel/serializer-protobuf) + (Apache License, Version 2.0) SeaTunnel Common (org.apache.seatunnel:seatunnel-common:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-common) + (Apache License, Version 2.0) SeaTunnel API (org.apache.seatunnel:seatunnel-api:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-api) + (Apache License, Version 2.0) SeaTunnel Config Base (org.apache.seatunnel:seatunnel-config-base:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-config-base) + (Apache License, Version 2.0) SeaTunnel Config Shade (org.apache.seatunnel:seatunnel-config-shade:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-config-shade) + (Apache License, Version 2.0) SeaTunnel Core Starter (org.apache.seatunnel:seatunnel-core-starter:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-core-starter) + (Apache License, Version 2.0) SeaTunnel Core Starter (org.apache.seatunnel:seatunnel-core-starter:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-core-starter) + (Apache License, Version 2.0) SeaTunnel Engine Client (org.apache.seatunnel:seatunnel-engine-client:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-engine-client) + (Apache License, Version 2.0) SeaTunnel Engine Common (org.apache.seatunnel:seatunnel-engine-common:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-engine-common) + (Apache License, Version 2.0) SeaTunnel Engine Core (org.apache.seatunnel:seatunnel-engine-core:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-engine-core) + (Apache License, Version 2.0) SeaTunnel Guava (org.apache.seatunnel:seatunnel-guava:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-guava) + (Apache License, Version 2.0) SeaTunnel Jackson (org.apache.seatunnel:seatunnel-jackson:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-jackson) + (Apache License, Version 2.0) SeaTunnel Plugin Discovery (org.apache.seatunnel:seatunnel-plugin-discovery:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-plugin-discovery) + (Apache License, Version 2.0) SeaTunnel Transform V2 (org.apache.seatunnel:seatunnel-transforms-v2:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-transforms-v2) + (Apache License, Version 2.0) SeaTunnel Serializer Api (org.apache.seatunnel:serializer-api:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/serializer-api) + (Apache License, Version 2.0) SeaTunnel Serializer Protobuf (org.apache.seatunnel:serializer-protobuf:2.3.10 - https://mvnrepository.com/artifact/org.apache.seatunnel/serializer-protobuf) (The Apache Software License, Version 2.0) Guava: Google Core Libraries for Java (com.google.guava:guava:19.0 - https://github.com/google/guava) (Apache License 2.0) Hibernate Validator Engine (org.hibernate.validator:hibernate-validator:6.2.2.Final - http://hibernate.org/validator/hibernate-validator) diff --git a/seatunnel-web-it/README.md b/seatunnel-web-it/README.md index 5f224d5ad..1922de3d5 100644 --- a/seatunnel-web-it/README.md +++ b/seatunnel-web-it/README.md @@ -2,15 +2,15 @@ Build seatunnel-web ./mvnw clean install -DskipTests Run seatunnel-web-it integration tests -./mvnw -T 1C -B verify -DskipUT=true -DskipIT=false -DSEATUNNEL_HOME=/some/path/apache-seatunnel-2.3.8 -DST_WEB_BASEDIR_PATH=seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/apache-seatunnel-web-1.0.3-SNAPSHOT +./mvnw -T 1C -B verify -DskipUT=true -DskipIT=false -DSEATUNNEL_HOME=/some/path/apache-seatunnel-2.3.10 -DST_WEB_BASEDIR_PATH=seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/apache-seatunnel-web-1.0.3-SNAPSHOT NOTE: Please remember to update the versions according to the latest supported versions. If you're using a version of Java higher than Java 8 for running the tests, add the following VM options: -DitJvmArgs="-Xmx1024m --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED" While running integrations tests from IDE, ensure following VM options are set -SEATUNNEL_HOME=/some/path/apache-seatunnel-2.3.8 +SEATUNNEL_HOME=/some/path/apache-seatunnel-2.3.10 ST_WEB_BASEDIR_PATH=/some/path/seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/apache-seatunnel-web-1.0.3-SNAPSHOT By default, integration tests use the H2 database. If you want to use the MySQL database, update the MySQL database details in src/test/resources/application.yml and run the seatunnel-web-it integration tests with the -DdbType=mysql option as shown below: -./mvnw -T 1C -B verify -DskipUT=true -DskipIT=false -DdbType=mysql -DSEATUNNEL_HOME=/some/path/apache-seatunnel-2.3.8 -DST_WEB_BASEDIR_PATH=seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/apache-seatunnel-web-1.0.3-SNAPSHOT +./mvnw -T 1C -B verify -DskipUT=true -DskipIT=false -DdbType=mysql -DSEATUNNEL_HOME=/some/path/apache-seatunnel-2.3.10 -DST_WEB_BASEDIR_PATH=seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/apache-seatunnel-web-1.0.3-SNAPSHOT diff --git a/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/ConnectorInfoDeserializer.java b/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/ConnectorInfoDeserializer.java index fb164e419..98dbad086 100644 --- a/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/ConnectorInfoDeserializer.java +++ b/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/ConnectorInfoDeserializer.java @@ -16,11 +16,10 @@ */ package org.apache.seatunnel.app.domain; +import org.apache.seatunnel.api.common.PluginIdentifier; import org.apache.seatunnel.app.domain.response.connector.ConnectorInfo; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; import com.fasterxml.jackson.core.JsonParser; -import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.DeserializationContext; import com.fasterxml.jackson.databind.JsonDeserializer; import com.fasterxml.jackson.databind.JsonNode; @@ -32,7 +31,7 @@ public class ConnectorInfoDeserializer extends JsonDeserializer { @Override public ConnectorInfo deserialize( JsonParser jsonParser, DeserializationContext deserializationContext) - throws IOException, JsonProcessingException { + throws IOException { JsonNode node = jsonParser.getCodec().readTree(jsonParser); JsonNode pluginIdentifierNode = node.get("pluginIdentifier"); String artifactId = node.get("artifactId").asText(); diff --git a/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/PluginIdentifierDeserializer.java b/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/PluginIdentifierDeserializer.java index 3d097aa48..665dd4b1e 100644 --- a/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/PluginIdentifierDeserializer.java +++ b/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/domain/PluginIdentifierDeserializer.java @@ -16,7 +16,7 @@ */ package org.apache.seatunnel.app.domain; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; +import org.apache.seatunnel.api.common.PluginIdentifier; import com.fasterxml.jackson.core.JsonParser; import com.fasterxml.jackson.core.JsonProcessingException; diff --git a/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/utils/JSONTestUtils.java b/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/utils/JSONTestUtils.java index dd9c146b4..74837914a 100644 --- a/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/utils/JSONTestUtils.java +++ b/seatunnel-web-it/src/test/java/org/apache/seatunnel/app/utils/JSONTestUtils.java @@ -17,13 +17,13 @@ package org.apache.seatunnel.app.utils; +import org.apache.seatunnel.api.common.PluginIdentifier; import org.apache.seatunnel.app.common.Constants; import org.apache.seatunnel.app.domain.ConnectorInfoDeserializer; import org.apache.seatunnel.app.domain.JobExecutorResDeserializer; import org.apache.seatunnel.app.domain.PluginIdentifierDeserializer; import org.apache.seatunnel.app.domain.response.connector.ConnectorInfo; import org.apache.seatunnel.app.domain.response.executor.JobExecutorRes; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; import org.apache.commons.lang3.StringUtils; diff --git a/tools/dependencies/known-dependencies.txt b/tools/dependencies/known-dependencies.txt index 2e110bef2..d53583149 100644 --- a/tools/dependencies/known-dependencies.txt +++ b/tools/dependencies/known-dependencies.txt @@ -96,8 +96,8 @@ tomcat-embed-el-9.0.63.jar h2-2.1.214.jar auto-service-annotations-1.0.1.jar jsr305-3.0.0.jar -checkpoint-storage-api-2.3.8.jar -checkpoint-storage-local-file-2.3.8.jar +checkpoint-storage-api-2.3.10.jar +checkpoint-storage-local-file-2.3.10.jar clickhouse-cli-client-0.3.2-patch11-shaded.jar clickhouse-grpc-client-0.3.2-patch11-shaded.jar clickhouse-http-client-0.3.2-patch11-shaded.jar @@ -116,22 +116,22 @@ protostuff-api-1.8.0.jar protostuff-collectionschema-1.8.0.jar protostuff-core-1.8.0.jar protostuff-runtime-1.8.0.jar -seatunnel-api-2.3.8.jar -seatunnel-common-2.3.8.jar -seatunnel-config-base-2.3.8.jar -seatunnel-config-shade-2.3.8.jar -seatunnel-core-starter-2.3.8.jar -seatunnel-engine-client-2.3.8.jar -seatunnel-engine-common-2.3.8.jar -seatunnel-engine-core-2.3.8.jar -seatunnel-guava-2.3.8-optional.jar -seatunnel-jackson-2.3.8-optional.jar -seatunnel-plugin-discovery-2.3.8.jar -seatunnel-transforms-v2-2.3.8.jar -seatunnel-config-sql-2.3.8.jar -seatunnel-hazelcast-shade-2.3.8-optional.jar -serializer-api-2.3.8.jar -serializer-protobuf-2.3.8.jar +seatunnel-api-2.3.10.jar +seatunnel-common-2.3.10.jar +seatunnel-config-base-2.3.10.jar +seatunnel-config-shade-2.3.10.jar +seatunnel-core-starter-2.3.10.jar +seatunnel-engine-client-2.3.10.jar +seatunnel-engine-common-2.3.10.jar +seatunnel-engine-core-2.3.10.jar +seatunnel-guava-2.3.10-optional.jar +seatunnel-jackson-2.3.10-optional.jar +seatunnel-plugin-discovery-2.3.10.jar +seatunnel-transforms-v2-2.3.10.jar +seatunnel-config-sql-2.3.10.jar +seatunnel-hazelcast-shade-2.3.10-optional.jar +serializer-api-2.3.10.jar +serializer-protobuf-2.3.10.jar swagger-annotations-2.2.14.jar commons-codec-1.11.jar commons-compress-1.20.jar