Add kafka-connect extension from Jet contrib repo [HZ-1805 HZ-1808] #23312

ldziedziul · 2023-01-04T20:11:26Z

Adds old kafka-connect extension along with unit tests. Integration tests will be added in the next PR

Fixes https://hazelcast.atlassian.net/browse/HZ-1805

extensions/kafka-connect/README.md

extensions/kafka-connect/src/main/java/com/hazelcast/jet/kafka/connect/KafkaConnectSources.java

...ka-connect/src/main/java/com/hazelcast/jet/kafka/connect/impl/SourceOffsetStorageReader.java

...nect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectRandomIntIntegrationTest.java

...ons/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectSourcesTest.java

...s/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/impl/DummySourceConnector.java

hazelcast/src/test/java/com/hazelcast/internal/util/PreconditionsTest.java

...ons/kafka-connect/src/main/java/com/hazelcast/jet/kafka/connect/impl/KafkaConnectSource.java

ldziedziul · 2023-01-23T22:16:07Z

I don't like adding binaries (e.g. jars) into the repository until there is no other option. They could contain whatever malicious code. Could we prepare the connect-connector classes from sources or reference the artifact from the Maven central repo?
I didn't complete the review yet, but I'm failing it fast so we can discuss it or work on the improvements.

The connector are unavailable in the maven repository. There are sources we can build them but it will take time and complicate the build. Maybe we could add this jar to our jfrog repository like here: https://github.com/hazelcast/hazelcast/pull/23312/files#diff-563cf06ae27c7192b68fb111982ff428996336712840ac47dd50a39f09df44e8R53

hz-devops-test · 2023-01-23T22:50:52Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:license-maven-plugin:2.0.0:add-third-party (add-third-party) on project hazelcast-jet-elasticsearch-7: could not init goal AddThirdPartyMojo for reason : null: ConcurrentModificationException -> [Help 1]
--------------------------

ldziedziul · 2023-01-24T09:53:53Z

run-lab-run

kwart

Maybe we could make a simple replacement for the Random connector on our own.

kwart · 2023-01-26T09:25:49Z

...a-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java

+    private static final String CONNECTOR_URL = "https://hazelcast.jfrog.io/artifactory/download-local"
+            + "/tests/confluentinc-kafka-connect-jdbc-10.6.3.zip";


I think it'll be a blocker for Viridian, but please doublecheck with the legal dep.

kwart · 2023-01-26T09:34:08Z

...nect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectRandomIntIntegrationTest.java

+        JobConfig jobConfig = new JobConfig();
+        jobConfig.addJar(Objects.requireNonNull(this.getClass()
+                        .getClassLoader()
+                        .getResource("random-connector-1.0-SNAPSHOT.jar"))


And if I'm looking into the project correctly, we are out of luck.

The https://github.com/sasakitoa/kafka-connect-random project does neither have the license filled in the metadata (GH or POM) nor has it included within the sources. We can't use it.

hz-devops-test · 2023-02-07T13:03:55Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.2.1:checkstyle (default) on project hazelcast-jet-kafka-connect: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectDatagenIntegrationTest.java:47: Line is longer than 123 characters (found 155). [LineLength]
--------------------------

hz-devops-test · 2023-02-07T13:07:14Z

The job Hazelcast-pr-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.2.1:checkstyle (default) on project hazelcast-jet-kafka-connect: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.38 with /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/kafka-connect/../../checkstyle/checkstyle_jet.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-compiler/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectDatagenIntegrationTest.java:47: Line is longer than 123 characters (found 155). [LineLength]
--------------------------

...kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/impl/KafkaConnectSourceTest.java

...onnect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectDatagenIntegrationTest.java

...a-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java

hz-devops-test · 2023-02-08T11:02:03Z

The job Hazelcast-pr-builder of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] COMPILATION ERROR : 
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,110] illegal forward reference
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project hazelcast-jet-kafka-connect: Compilation failure: Compilation failure: 
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,110] illegal forward reference
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,87] illegal forward reference
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,110] illegal forward reference
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-builder_2/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,87] illegal forward reference
--------------------------

hz-devops-test · 2023-02-08T11:03:58Z

The job Hazelcast-pr-EE-compiler of your PR failed. (Hazelcast internal details: build log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
---------SUMMARY----------
--------------------------
[ERROR] COMPILATION ERROR : 
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-EE-compiler/hazelcast/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,110] illegal forward reference
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (default-testCompile) on project hazelcast-jet-kafka-connect: Compilation failure: Compilation failure: 
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-EE-compiler/hazelcast/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,110] illegal forward reference
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-EE-compiler/hazelcast/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,87] illegal forward reference
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-EE-compiler/hazelcast/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,110] illegal forward reference
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/Hazelcast-pr-EE-compiler/hazelcast/extensions/kafka-connect/src/test/java/com/hazelcast/jet/kafka/connect/KafkaConnectJdbcIntegrationTest.java:[52,87] illegal forward reference
--------------------------

ldziedziul added Source: Internal PR or issue was opened by an employee Type: Feature Request Add to Release Notes Team: Integration labels Jan 4, 2023

ldziedziul added this to the 5.3.0 milestone Jan 4, 2023

ldziedziul self-assigned this Jan 4, 2023

ldziedziul added 2 commits January 4, 2023 21:12

Add old kafka-connect extension [HZ-1805]

cfe3601

Change package name

44b7028

ldziedziul force-pushed the import-kafka-connect branch from a2afb2c to 44b7028 Compare January 4, 2023 20:12

ldziedziul added 8 commits January 5, 2023 11:29

Remove shading warnings

cee5d4f

Remove redundant abstraction

a062e85

Fix thread interruption

c9d9929

Add tests for SourceOffsetStorageReader

61755cf

Add precondition for properties

1d90761

Add tests to KafkaConnectSources

2cd2895

Small cleanup

0983fc8

Add KafkaConnectSource tests

e4a9bf2

ldziedziul force-pushed the import-kafka-connect branch from 7f983c2 to e4a9bf2 Compare January 12, 2023 10:24

ldziedziul requested review from TomaszGaweda, frant-hartm and orcunc January 12, 2023 11:43

ldziedziul marked this pull request as ready for review January 12, 2023 11:43

hazelcast deleted a comment from hz-devops-test Jan 12, 2023

TomaszGaweda reviewed Jan 13, 2023

View reviewed changes

ldziedziul added 4 commits January 13, 2023 13:50

Use static imports

80b1028

Fix README formatting

6c84dc4

Fix artifact in README

aea297b

Add nonnull annotation

8a7fbb9

ldziedziul added 3 commits January 23, 2023 18:50

Add Beta annotation

1280e90

Simplify shading

c125aa9

Remove redundant sleep in tests

49b5282

ldziedziul requested review from kwart and frant-hartm January 23, 2023 22:16

ldziedziul added 2 commits January 24, 2023 11:57

Use recompiled connector jar from jfrog repository

c504932

Make sonar happy

9a943b8

kwart requested changes Jan 26, 2023

View reviewed changes

ldziedziul added 2 commits February 7, 2023 13:36

Use Datagen connector for integration tests

1aacd3d

Avoid constant duplicates

c079437

ldziedziul requested a review from kwart February 7, 2023 12:57

frant-hartm approved these changes Feb 7, 2023

View reviewed changes

kwart reviewed Feb 7, 2023

View reviewed changes

ldziedziul added 4 commits February 8, 2023 11:28

Use official repository url for test connectors

f1bfb48

Remove redundant assertion

5f6b5af

Rename test methods

2042a3e

Use JUnit Rule for enabling logging

b8cdf9d

ldziedziul requested a review from kwart February 8, 2023 10:58

Fix variable order

06bd456

kwart approved these changes Feb 8, 2023

View reviewed changes

ldziedziul merged commit 3743802 into hazelcast:master Feb 8, 2023

ldziedziul deleted the import-kafka-connect branch February 8, 2023 12:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kafka-connect extension from Jet contrib repo [HZ-1805 HZ-1808] #23312

Add kafka-connect extension from Jet contrib repo [HZ-1805 HZ-1808] #23312

ldziedziul commented Jan 4, 2023 •

edited

ldziedziul commented Jan 23, 2023

hz-devops-test commented Jan 23, 2023

ldziedziul commented Jan 24, 2023

kwart left a comment

kwart Jan 26, 2023

kwart Jan 26, 2023

hz-devops-test commented Feb 7, 2023

hz-devops-test commented Feb 7, 2023

hz-devops-test commented Feb 8, 2023

hz-devops-test commented Feb 8, 2023

		private static final String CONNECTOR_URL = "https://hazelcast.jfrog.io/artifactory/download-local"
		+ "/tests/confluentinc-kafka-connect-jdbc-10.6.3.zip";

Add kafka-connect extension from Jet contrib repo [HZ-1805 HZ-1808] #23312

Add kafka-connect extension from Jet contrib repo [HZ-1805 HZ-1808] #23312

Conversation

ldziedziul commented Jan 4, 2023 • edited

ldziedziul commented Jan 23, 2023

hz-devops-test commented Jan 23, 2023

ldziedziul commented Jan 24, 2023

kwart left a comment

Choose a reason for hiding this comment

kwart Jan 26, 2023

Choose a reason for hiding this comment

kwart Jan 26, 2023

Choose a reason for hiding this comment

hz-devops-test commented Feb 7, 2023

hz-devops-test commented Feb 7, 2023

hz-devops-test commented Feb 8, 2023

hz-devops-test commented Feb 8, 2023

ldziedziul commented Jan 4, 2023 •

edited