Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51413

Implementation of producer-consumer basic communication

    XMLWordPrintable

    Details

    • Sprint:
      GSoC - Coding Phase 1
    • Similar Issues:

      Description

      Option 1:

      • In KafkaComputerLauncher invoke SlaveComputer#setChannel()
        • Current interface: InputStream in, OutputStream out, TaskListener taskListener, Channel.Listener listener
        • We need need to produce KafkaRemotingOutputStream(KafkaProducer) and KafkaRemotingInputStream(KafkaConsumer)
        • It is not efficient, but should be a good start

      Option 2:

      • Patches in Jenkins core may be needed (unhardcode Channel Builder)
      • CommandTransport for Remoting Kafka. Get Kafka Consumer and Producer connected into it
      • RemotingKafkaChannelBuilder
      • In KafkaComputerLauncher invoke SlaveComputer#setChannel()
        • Current interface: InputStream in, OutputStream out, TaskListener taskListener, Channel.Listener listener
        • We need need to produce KafkaRemotingOutputStream(KafkaProducer) and KafkaRemotingInputStream(KafkaConsumer)
        • It is not efficient, but should be a good start

      Core patch would be needed here:

      • Support passing ChannelBuilder
      public void setChannel(InputStream in, OutputStream out, OutputStream launchLog, Channel.Listener listener) throws IOException, InterruptedException {
              ChannelBuilder cb = new ChannelBuilder(nodeName,threadPoolForRemoting)
                  .withMode(Channel.Mode.NEGOTIATE)
                  .withHeaderStream(launchLog);
      
              for (ChannelConfigurator cc : ChannelConfigurator.all()) {
                  cc.onChannelBuilding(cb,this);
              }
      
              Channel channel = cb.build(in,out);
              setChannel(channel,launchLog,listener);
          }
      

      Option 3. Create KafkaComputer and maybe copy-paste some code?

      • Close to option 2, but without patching the core
      • KafkaCommandTransport as in Phase 2
      • KafkaChannelBuilder as in Option 2, which creates channel with KafkaCommandTransport
      • Creates a new Computer type (Kafka Agent), which is fine
        • "KafkaPermanentAgent" as a name?
        • It can inherit SlaveComputer
        • We need to replace setChannel() implementation by one using KafkaCommandTransport (see below)
          • It overrides old stream-based implementations and throw IOException there
      • Oleg: We need to check that TcpAgentListener won't dare to invoke the agent if somebody tries to connect. Oleg to check it. It is for the next phase, no need to do it now
      • Connection: KafkaComputerLauncher invokes the new setChannel() implementation
      public void setChannel(OutputStream launchLog, Channel.Listener listener) throws IOException, InterruptedException {
              ChannelBuilder cb = new KafkaChannelBuilder(nodeName, threadPoolForRemoting)
                  .withMode(Channel.Mode.NEGOTIATE)
                  .withHeaderStream(launchLog);
      
              for (ChannelConfigurator cc : ChannelConfigurator.all()) {
                  cc.onChannelBuilding(cb,this);
              }
      
              Channel channel = cb.build(in,out);
              setChannel(channel, launchLog, listener);
          }
      

        Attachments

          Issue Links

            Activity

            Show
            pvtuan10 Pham Vu Tuan added a comment - https://github.com/jenkinsci/remoting-kafka-plugin/pull/2
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Pham Vu Tuan
            Path:
            .gitignore
            agent/pom.xml
            agent/run.sh
            agent/src/main/java/io/jenkins/plugins/remotingkafka/Agent.java
            agent/src/main/java/io/jenkins/plugins/remotingkafka/Options.java
            commands.sh
            kafka-client-lib/pom.xml
            kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConstants.java
            kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConsumerPool.java
            kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaProducerClient.java
            plugin/pom.xml
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration.java
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration.java
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher.java
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteArrayCommandTransport.java
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteBufferCommandTransport.java
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaChunkedCommandTransport.java
            plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaClassicCommandTransport.java
            plugin/src/main/resources/index.jelly
            plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration/config.jelly
            plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration/config.jelly
            plugin/src/main/resources/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher/config.jelly
            plugin/src/main/resources/io/jenkins/plugins/remotingkafka/Messages.properties
            pom.xml
            http://jenkins-ci.org/commit/remoting-kafka-plugin/64e5ce02a2e8d8c01ca82a0cf89ddb59cc87d760
            Log:
            [JENKINS-51413, JENKINS-51414] Implement basic master-agent communication, master command invocation (#2)

            • Implement basic master-agent communication
            • Command transport implementation for Kafka
            • Implement kafka producer and consumer connection pool
            • Reorganize and fix build problem
            • Refactor packagings

            *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

            Functionality will be removed from GitHub.com on January 31st, 2019.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Pham Vu Tuan Path: .gitignore agent/pom.xml agent/run.sh agent/src/main/java/io/jenkins/plugins/remotingkafka/Agent.java agent/src/main/java/io/jenkins/plugins/remotingkafka/Options.java commands.sh kafka-client-lib/pom.xml kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConstants.java kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConsumerPool.java kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaProducerClient.java plugin/pom.xml plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteArrayCommandTransport.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteBufferCommandTransport.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaChunkedCommandTransport.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaClassicCommandTransport.java plugin/src/main/resources/index.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration/config.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration/config.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher/config.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/Messages.properties pom.xml http://jenkins-ci.org/commit/remoting-kafka-plugin/64e5ce02a2e8d8c01ca82a0cf89ddb59cc87d760 Log: [JENKINS-51413, JENKINS-51414] Implement basic master-agent communication, master command invocation (#2) Implement basic master-agent communication Command transport implementation for Kafka Implement kafka producer and consumer connection pool Reorganize and fix build problem Refactor packagings * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.
            Hide
            pvtuan10 Pham Vu Tuan added a comment -

            Oleg Nenashev I think for option 3 we still need to include option 1, which passes KafkaInputStream (extends InputStream) and KafkaOutputStream (OutputStream) as parameters for setChannel() method. Am I correct?

            Show
            pvtuan10 Pham Vu Tuan added a comment - Oleg Nenashev I think for option 3 we still need to include option 1, which passes KafkaInputStream (extends InputStream) and KafkaOutputStream (OutputStream) as parameters for setChannel() method. Am I correct?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Notes:

            • we agreed to start with option 3 for now
            • Benefit: you get some command diagnostics in Kafka
            Show
            oleg_nenashev Oleg Nenashev added a comment - Notes: we agreed to start with option 3 for now Benefit: you get some command diagnostics in Kafka

              People

              • Assignee:
                pvtuan10 Pham Vu Tuan
                Reporter:
                pvtuan10 Pham Vu Tuan
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: