Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2142

FATAL: Java heap space in Perforce Plugin

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: perforce-plugin
    • Labels:
      None
    • Environment:
      Platform: PC, OS: Windows XP
    • Similar Issues:

      Description

      During Hudson build I receive the following error
      ------
      [workspace] $ p4 sync -f //depot/Synchro/dev/main/...#head
      FATAL: Java heap space
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Unknown Source)
      at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
      at java.lang.AbstractStringBuilder.append(Unknown Source)
      at java.lang.StringBuilder.append(Unknown Source)
      at
      com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:194)
      at com.tek42.perforce.parse.Workspaces.syncTo(Workspaces.java:90)
      at com.tek42.perforce.parse.Workspaces.syncToHead(Workspaces.java:71)
      at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:318)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:596)
      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:251)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:225)
      at hudson.model.Run.run(Run.java:784)
      at hudson.model.Build.run(Build.java:85)
      at hudson.model.ResourceController.execute(ResourceController.java:70)
      at hudson.model.Executor.run(Executor.java:88)
      ------
      Hudson is currently running with CATALINA_OPTS="-DHUDSON_HOME=C:\.hudson
      -Xmx2048M -Xms2048M"

      Are there any recommend settings for the Perforce plugin?

        Attachments

          Activity

          fowlesp fowlesp created issue -
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          Un-assigning this from digerata per request by him.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - Un-assigning this from digerata per request by him.
          Hide
          javadude Carl Quinn added a comment -

          It won't be possible to debug or fix this without more information about the Perforce depot being sync'd. My guess is that the plugin is getting a gigantic response back from p4 in this case and there isn't enough memory to handle it.

          I'll close this issue as CNR if no one objects.

          Show
          javadude Carl Quinn added a comment - It won't be possible to debug or fix this without more information about the Perforce depot being sync'd. My guess is that the plugin is getting a gigantic response back from p4 in this case and there isn't enough memory to handle it. I'll close this issue as CNR if no one objects.
          Hide
          rpetti Rob Petti added a comment -

          No objection.

          Show
          rpetti Rob Petti added a comment - No objection.
          rpetti Rob Petti made changes -
          Field Original Value New Value
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Incomplete [ 4 ]
          Hide
          buej buej added a comment -

          We have the same problem, and yes we try to sync a very large project (our whole 1,7 GB development branch).
          Sometimes it works, sometimes it doesn't. If you need anymore information, contact me.

          Show
          buej buej added a comment - We have the same problem, and yes we try to sync a very large project (our whole 1,7 GB development branch). Sometimes it works, sometimes it doesn't. If you need anymore information, contact me.
          buej buej made changes -
          Resolution Incomplete [ 4 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          rpetti Rob Petti added a comment -

          @buej: how many files do you have in your view? What is your heapsize set to in the jvm running hudson?

          Show
          rpetti Rob Petti added a comment - @buej: how many files do you have in your view? What is your heapsize set to in the jvm running hudson?
          Hide
          buej buej added a comment -

          47500 files. our hudson is running in jetty webserver which is started with -Xmx1200m.

          Show
          buej buej added a comment - 47500 files. our hudson is running in jetty webserver which is started with -Xmx1200m.
          Hide
          rpetti Rob Petti added a comment -

          Can't reproduce this, even with a larger number of files and less heap space. If this is still occurring on your end, then please reopen and provide more information about your setup.

          Show
          rpetti Rob Petti added a comment - Can't reproduce this, even with a larger number of files and less heap space. If this is still occurring on your end, then please reopen and provide more information about your setup.
          rpetti Rob Petti made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Cannot Reproduce [ 5 ]
          Hide
          buej buej added a comment -

          We had this problem on 2 different hudson jobs (each bound to a different hudson-slave) and it was reproducable the time we had it (started the job a second time -> same error). After force syncing manually, the problem didn't appear anymore.
          The whole workspace folder of these jobs have been deleted before... so the force sync really had to sync everything.

          We use hudson version 1.303 and perforce plugin version 1.0.16.

          Maybe it relates to the number of changes that are described at the beginning? As you see below, the last sync'd change was 314323 and the head one was 314881... there are quite a lot big changes on this branch in between.

          Please let me know if you need to know something else (before you close the job this time).

          Started by user admin
          Building remotely on MPATEST1
          Performing sync with Perforce for: //depot/rel_3.12/...
          Changing client to REL_3.12_Hudson_tests-mpatest1
          [rel_3.12_tests_infx] $ p4 workspace -o REL_3.12_Hudson_tests-mpatest1
          Changing P4 Client Root to: E:\builds\hudson\workspace\rel_3.12_tests_infx\
          Changing P4 Client View to: //depot/rel_3.12/... //REL_3.12_Hudson_tests-mpatest1/...
          [rel_3.12_tests_infx] $ p4 -s client -i
          Last sync'd change: 314323
          [rel_3.12_tests_infx] $ p4 changes -m 25 //depot/rel_3.12/...
          [rel_3.12_tests_infx] $ p4 changes -m 25 //depot/rel_3.12/...@314716
          [rel_3.12_tests_infx] $ p4 changes -m 25 //depot/rel_3.12/...@314357
          [rel_3.12_tests_infx] $ p4 describe -s 314881
          [rel_3.12_tests_infx] $ p4 describe -s 314880
          [rel_3.12_tests_infx] $ p4 describe -s 314860
          [rel_3.12_tests_infx] $ p4 describe -s 314848
          [rel_3.12_tests_infx] $ p4 describe -s 314838
          [rel_3.12_tests_infx] $ p4 describe -s 314833
          [rel_3.12_tests_infx] $ p4 describe -s 314801
          [rel_3.12_tests_infx] $ p4 describe -s 314799
          [rel_3.12_tests_infx] $ p4 describe -s 314798
          [rel_3.12_tests_infx] $ p4 describe -s 314797
          [rel_3.12_tests_infx] $ p4 describe -s 314789
          [rel_3.12_tests_infx] $ p4 describe -s 314779
          [rel_3.12_tests_infx] $ p4 describe -s 314767
          [rel_3.12_tests_infx] $ p4 describe -s 314765
          [rel_3.12_tests_infx] $ p4 describe -s 314754
          [rel_3.12_tests_infx] $ p4 describe -s 314745
          [rel_3.12_tests_infx] $ p4 describe -s 314730
          [rel_3.12_tests_infx] $ p4 describe -s 314728
          [rel_3.12_tests_infx] $ p4 describe -s 314727
          [rel_3.12_tests_infx] $ p4 describe -s 314725
          [rel_3.12_tests_infx] $ p4 describe -s 314724
          [rel_3.12_tests_infx] $ p4 describe -s 314723
          [rel_3.12_tests_infx] $ p4 describe -s 314721
          [rel_3.12_tests_infx] $ p4 describe -s 314720
          [rel_3.12_tests_infx] $ p4 describe -s 314717
          [rel_3.12_tests_infx] $ p4 describe -s 314699
          [rel_3.12_tests_infx] $ p4 describe -s 314698
          [rel_3.12_tests_infx] $ p4 describe -s 314660
          [rel_3.12_tests_infx] $ p4 describe -s 314652
          [rel_3.12_tests_infx] $ p4 describe -s 314646
          [rel_3.12_tests_infx] $ p4 describe -s 314639
          [rel_3.12_tests_infx] $ p4 describe -s 314636
          [rel_3.12_tests_infx] $ p4 describe -s 314611
          [rel_3.12_tests_infx] $ p4 describe -s 314539
          [rel_3.12_tests_infx] $ p4 describe -s 314518
          [rel_3.12_tests_infx] $ p4 describe -s 314512
          [rel_3.12_tests_infx] $ p4 describe -s 314509
          [rel_3.12_tests_infx] $ p4 describe -s 314503
          [rel_3.12_tests_infx] $ p4 describe -s 314488
          [rel_3.12_tests_infx] $ p4 describe -s 314478
          [rel_3.12_tests_infx] $ p4 describe -s 314461
          [rel_3.12_tests_infx] $ p4 describe -s 314458
          [rel_3.12_tests_infx] $ p4 describe -s 314456
          [rel_3.12_tests_infx] $ p4 describe -s 314429
          [rel_3.12_tests_infx] $ p4 describe -s 314426
          [rel_3.12_tests_infx] $ p4 describe -s 314385
          [rel_3.12_tests_infx] $ p4 describe -s 314381
          [rel_3.12_tests_infx] $ p4 describe -s 314377
          [rel_3.12_tests_infx] $ p4 describe -s 314365
          [rel_3.12_tests_infx] $ p4 describe -s 314358
          [rel_3.12_tests_infx] $ p4 describe -s 314354
          [rel_3.12_tests_infx] $ p4 describe -s 314345
          [rel_3.12_tests_infx] $ p4 describe -s 314340
          Sync'ing workspace to depot.
          ForceSync flag is set, forcing: p4 sync //depot/rel_3.12/...
          [rel_3.12_tests_infx] $ p4 sync -f //depot/rel_3.12/...@314881
          FATAL: Java heap space
          java.lang.OutOfMemoryError: Java heap space
          	at java.util.Arrays.copyOf(Arrays.java:2882)
          	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
          	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
          	at java.lang.StringBuilder.append(StringBuilder.java:119)
          	at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:194)
          	at com.tek42.perforce.parse.Workspaces.syncTo(Workspaces.java:90)
          	at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:346)
          	at hudson.model.AbstractProject.checkout(AbstractProject.java:809)
          	at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:314)
          	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:266)
          	at hudson.model.Run.run(Run.java:923)
          	at hudson.model.Build.run(Build.java:112)
          	at hudson.model.ResourceController.execute(ResourceController.java:93)
          	at hudson.model.Executor.run(Executor.java:119)
          
          Show
          buej buej added a comment - We had this problem on 2 different hudson jobs (each bound to a different hudson-slave) and it was reproducable the time we had it (started the job a second time -> same error). After force syncing manually, the problem didn't appear anymore. The whole workspace folder of these jobs have been deleted before... so the force sync really had to sync everything. We use hudson version 1.303 and perforce plugin version 1.0.16. Maybe it relates to the number of changes that are described at the beginning? As you see below, the last sync'd change was 314323 and the head one was 314881... there are quite a lot big changes on this branch in between. Please let me know if you need to know something else (before you close the job this time). Started by user admin Building remotely on MPATEST1 Performing sync with Perforce for: //depot/rel_3.12/... Changing client to REL_3.12_Hudson_tests-mpatest1 [rel_3.12_tests_infx] $ p4 workspace -o REL_3.12_Hudson_tests-mpatest1 Changing P4 Client Root to: E:\builds\hudson\workspace\rel_3.12_tests_infx\ Changing P4 Client View to: //depot/rel_3.12/... //REL_3.12_Hudson_tests-mpatest1/... [rel_3.12_tests_infx] $ p4 -s client -i Last sync'd change: 314323 [rel_3.12_tests_infx] $ p4 changes -m 25 //depot/rel_3.12/... [rel_3.12_tests_infx] $ p4 changes -m 25 //depot/rel_3.12/...@314716 [rel_3.12_tests_infx] $ p4 changes -m 25 //depot/rel_3.12/...@314357 [rel_3.12_tests_infx] $ p4 describe -s 314881 [rel_3.12_tests_infx] $ p4 describe -s 314880 [rel_3.12_tests_infx] $ p4 describe -s 314860 [rel_3.12_tests_infx] $ p4 describe -s 314848 [rel_3.12_tests_infx] $ p4 describe -s 314838 [rel_3.12_tests_infx] $ p4 describe -s 314833 [rel_3.12_tests_infx] $ p4 describe -s 314801 [rel_3.12_tests_infx] $ p4 describe -s 314799 [rel_3.12_tests_infx] $ p4 describe -s 314798 [rel_3.12_tests_infx] $ p4 describe -s 314797 [rel_3.12_tests_infx] $ p4 describe -s 314789 [rel_3.12_tests_infx] $ p4 describe -s 314779 [rel_3.12_tests_infx] $ p4 describe -s 314767 [rel_3.12_tests_infx] $ p4 describe -s 314765 [rel_3.12_tests_infx] $ p4 describe -s 314754 [rel_3.12_tests_infx] $ p4 describe -s 314745 [rel_3.12_tests_infx] $ p4 describe -s 314730 [rel_3.12_tests_infx] $ p4 describe -s 314728 [rel_3.12_tests_infx] $ p4 describe -s 314727 [rel_3.12_tests_infx] $ p4 describe -s 314725 [rel_3.12_tests_infx] $ p4 describe -s 314724 [rel_3.12_tests_infx] $ p4 describe -s 314723 [rel_3.12_tests_infx] $ p4 describe -s 314721 [rel_3.12_tests_infx] $ p4 describe -s 314720 [rel_3.12_tests_infx] $ p4 describe -s 314717 [rel_3.12_tests_infx] $ p4 describe -s 314699 [rel_3.12_tests_infx] $ p4 describe -s 314698 [rel_3.12_tests_infx] $ p4 describe -s 314660 [rel_3.12_tests_infx] $ p4 describe -s 314652 [rel_3.12_tests_infx] $ p4 describe -s 314646 [rel_3.12_tests_infx] $ p4 describe -s 314639 [rel_3.12_tests_infx] $ p4 describe -s 314636 [rel_3.12_tests_infx] $ p4 describe -s 314611 [rel_3.12_tests_infx] $ p4 describe -s 314539 [rel_3.12_tests_infx] $ p4 describe -s 314518 [rel_3.12_tests_infx] $ p4 describe -s 314512 [rel_3.12_tests_infx] $ p4 describe -s 314509 [rel_3.12_tests_infx] $ p4 describe -s 314503 [rel_3.12_tests_infx] $ p4 describe -s 314488 [rel_3.12_tests_infx] $ p4 describe -s 314478 [rel_3.12_tests_infx] $ p4 describe -s 314461 [rel_3.12_tests_infx] $ p4 describe -s 314458 [rel_3.12_tests_infx] $ p4 describe -s 314456 [rel_3.12_tests_infx] $ p4 describe -s 314429 [rel_3.12_tests_infx] $ p4 describe -s 314426 [rel_3.12_tests_infx] $ p4 describe -s 314385 [rel_3.12_tests_infx] $ p4 describe -s 314381 [rel_3.12_tests_infx] $ p4 describe -s 314377 [rel_3.12_tests_infx] $ p4 describe -s 314365 [rel_3.12_tests_infx] $ p4 describe -s 314358 [rel_3.12_tests_infx] $ p4 describe -s 314354 [rel_3.12_tests_infx] $ p4 describe -s 314345 [rel_3.12_tests_infx] $ p4 describe -s 314340 Sync'ing workspace to depot. ForceSync flag is set, forcing: p4 sync //depot/rel_3.12/... [rel_3.12_tests_infx] $ p4 sync -f //depot/rel_3.12/...@314881 FATAL: Java heap space java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuilder.append(StringBuilder.java:119) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:194) at com.tek42.perforce.parse.Workspaces.syncTo(Workspaces.java:90) at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:346) at hudson.model.AbstractProject.checkout(AbstractProject.java:809) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:314) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:266) at hudson.model.Run.run(Run.java:923) at hudson.model.Build.run(Build.java:112) at hudson.model.ResourceController.execute(ResourceController.java:93) at hudson.model.Executor.run(Executor.java:119)
          buej buej made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          rpetti Rob Petti added a comment -

          The changelog information is saved to disk and goes out of scope before the sync runs, so this is purely because you have too many files and not enough heap space allocated. I can probably get around it since there shouldn't be any need to keep the entire log of the sync operation in memory.

          Your stacktrace looks odd to me... It doesn't seem to match the 1.0.16 code at all. I highly recommend upgrading to the latest version of the plugin. If you are using a custom build, then I'm not sure if there's anything I can do for you.

          Show
          rpetti Rob Petti added a comment - The changelog information is saved to disk and goes out of scope before the sync runs, so this is purely because you have too many files and not enough heap space allocated. I can probably get around it since there shouldn't be any need to keep the entire log of the sync operation in memory. Your stacktrace looks odd to me... It doesn't seem to match the 1.0.16 code at all. I highly recommend upgrading to the latest version of the plugin. If you are using a custom build, then I'm not sure if there's anything I can do for you.
          Hide
          buej buej added a comment -

          Sorry, it was p4 plugin version 1.0.13. We will upgrade to the latest plugin version soon.

          Show
          buej buej added a comment - Sorry, it was p4 plugin version 1.0.13. We will upgrade to the latest plugin version soon.
          Hide
          rpetti Rob Petti added a comment -

          If it still fails after the upgrade, then could you provide me with a new stack trace? The code from 1.0.13 doesn't have the perforce api libraries included, so it's difficult to debug.

          Show
          rpetti Rob Petti added a comment - If it still fails after the upgrade, then could you provide me with a new stack trace? The code from 1.0.13 doesn't have the perforce api libraries included, so it's difficult to debug.
          Hide
          ssbarnea Sorin Sbarnea added a comment - - edited

          I did get the same running Hudson 1.376 and Perforce plugin 1.1.6.

          And yes, it it a huge sync, over 10gb and >100.000 files but this should not be an excuse.

          Show
          ssbarnea Sorin Sbarnea added a comment - - edited I did get the same running Hudson 1.376 and Perforce plugin 1.1.6. And yes, it it a huge sync, over 10gb and >100.000 files but this should not be an excuse.
          Hide
          ssbarnea Sorin Sbarnea added a comment -

          I'm running Hudson with 3000mb of memory and get the same results

          15:15:22 [alf-ps-windows] $ p4 sync f //hudson_alf-ps-windows-1162767326/...@698875
          16:41:55 FATAL: Java heap space
          16:41:55 java.lang.OutOfMemoryError: Java heap space
          16:41:55 at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
          16:41:55 at java.lang.StringBuilder.<init>(StringBuilder.java:80)
          16:41:55 at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:344)
          16:41:55 at com.tek42.perforce.parse.Workspaces.syncTo(Workspaces.java:117)
          16:41:55 at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:605)
          16:41:55 at hudson.model.AbstractProject.checkout(AbstractProject.java:1082)
          16:41:55 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
          16:41:55 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
          16:41:55 at hudson.model.Run.run(Run.java:1273)
          16:41:55 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          16:41:55 at hudson.model.ResourceController.execute(ResourceController.java:88)
          16:41:55 at hudson.model.Executor.run(Executor.java:137)

          Show
          ssbarnea Sorin Sbarnea added a comment - I'm running Hudson with 3000mb of memory and get the same results 15:15:22 [alf-ps-windows] $ p4 sync f //hudson_alf-ps-windows -1162767326/...@698875 16:41:55 FATAL: Java heap space 16:41:55 java.lang.OutOfMemoryError: Java heap space 16:41:55 at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45) 16:41:55 at java.lang.StringBuilder.<init>(StringBuilder.java:80) 16:41:55 at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:344) 16:41:55 at com.tek42.perforce.parse.Workspaces.syncTo(Workspaces.java:117) 16:41:55 at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:605) 16:41:55 at hudson.model.AbstractProject.checkout(AbstractProject.java:1082) 16:41:55 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479) 16:41:55 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411) 16:41:55 at hudson.model.Run.run(Run.java:1273) 16:41:55 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) 16:41:55 at hudson.model.ResourceController.execute(ResourceController.java:88) 16:41:55 at hudson.model.Executor.run(Executor.java:137)
          Hide
          ssbarnea Sorin Sbarnea added a comment -

          Now the question is who is getting out of memory: the server or the slave. I think it is the slave and in this case I don't know how to increase the memory of the slave (node).

          Anyway, the p4 plugin doesn't need so much memory.

          Show
          ssbarnea Sorin Sbarnea added a comment - Now the question is who is getting out of memory: the server or the slave. I think it is the slave and in this case I don't know how to increase the memory of the slave (node). Anyway, the p4 plugin doesn't need so much memory.
          Hide
          rpetti Rob Petti added a comment -

          ssbarnea, you have 3GB of memory, but have you configured your java VM to actually use it? You need to supply the jvm flag: -Xmx3000m in order to use the full 3GB (the default is something considerably smaller.)

          The output of the p4 commands is stored and parsed on the master, not the slave, so that's where the memory is needed.

          Fixing this basically requires me to rewrite large portions of the tek42 library code, so for the time being please just sync the workspace manually the first time. I can't imagine you are editing so many files at a time that you'd run into this again after the initial sync.

          Show
          rpetti Rob Petti added a comment - ssbarnea, you have 3GB of memory, but have you configured your java VM to actually use it? You need to supply the jvm flag: -Xmx3000m in order to use the full 3GB (the default is something considerably smaller.) The output of the p4 commands is stored and parsed on the master , not the slave, so that's where the memory is needed. Fixing this basically requires me to rewrite large portions of the tek42 library code, so for the time being please just sync the workspace manually the first time. I can't imagine you are editing so many files at a time that you'd run into this again after the initial sync.
          Hide
          ssbarnea Sorin Sbarnea added a comment -

          My options were made on JAVA_OPTS but I discovered that when I installed tomcat I forgot to `tune` the starter catalina.sh to use 3GB of RAM.

          Thanks for letting me know that this was a server side issue, previously I tried to change the settings on the node config.

          I'm wondering why this still happens considering that 100.000 lines or 200 bytes would take only ~20mb or RAM.

          Aren't any easier workarounds, like not storing the p4 sync results at all?

          Show
          ssbarnea Sorin Sbarnea added a comment - My options were made on JAVA_OPTS but I discovered that when I installed tomcat I forgot to `tune` the starter catalina.sh to use 3GB of RAM. Thanks for letting me know that this was a server side issue, previously I tried to change the settings on the node config. I'm wondering why this still happens considering that 100.000 lines or 200 bytes would take only ~20mb or RAM. Aren't any easier workarounds, like not storing the p4 sync results at all?
          Hide
          rpetti Rob Petti added a comment -

          Not storing the p4 sync results is the part that would require me to rewrite large portions of the tek42 code. It's built into how the library deals with output from all perforce commands, not just the sync code. All the output is stored, then parsed, then split, then parsed again. Certainly not efficient when you have over 2 million files being synced at once like some people seem to have. We would need to rewrite it so it would use a buffered stream that's provided to all levels of abstraction simultaneously, which would also likely require multithreading. A quick hack would be to throw out all output when 'sync' is in the command line, but then you would have the unfortunate side effect of not having any feedback if something goes wrong.

          The tek42 library simply wasn't designed to handle large operations like this, but given that the official p4java doesn't work correctly on all platforms, it's the best we've got.

          It would only take ~20mb if it was pure data. The tek42 libraries also make a copy of it and split it into an array of lines. At the end of the day, it could be taking up as much as 3 or 4 times the amound of ram you would normally expect. Depending on the JVM you are using, the default max heap size value could be as low as 64mb-128mb, so that would easily fill it up in no time. We run with -Xmx2000m, and have never run into this issue.

          Getting this fixed is on my list of things to do, but I'm getting slammed at work and haven't had much time for OSS at the moment.

          Show
          rpetti Rob Petti added a comment - Not storing the p4 sync results is the part that would require me to rewrite large portions of the tek42 code. It's built into how the library deals with output from all perforce commands, not just the sync code. All the output is stored, then parsed, then split, then parsed again. Certainly not efficient when you have over 2 million files being synced at once like some people seem to have. We would need to rewrite it so it would use a buffered stream that's provided to all levels of abstraction simultaneously, which would also likely require multithreading. A quick hack would be to throw out all output when 'sync' is in the command line, but then you would have the unfortunate side effect of not having any feedback if something goes wrong. The tek42 library simply wasn't designed to handle large operations like this, but given that the official p4java doesn't work correctly on all platforms, it's the best we've got. It would only take ~20mb if it was pure data. The tek42 libraries also make a copy of it and split it into an array of lines. At the end of the day, it could be taking up as much as 3 or 4 times the amound of ram you would normally expect. Depending on the JVM you are using, the default max heap size value could be as low as 64mb-128mb, so that would easily fill it up in no time. We run with -Xmx2000m, and have never run into this issue. Getting this fixed is on my list of things to do, but I'm getting slammed at work and haven't had much time for OSS at the moment.
          Hide
          rpetti Rob Petti added a comment -

          85b8bf8 should solve this issue for most if not all users. It basically throws away all but the first 50 lines of the response from 'p4 sync'. Errors are still checked for on the fly, but "synced"/"refreshed" notifications are ignored.

          Show
          rpetti Rob Petti added a comment - 85b8bf8 should solve this issue for most if not all users. It basically throws away all but the first 50 lines of the response from 'p4 sync'. Errors are still checked for on the fly, but "synced"/"refreshed" notifications are ignored.
          rpetti Rob Petti made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 132215 ] JNJira + In-Review [ 186516 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              fowlesp fowlesp
            • Votes:
              3 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: