Composer update runs out of memory

Running the "composer update" command results in a fatal error:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 79 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 100

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 71 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/Solver.php on line 700

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 104

I already tried removing the vendors in my Symfony2 project. After removing the vendors and running a "composer install" command I get no errors. Running a "composer update" directly after the install results in the same memory error. The current memory limit is set to 512MB, that should be enough, right?

What can be the cause of the sudden memory issue? Can I take other steps in solving this issue? As far as I can tell, the dependencies haven't changed and I don't include many vendors.

Any help is much appreciated!

Kind regards,
Jeroen van Rooij

Ps. I'm using the latest composer version and al the requires in the json are set to a fixed version

Could you maybe share your composer.json so it's at least possible to try and reproduce it?

Yeah ofcourse, sorry!

{
    "name": "lsg/lsg.nl",
    "description": "Lsg website",
    "autoload": {
        "psr-0": { "": "src/" }
    },
    "repositories": [
        { "type": "vcs", "url": "git@github.com:lsg-it/ExtensionBundle.git" },
        { "type": "vcs", "url": "git@github.com:lsg-it/CmsBundle.git" }
    ],
    "require": {
        "php": ">=5.3.3",
        "symfony/symfony": "v2.1.7",
        "doctrine/orm": "2.3.1",
        "doctrine/doctrine-bundle": "v1.0.0",
        "doctrine/doctrine-fixtures-bundle": "2.1.*",
        "twig/extensions": "1.0.*",
        "symfony/assetic-bundle": "v2.1.1",
        "symfony/swiftmailer-bundle": "v2.1.7",
        "symfony/monolog-bundle": "v2.1.7",
        "sensio/distribution-bundle": "v2.1.7",
        "sensio/framework-extra-bundle": "v2.1.7",
        "sensio/generator-bundle": "v2.1.7",
        "jms/security-extra-bundle": "1.2.0",
        "jms/di-extra-bundle": "1.1.1",
        "nelmio/solarium-bundle": "v1.1.0",
        "friendsofsymfony/facebook-bundle": "v1.1.0",
        "hwi/oauth-bundle": "*",
        "gedmo/doctrine-extensions": "v2.3.4",
        "lsg/cms-bundle": "master",
        "lsg/extension-bundle": "master"
    },
    "scripts": {
        "post-install-cmd": [
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile"
        ],
        "post-update-cmd": [
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile"
        ]
    },
    "minimum-stability": "dev",
    "extra": {
        "symfony-app-dir": "app",
        "symfony-web-dir": "web",
        "symfony-assets-install": "symlink"
    }
}

Those two Lsg requires are internal, private bundles. The json of those two bundles are:

{
    "name": "lsg/cms-bundle",
    "type": "symfony-bundle",
    "description": "LSG CMS Bundle",
   "repositories": [
        { "type": "vcs", "url": "git@github.com:lsg-it/ExtensionBundle.git" }
    ],
    "require": {
        "php": ">=5.3.3",
        "symfony/symfony": "2.1.*",
        "white-october/pagerfanta-bundle": "2.1.*",
        "craue/formflow-bundle": "1.1.0",
        "lsg/extension-bundle": "master"
    },
    "autoload": {
        "psr-0": {
            "Lsg\\CmsBundle": ""
        }
    },
    "target-dir": "Lsg/CmsBundle"
}

And the last one:

{
    "name": "lsg/extension-bundle",
    "type": "symfony-bundle",
    "description": "Standard LSG extensions",
    "require": {
        "php": ">=5.3.3",
        "symfony/symfony": "2.1.*",
        "gedmo/doctrine-extensions": "v2.3.4",
        "twig/twig": "1.*"
    },
    "autoload": {
        "psr-0": {
            "Lsg\\ExtensionBundle": ""
        }
    },
    "target-dir": "Lsg/ExtensionBundle"
}

We're receiving this error too. Increasing to 1G in php.ini worked.

.json below...

{
"name": "symfony/framework-standard-edition",
"description": "The "Symfony Standard Edition" distribution",
"autoload": {
"psr-0": { "": "src/" }
},
"require": {
"php": ">=5.3.3",
"symfony/symfony": "2.1.",
"doctrine/orm": ">=2.2.3,<2.4-dev",
"doctrine/doctrine-bundle": "1.0.",
"twig/extensions": "1.0.",
"symfony/assetic-bundle": "2.1.",
"symfony/swiftmailer-bundle": "2.1.",
"symfony/monolog-bundle": "2.1.",
"sensio/distribution-bundle": "2.1.",
"sensio/framework-extra-bundle": "2.1.",
"sensio/generator-bundle": "2.1.",
"jms/security-extra-bundle": "1.2.",
"jms/di-extra-bundle": "1.1.",
"dholmes/error-handling": "master",
"dholmes/doctrine-extras": "master",
"friendsofsymfony/jsrouting-bundle": "1.0.2",
"amazonwebservices/aws-sdk-for-php": "1.5.15",
"beberlei/DoctrineExtensions": "v0.1",
"doctrine/doctrine-fixtures-bundle": "dev-master",
"EHER/phpunit": ">=1.2",
"phake/phake": "dev-master",
"stripe/stripe-php": "v1.7.7",
"imagine/Imagine": "dev-master",
"michelf/php-markdown": "1.2.5",
"cboden/Ratchet": "0.2.",
"react/zmq": "dev-master"
},
"repositories": [
{
"type": "vcs",
"url": "https://github.com/danielholmes/error-handling"
},
{
"type": "vcs",
"url": "https://github.com/danielholmes/doctrine-extras"
},
{
"type": "package",
"package": {
"name": "michelf/php-markdown",
"version": "1.2.5",
"source": {
"url": "git://github.com/michelf/php-markdown.git",
"type": "git",
"reference": "origin/extra"
}
}
}
],
"scripts": {
"post-install-cmd": [
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::buildBootstrap",
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::clearCache",
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installAssets",
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installRequirementsFile"
],
"post-update-cmd": [
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::buildBootstrap",
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::clearCache",
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installAssets",
"Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installRequirementsFile"
]
},
"minimum-stability": "dev",
"extra": {
"symfony-app-dir": "app",
"symfony-web-dir": "web"
}
}

I also can confirm this issue, after updating composer I can't update our projects without raising the memory_limit from 512MB to 1GB. Seems that memory usage increased quite a lot. Here is one of our composer.json files that are problematic:

{
    "name": "symfony/framework-standard-edition",
    "description": "The \"Symfony Standard Edition\" distribution",
    "autoload": {
        "psr-0": { "": "src/" }
    },
    "require": {
        "php": ">=5.3.3",
        "symfony/symfony": "2.1.*",
        "doctrine/orm": ">=2.2.3,<2.4-dev",
        "doctrine/doctrine-bundle": "1.0.*-dev",
        "twig/extensions": "1.0.*-dev",
        "symfony/assetic-bundle": "2.1.*-dev",
        "symfony/swiftmailer-bundle": "2.1.*-dev",
        "symfony/monolog-bundle": "2.1.*-dev",
        "sensio/distribution-bundle": "2.1.*-dev",
        "sensio/framework-extra-bundle": "2.1.*-dev",
        "sensio/generator-bundle": "2.1.*-dev",
        "jms/security-extra-bundle": "1.2.*",
        "jms/di-extra-bundle": "1.1.*-dev",
        "friendsofsymfony/rest-bundle" : "0.10.*-dev",
        "doctrine/doctrine-migrations-bundle" : "dev-master",
        "richsage/rms-push-notifications-bundle" : "*",
        "imagine/Imagine": "dev-develop",
        "stof/doctrine-extensions-bundle": "dev-master",
        "mlpz/postmark-bundle": "*",
        "jms/serializer-bundle" : "dev-master",
        "craue/twigextensions-bundle": "dev-master",
        "wrep/bugsnag-php-symfony" : "dev-master",
        "wrep/notificato-symfony" : "1.0.*",
        "wrep/daemonizable-command": "dev-master"
    },
    "scripts": {
        "post-install-cmd": [
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile"
        ],
        "post-update-cmd": [
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile"
        ]
    },
    "config": {
        "bin-dir": "bin"
    },
    "minimum-stability": "dev",
    "extra": {
        "symfony-app-dir": "app",
        "symfony-web-dir": "web"
    }
}

I just added the wrep/daemonizable-command package and runned composer update wrep/daemonizable-command, but that crashed with this error:

Loading composer repositories with package information
Updating dependencies (including require-dev)
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 32 bytes) in phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 52
PHP Stack trace:
PHP   1. {main}() /usr/bin/composer:0
PHP   2. require() /usr/bin/composer:15
PHP   3. Composer\Console\Application->run() phar:///usr/bin/composer/bin/composer:43
PHP   4. Symfony\Component\Console\Application->run() phar:///usr/bin/composer/src/Composer/Console/Application.php:83
PHP   5. Composer\Console\Application->doRun() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:119
PHP   6. Symfony\Component\Console\Application->doRun() phar:///usr/bin/composer/src/Composer/Console/Application.php:117
PHP   7. Symfony\Component\Console\Application->doRunCommand() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:212
PHP   8. Symfony\Component\Console\Command\Command->run() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:882
PHP   9. Composer\Command\UpdateCommand->execute() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:244
PHP  10. Composer\Installer->run() phar:///usr/bin/composer/src/Composer/Command/UpdateCommand.php:102
PHP  11. Composer\Installer->doInstall() phar:///usr/bin/composer/src/Composer/Installer.php:208
PHP  12. Composer\DependencyResolver\Solver->solve() phar:///usr/bin/composer/src/Composer/Installer.php:446
PHP  13. Composer\DependencyResolver\RuleWatchGraph->insert() phar:///usr/bin/composer/src/Composer/DependencyResolver/Solver.php:170
PHP  14. SplDoublyLinkedList->unshift() phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php:52

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 32 bytes) in phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 52

Call Stack:
    0.0035     630856   1. {main}() /usr/bin/composer:0
    0.0091     817720   2. require('phar:///usr/bin/composer/bin/composer') /usr/bin/composer:15
    0.0454    5095992   3. Composer\Console\Application->run() phar:///usr/bin/composer/bin/composer:43
    0.0502    5671944   4. Symfony\Component\Console\Application->run() phar:///usr/bin/composer/src/Composer/Console/Application.php:83
    0.0519    5917824   5. Composer\Console\Application->doRun() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:119
    0.0530    6057016   6. Symfony\Component\Console\Application->doRun() phar:///usr/bin/composer/src/Composer/Console/Application.php:117
    0.0544    6057016   7. Symfony\Component\Console\Application->doRunCommand() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:212
    0.0544    6057016   8. Symfony\Component\Console\Command\Command->run() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:882
    0.0555    6055984   9. Composer\Command\UpdateCommand->execute() phar:///usr/bin/composer/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:244
    0.2593   13311072  10. Composer\Installer->run() phar:///usr/bin/composer/src/Composer/Command/UpdateCommand.php:102
    0.2611   13478184  11. Composer\Installer->doInstall() phar:///usr/bin/composer/src/Composer/Installer.php:208
    3.2977   53035864  12. Composer\DependencyResolver\Solver->solve() phar:///usr/bin/composer/src/Composer/Installer.php:446
   53.1716  536465808  13. Composer\DependencyResolver\RuleWatchGraph->insert() phar:///usr/bin/composer/src/Composer/DependencyResolver/Solver.php:170
   53.1716  536466480  14. SplDoublyLinkedList->unshift() phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php:52

 I'm also receiving this issue on PHP 5.3.15 on Mac OSX. Memory limit is at 512mb and never had a problem in the past (until updating composer a couple of days ago).

EDIT: I can confirm that upping a 1gb limit fixes, but surely an issue.

Composer JSON:

{
    "autoload": {
        "psr-0":         {
            "": "src/"
        }
    },
    "config": {
        "bin-dir": "bin"
    },
    "description": "Tickit Application",
    "extra": {
        "symfony-app-dir": "app",
        "symfony-web-dir": "web"
    },
    "minimum-stability": "dev",
    "name": "tickit-project/tickit",
    "require": {
        "doctrine/doctrine-bundle": "1.2.x-dev",
        "doctrine/doctrine-fixtures-bundle": "dev-master",
        "doctrine/orm": "2.3.*",
        "friendsofsymfony/user-bundle": "2.0.x-dev",
        "jms/di-extra-bundle": "1.3.x-dev",
        "jms/security-extra-bundle": "1.4.x-dev",
        "php": ">=5.3.3",
        "sensio/distribution-bundle": "2.3.x-dev",
        "sensio/framework-extra-bundle": "2.3.x-dev",
        "sensio/generator-bundle": "2.3.x-dev",
        "stof/doctrine-extensions-bundle": "1.1.x-dev",
        "symfony/assetic-bundle": "2.1.x-dev",
        "symfony/monolog-bundle": "2.2.x-dev",
        "symfony/swiftmailer-bundle": "2.2.x-dev",
        "symfony/symfony": "2.3.x-dev",
        "symfony/console": "2.3.x-dev",
        "twig/extensions": "1.0.x-dev",
        "psr/log": "1.0.0",
        "tickit-project/cache-bundle": "1.0.x-dev",
        "squizlabs/php_codesniffer": "1.*",
        "behat/symfony2-extension": "dev-master",
        "satooshi/php-coveralls": "dev-master"
    },
    "scripts": {
        "post-install-cmd": [
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile"
    ],
        "post-update-cmd": [
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
"Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile"
    ]
    }
}

Same issue since updating Composer. 1GB workaround does work, but something must be wrong.

Composer: 950fc7e
Mac OSX: 10.8.3
PHP: 5.3.15

Now I've hit 2GB memory limit ;) I'm getting 16GB RAM tonight so gonna increase the php memory limit to around 10, should last for few months of updates :P

Updating my CLI to php54 fixed it for now.

Yup, upgrading to php5.4.x fixes a memory issue when using php 5.3.x. For example, I was running composer install without a .lock file. With php5.3.10, composer required over 3GB of memory. With php5.4.16, composer consumed less than 1GB.

Needing 512MiB RAM for a package-manager is already a bit much, but I also exceed it. The interesting part is that I exceed 512 MiB RAM (up to around 800 MiB RAM according to my process output) when I use my private Satis repository (which is actually just a cache for all Github public repositories). I do not exceed it otherwise. Just by adding my satis repository, composer consumes almost 400 MiB more RAM.

I'm on PHP 5.4.14.

To all the others raising their PHP CLI memory_limit globally, I recommend against that. Instead I suggest to put it in the command line, like so:

php -dmemory_limit=1G bin/composer.phar update .. 

Otherwise you could potentially miss certain unintended memory related behaviour in your own work.

Chiming in with my experience. I'm also using a private Satis repository in addtion to the zend framework repository. I was having problems with composer update on PHP 5.3.26 using over 2GB of memory.

Switching to 5.4 solved everything, now using under 400MB.

ping @Seldaek

Thanks @Dynom, this works for those of us on PHP <5.4

php -dmemory_limit=1G composer.phar update

Same problem

Same here. Mac OS X. PHP 5.3.15.
Fixed by updating php to 5.4.10.

Happening to me on 5.5. Guessing something somewhere is not cleaning up after itself and leaving objects hanging around?

Same problem with php 5.3.15 on Mac OS X.

Same problem here. I can't get it to composer update as I only have 1gb total for this particular vps.

PHP 5.4.11 (cli) (built: Feb 20 2013 19:02:54)

----> Update
Not sure if it helps but I noticed that my local php version was: 5.4.19 so although I had 5.4 on the production server it was a few updates out.

I updated the production server from 5.4.11 to 5.4.19 and composer update seems to be working now.

Had this issue until I upgraded to PHP 5.4.19. composer update now appears to work.

I'm also have a problem, but yesterday it worked fine with the same composer.json

Ugh, this is a killer on shared hosting.

@kevinm1 it's not, you are not supposed to run composer update on your production server. You should run update on your dev box then deploy the composer.lock file and run install which will run with very low memory footprint and very fast since it doesn't have to check for new packages but merely applies what's in the lock file.

Yep, same here - have to increase memory limit to 1G. And indeed it eats up to 980M.

Interesting thing is that this started to happen only after I added local Satis composer repository which in fact holds only cached .zip packages mostly if not only from github.

Without local Satis repository, composer uses around of 240mb.
And if use local Satis repository with turned off packagist it also uses around 240mb. Example:

"repositories": [
{ "type": "composer", "url": "http://packagist.example.com" },
{ "packagist": false }
],

Happens on Gentoo with PHP 5.4.16

I was on PHP 5.5.5. Cannot solve this. Tried to disable lots of extension: opcode, xdebug,..I actually ended up creating swap file. My VPS is 512MB (a cheap digital ocean instance) with a handful of service: nginx, mariadb, mongodb, redis. puma.

to clarify, I was not able to run a simple single command for installing laravel

composer create-project laravel/laravel

Here is my simple work around, hopefully help s.o:

df -h 
dd if=/dev/zero of=/swapfile bs=1M count=1024
sudo dd if=/dev/zero of=/swapfile bs=1M count=1024
mkswap /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo 'echo "/swapfile  none  swap  defaults  0  0" >> /etc/fstab' | sudo sh

free -m

confirm u see your swap there:
total used free shared buffers cached
Mem: 494 335 158 0 19 62
-/+ buffers/cache: 254 240
Swap: 1023 3 1020

Then install composer, and wat your RAM ;(

watch free -m

Most of the people getting errors related to this ticket seem to be trying to create projects and/or updating with composer on their servers. Best practice is running your composer create-project and composer update commands locally, using git to pull your project updates to the server, and then simply running composer install on the server to update your production dependancies.

I did what u said, and it fail right on that command...I actually ended up creating a swap file to expand memory..

While php -dmemory_limit=1G composer.phar update is ok, it is rather a bit to type (especially if you install it to /usr/bin). Any thoughts about passing the memory limit optionally directly to composer, something like

composer --memory=1G update

@kmmathis I'm getting the same problem when I do composer install.

composer install also gives me the same issue (PHP 5.3.3).
@uwej711 you can still php -dmemory_limit=xxx /usr/local/bin/composer ...
It passes with 512M memory_limit

maybe there is a substantial problem building dependency graph, that needs to be addressed?...

I think you might be on to something there @nfx ;-)

I'd like to remind everyone, including myself, that Composer is OSS and can be forked and improved by yourself!

Same problem here, memory_limit = 512M;, the only way to solve this is increasing the memory? another way?

@frangeris Increasing swap space seems to have helped.

I received this error with PHP 5.5.6 x64 on Win7, memory_limit = 1G resolved the issue. 512M did not.

Same as others, 1G solved my issue but even at 512M it had errors.

Same problem ;-(

@Seldaek "deploy the composer.lock file and run install"

I know this is probably basic for you but could you please tell me how to do this on the command line?
I have an idea but I wouldn't want to screw up :)

Thanks!

• Vincent

@musixite The composer.lock is basically a copy of composer.json (with some timestamps added). When you run composer update it will attempt to get new versions of your dependencies and will then update the composer.lock file with these version numbers. This takes quite a lot of memory.

composer install on the other hand will look in composer.lock and install the versions specified there. This takes less memory.

So update should only be used when developing as it could break things. Only install should be used in production as it will only install things that have (hopefully!) been tested.

What @Seldaek is saying is that you should push to your project to your production server with a composer.lock file attached and then run composer install.

Hope that helps!

Running into this issue also. There shouldn't be a reason for composer to need this much memory.

Same here with 512M RAM using PHP 5.5.3

Getting this problem on our dev server too, for the first time. Trying to set the memory limit in the command line just gives us this:

$ php -dmemory_limit=1G composer.phar update 
Loading composer repositories with package information
Updating dependencies (including require-dev)
Killed
$

However, running the command about six times eventually completed without an error. I don't know if each run leaves enough cached data behind to pick up where it left off, working further on each run, or something else was affecting how much memory it needed.

@judgej I guess you are just at the edge of your server's memory so it gets killed or not depending on the amount of ram used when you start composer. Anyway you should avoid running update on servers at all, rather commit the composer.lock file and run install on servers, which uses very little memory.

Servers are where I kind of do my development. The composer.lock and install steps I can understand for migrating to other server instances - test/staging/production/whatever - but am I in the minority doing work on a "server" rather than my own laptop or workstation? I have too many things on the go at once to work any other way (or maybe I need a bigger laptop with a zillion VMs :-).

On the plus side, at least I know I'm not blowing the 1G limit by too much ;-)

So out of curiosity, is there a fundamental reason why an update has to use that much memory, or is it something that is lower on the priorities to fix or optimise because people should only be doing this on a machine with enough resources? Just wondering if there is hope for using less memory (with fixes by whoever), or if it ain't gonna happen (because it can't) and it must be handled by the resources we must throw at it.

@judgej I don't know if you are in the minority or not, I think it depends on the sort of demographic you are looking at. In the people I know doing mostly the things I do, most work on their own laptops, either in VMs or in the native OS. That usually gives you more allowances regarding memory. I'm sure other communities/groups tend to still rely more on dev servers. By the way the Killed line does not mean it reached the php limit (1G) but that the OS ran out of memory and decided to kill that process since it's hoarding all of it.

As for why it's using so much, unfortunately it's not such an easy fix, I wish I'll have time to dedicate to that some day, but right now it's using lots because it loads tons of packages in memory and php isn't the most memory efficient beast out there. Upgrading to php 5.4 or even 5.5 should help on that front by the way, 5.3 is quite wasteful.

Being able to see how memory is being used graphically would be cool, one day, for seeing just where it all goes.

I kind of guessed the Killed came from a different source, as PHP did not have a chance to report any reason for aborting. Just ZAP - you're dead.

Thanks for spending your time explaining, much appreciated :-)

Seeing all this, is there much point in excluding the lock file from version control? If we're encouraging install rather than update on deployed servers (testing/staging/production), surely the lock file should always be deployed?

@froddd there is strictly no point about excluding the lock file from version control for projects (and the doc advices to version it). If you run composer update during your deployment process, you have strictly no guarantee that you will get the same dependencies than during your local development or your CI build (you are likely to get newer versions). This means that running composer update on deployment implies deploying untested code.
The only case where it can make sense to exclude the lock file is when developing a library which is only meant to be used as a dependency (so its lock file will impact only its own development and testing), and you want your CI server to get the newest version of the dependencies each time

Could it be possible to add to the system requirements a note about the high memory usage? @judgej is not the only one who needs to run it on low memory systems, and most of my VMs usually don't have 512MB+ free for PHP, it's more in the range of 128-256MB. We don't all have high-end computers in an enterprise environment. ;)

I actually had to copy the composer.json to a production server with lots of memory, run composer updateon it, delete the vendor directory after it had fetched all its packages, then copy the resulting composer.lock file back to my development machine to run a composer install. That worked, but is far from ideal.

I have been thinking about these memory issues for the last few days. Is there a discussion or overview somewhere that lays out the issues that is causing the memory usage to be so high? I'm wondering if we understand the problems more, we may be able to help. Is there a way to work out the dependencies without opening up all the packages in memory, i.e. just getting hold of the composer.json files for each package without the rest of the files it contains (I'm assuming composer.json is all that is needed - but I may be wrong)?

Or could packages be extracted to a temporary folder rather then into memory, using a shell command (when run on Linux), since the shell command does not use PHP's memory allocation and memory is cleaned up a little better when the shell command ends.

This is becoming a bigger issue as more frameworks use composer for all their packages. I'd like to help, so if there are any pointers on where to get started (a description of perhaps why this is not a trivial problem to solve) then that would help a lot.

@judgej a Package object for Composer contains only the information of the composer.json file (and not even all of them during the solving of dependencies). Composer does not download the packages themselves until it install them. your suggestion are all talking about moving the extraction of packages out of PHP, while the issue comes from the amount of extracted data which is needed by the PHP code

The issue is that the algorithm resolving dependencies needs to be fed with all potential matches for a constraint, and when you require ~2.0 for symfony/symfony, it already represents 65 packages only for it (without considering dependencies of symfony yet) due to the number of released versions supporting composer, and twice more when requiring symfony/console (as symfony/symfony needs to be considered as well).
As you can see, this requires loading lots of information in memory. Some memory optimizations have already been implemented to limit the data loaded in the algorithm first (avoiding to include the 2.0 packages if you require ~2.1 for instance), but optimizing it further (while keeping the really complex algorithm free of bugs) is a really complex task.

We are talking about optimizing the most complex part of composer here, for which a long time of development was already used (to give you a quick idea, during the first months of development of Composer, @naderman worked quite only on the dependency solver while @Seldaek was building the surrounding infrastructure).

@stof Thanks, that makes a lot of sense. So it is not necessary to download a package to see its metadata (composer.json), which is good. However, you need to download the full (-ish, depending on constraints) history of each package so the information needed to make the right dependency choices is available. I can see there would be many possible combinations to work through, with a lot of data, and PHP is not good at tidying up memory as it goes along.

How does this stuff all fit in one tiny script? ;-) Incredible work.

I wonder how it would work if taken online, with a database to manage the data? Just composer.json in and composer.lock out? A throw-away thought...but I've got to ask the stupid questions.

get similar issue, when i try to run "composer update" with "minimum-stability": "dev" , i just change to "minimum-stability": "stable" and it's work :)

Hitting 3G now... at this pace... (1G a month)... I will be need to change my computer in 4 month... :-S

HHVM can run phar files now. One of my coworkers tried it out with composer and it used less cpu, time, and memory.

Indeed, if you have HHVM available, that'll help a lot.

yeah, HHVM improves things a lot (especially regarding time as it is far more efficient at running complex computations)

Same problem here. To all those who recommend to increase memory limit - memory leaks cannot and must not be fixed by throwing more memory at it.

Well it's not a leak. It simply needs a lot of memory.

A simple PHP script needs 1GB and more?:) You must be kidding me...:) It's either a leak or bad programming.

Yeah it's not that simple. I guess we're terrible programmers and you should rewrite it, looking forward to the new and improved composer!

Don't take it personally but please ... I have had the same composer.json for a while and haven't added any dependencies to that since then. Today when I decided to run composer update it started complaining about memory... As you see above I'm not the only one here with this problem. Should we all buy a separate PC with 16 GB to just run composer?
It's a really really great and useful piece of software otherwise - thanks for that!:)

@svenvarkel the difference is that there are more packages matching the same constraint now that a few months ago, as new releases have been done.
and we all know that Composer should be able to use less memory ideally. But refactoring the solver to use less memory is a huge task (and it must be done without introducing bugs in this very complex algorithm), and all composer devs are working on it on their free time only.
Complaining about memory usage will not help reducing it (some could even argue that it goes the opposite way by making devs deal with comments in the issue tracker). Saying something like It's either a leak or bad programming does not help either. Feel free to contribute if you have a better way to solve the dependencies.

@svenvarkel to work out the dependencies it is necessary to explore a lot of combinations of different versions of everything, including dependencies of dependencies. To update, say, Laravel, there may be many dozens of packages that each come in many versions.

You can prune a few right at the start from the version ranges in your composer.json. Then you load the dependencies, and they may have dependencies and so on. Each version of a dependency you look at at will have different dependency version requirements, so you cannot prune those versions so easily.

By the time you have explored all the possible combinations and thrown out those that are not permitted (e..g package A version A1 depends on package C minimum version C4, but package E cannot go above version C3, none of which you can determine until the complete branch is loaded into memory), then you have massive amounts of data structures in memory.

The trouble is, those tree structures increase in size exponentially as the number of versions of each package increases. What composer did a year ago was good, reasonable, and did the job very well. Unfortunately it does not scale with an exponentially increasing usage of memory. Don't blame the skill of the programmer - what it does is fantastic. The problem is that it is a victim of its own success.

It does need to take a new approach; it has to. Throwing memory at a problem that needs more memory at a slowly increasing rate is fine. When it increases this fast, we will ALL hit this problem sooner or later. I don't know the solution - it may involve handling its data structures on disk and just keeping a simple index in memory.

What I can say, is that this not a trivial problem to solve (I can see that, now that time has been spent to explain it further up the ticket). It is not a black-and-white issue of "bad coding uses too much memory" - it is just a scalability problem that was going to be hit and now we have to find a solution, in as helpful and respectful way as we can.

Edit: It is a real problem, and needs to be fixed before the whole of composer slows to a crawl. I wonder if there is a better forum than here for throwing ideas around? Maybe there is a way to do it in groups or sets (a map reduce kind of approach)? Manage memory in composer maybe, so it is not limited by PHP's garbage collector? Maybe pause while the garbage collector tidies up now and then? I just don't know.

@judgej the issue is that we cannot put the datastructures on disks. The datastructures used in the solver have already been stripped down a lot 1 year ago. The remaining data are what is used by the algorithm itself. I don't see how putting them on disk could help here, as you would still need to read the files to know their content, and so would load the same data again..
The solution to meory issue is probably much more complex that saying "let's just use the disk" (thus, replacing memory usage by disk IO would probably make composer far slower, and people are already complaining about the speed currently)

Have you guys ever checked how Maven works? Maybe it would help and would avoid re-inventing something that's already invented?
I agree it's not trivial, excuse me and I used wrong wording (simple - above).

@stof I was thinking more of a database on disk , but I guess it does still need to be read to process and if the garbage collector does not clean up pruned branches (or whatever they are) then that's memory being eaten. For Linux at least, could composer shell out to run child processes to do some of the calculations? The child processes would not use the same memory allocation and the OS would clean them up after they die.

I realise I'm throwing suggestions in with little knowledge of the inner workings of composer, but it seems to be that the problem is only going to get worse without some fundamental change.

@svenvarkel nothing was 'invented'. AFAIK, the solver logic was inspired by rubygems, (or some other ruby package manager)

@judgej I'm not sure it is really possible to split the SAT solver into multiple processes. It needs to be tested though.

@svenvarkel from a quick Google search (and so nothing guaranteed here as I'm not a Maven expert at all), the dependency solver seems rather simplistic. It seems like if the root project requires <=3.8.1,<4.0 and another dependency requires =3.8.1, you could still get 3.8.2 installed, because the newest dependency is selected for a given depth in the tree, without taking care of constraints set deeper in the tree. The recommendation I saw is to define all dependencies you want to get explicitly at the root. This means that the dependency resolution would have to be done by hand by the developer instead of letting the tool do it. Such algorithm would indeed require much less computation and much less memory than the full dependency solving done by Composer.
Given that Maven is described as a build tool, not as a dependency manager, using a simpler algorithm may be a logical choice. From what I see, using a full solver requires using a Maven plugin.

and regarding making the solving of dependencies more efficient, an obvious way would be to move it to C in a PHP extension, as it could benefit from the C performance (both for memory usage and computation speed), but this would make composer much more difficult to use (you would need to install a C extension), to upgrade (the PHP code would have to check that the C extension implementing the solver is at the same version) or to develop (it would require having C developers). If we want to make it easy to use composer, we would then have to make the usage of the C extension optional, which would mean duplicating all the complex algorithm in C and in PHP.

what about inspiration from some linux package manager like APT: http://people.debian.org/~dburrows/model.pdf

I din't read the document (yet), but i'm using debian/apt for long time and it seems working good.
Except the fact it's written in C/C++, maybe the algorithm is better ?

The solver in Composer was ported from the C implementation of the libzypp sat solver http://en.opensuse.org/openSUSE:Libzypp_satsolver and its use is based on the findings of the EU research project EDOS: http://www.mancoosi.org/edos/ which analyzed package managers like APT in detail - I'd really appreciate it if we could limit the suggestions to properly researched and concrete solutions here.

Sorry, i just found EDOS too.

Can I throw this thought into the ring? I may be far off, but would this work?

It seems that ultimately the result of the dependency checking is to create the contents of composer.lock Given just that file alone, it is possible to install all the packages it references with little memory.

So, to generate composer.lock, is any local information needed other then the contents of composer.json? If we had access to a server with enough resources, could it be given composer.json and return composer.lock to enable an installation?

I have done this myself on one project - I copied composer.json to another server (actually a production server, but it was our biggest), run composer update, deleted all the packages it downloaded, then copied composer.lock back to my low-memory dev server to run a composer install. It appeared to work, but I'm not sure if it has any fundamental flaws.

Obviously this is just a workaround, but we need a workaround until a better solution is found. Is there a command-line option for composer.phar to tell it to generate composer.lock, but not actually bother to download all the full packages?

Edit: if I would update as a dry-run, but get access to the resulting composer.lock (composer.dry-run.lock perhaps?) then that would help so much. But like I say, maybe the generation of the lock file, and the downloading of the full packages, cannot be split into two distinct halves like that.

I've raised this as a feature request: #2656

@judgej The issue would happen for dependencies on platform packages (i.e. PHP and extensions). Otherwise, it is possible to do it on another server. For instance, I always run update on my local machine, as I want to test the code before deploying it anyway. The prod server will only run install on deployment.
And the idea of externalizing it is used in http://composer.borreli.com/ which even externalizes the installation. It will give you the vendor folder as well

@stof that "composer as a service" site is just what I imagined - brilliant.

The problem with dependencies on the PHP version is similar to the use of globals - they make testing hard. Every external factor must be overridable to be able to do any sensible testing, and could then be used to build against different environments to the one composer is running on.

@stof, previously I change memory_limit to use 1024M, everything get resolved and follow my work, but now, today I'm trying to run the same command "composer update" and again gives me the same error ...

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 71 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/Solver.php on line 700

How's posible running with 1024M? :/

I created an alias and that solved the issue:

alias php_turbo='php -c /vagrant/bin/memory_boost.ini'

╭─vagrant@box.dev /vagrant/Symfony git:(Feature/Queue*) ╰─ php_turbo -i | grep mem Loaded Configuration File => /vagrant/bin/memory_boost.ini memory_limit => 2048M => 2048M

I increased my memory limit to 1024M, problem solved. Still, it's funny having to use 1 GB RAM just to download 28 MB of vendor files, right? I can't imagine if we add a couple more dependencies then we're gonna need to buy extra RAM chips...

Started having this issue since today...
We are using a Laravel package on a private repo in our projects. So if the package is updated and pushed on our private repo all we need to do on our projects is call

composer update ourvendor/ourpackage

This has been working fine for several weeks now but suddenly we get the following error

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 67108864 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/Solver.php on line 170

We only update a specific package, which isn't large at all and still get the error..
Hopefully this issue can be addressed soon as we use a VPS with 512 MB as our staging environment so upgrading the RAM seems a bit overkill. And perhaps in time even 1GB won't be enough.. :(

The composer command is installed globally on my machine in /usr/local/bin/composer so a workaround is to call the composer update command as follows :

 php -dmemory_limit=1G /usr/local/bin/composer update ourvendor/ourpackage

I can confirm it's happening again.

To reproduce it:

{
    "name": "laravel/laravel",
    "description": "The Laravel Framework.",
    "keywords": ["framework", "laravel"],
    "license": "MIT",
    "require": {
        "laravel/framework": "4.1.*",
        "cartalyst/sentry": "2.1.*",
        "way/generators": "dev-master",
        "raveren/kint": "dev-master",
        "zackkitzmiller/tiny": "1.0.1",
        "pragmarx/glottos": "dev-master",
        "pragmarx/firewall": "dev-master",
        "davejamesmiller/laravel-breadcrumbs": "dev-master",
        "way/form": "dev-master"
    },
    "autoload": {
        "classmap": [
            "app/commands",
            "app/database/migrations",
            "app/database/seeds",
            "app/tests/TestCase.php"
        ],
        "psr-0": {
            "ConsultorioDigital": "app/"
        },
        "files": [
            "app/ConsultorioDigital/Support/helpers.php",
            "app/ConsultorioDigital/Exceptions/all.php"
        ]
    },
    "scripts": {
        "post-install-cmd": [
            "php artisan optimize"
        ],
        "post-update-cmd": [
            "php artisan clear-compiled",
            "php artisan optimize"
        ],
        "post-create-project-cmd": [
            "php artisan key:generate"
        ]
    },
    "config": {
        "preferred-install": "dist"
    },
    "minimum-stability": "dev"
}

Install packages

composer install

Remove a single vendor folder:

rm -rf vendor/pragmarx

And update with sources:

composer update --prefer-source

Removing the whole vendor folder:

rm -rf vendor

And installing with sources

composer install --prefer-source

Gives me no errors.

Hi,
Thanks for the detailed explanation @stof, I guess the way to go is to work with as much as possible restrained version until a better approach on the algorithm is found (any PR initilized on that subject ?)
The command we use now is rather extrem but works on my workstations at least.

php -d memory_limit=-1 /usr/local/bin/composer.phar update

@cenob8 we are having the exact same problem. chokes without memory when trying to update a specific package.

The same issue, my composer.json is:

{
    "name": "laravel/laravel",
    "description": "The Laravel Framework.",
    "keywords": ["framework", "laravel"],
    "license": "MIT",
    "repositories": [
        {
            "type": "composer",
            "url": "http://packages.cartalyst.com"
        }
    ],
    "require": {
        "laravel/framework": "4.1.*",
        "way/generators": "dev-master",
        "cartalyst/sentry-social": "3.0.*",
        "mrjuliuss/syntara": "1.*",
        "electrolinux/phpquery": "dev-master",
        "webignition/absolute-url-deriver": "*",
        "barryvdh/laravel-debugbar": "1.*"
    },
    "autoload": {
        "classmap": [
            "app/commands",
            "app/controllers",
            "app/models",
            "app/database/migrations",
            "app/database/seeds",
            "app/tests/TestCase.php"
        ]
    },
    "scripts": {
        "post-install-cmd": [
            "php artisan optimize"
        ],
        "post-update-cmd": [
            "php artisan clear-compiled",
            "php artisan optimize",
            "php artisan debugbar:publish"
        ],
        "post-create-project-cmd": [
            "php artisan key:generate"
        ]
    },
    "config": {
        "preferred-install": "dist"
    },
    "minimum-stability": "dev"
}

The error is the next:

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 32 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/RuleWatchGraph.php on line 52

As said before, for the moment, until they have a solution, you'll have to raise PHP memory for Composer to work:

php -d memory_limit=-1 /usr/local/bin/composer.phar update

or

php -d memory_limit=-1 /usr/local/bin/composer update

@antonioribeiro worked for me, but had to rebuild the server with more memory :|

I did some tests because @stof suggested to require a higher version.

I require "laravel/framework": "4.1.*", in my root composer.json, which in turn requires "symfony/http-foundation": "2.4.*", and a lot of other packages. This constantly results in a memory error (> 512MB).
If I add the symfony packages to my root composer.json, like this (copied from laravel/framework/composer.json)

"symfony/browser-kit": "2.4.*",
"symfony/console": "2.4.*",
"symfony/css-selector": "2.4.*",
"symfony/debug": "2.4.*",
"symfony/dom-crawler": "2.4.*",
"symfony/finder": "2.4.*",
"symfony/http-foundation": "2.4.*",
"symfony/http-kernel": "2.4.*",
"symfony/process": "2.4.*",
"symfony/routing": "2.4.*",
"symfony/translation": "2.4.*"

And run composer update again, this will work. When I time both commands without memory limits (time php -dmemory_limit=1G /usr/local/bin/composer update), after removing composer.lock and the vendor dir, the original composer.json took between 1m and 1m40s, the version with all the symfony requirements took around 30 seconds.

Isn't it possible to take the the requirements from the required packages also into account. So first download laravel/framework/composer.json, parse and add those requirements. Then load the those packages skip everything that doesn't match those requirements? (versus the current, only check the root requirements) Maybe just 1 or 2 levels extra.
(But this will perhaps make it harder to be specific in which packages do not match, but make it a lot faster..)

I am having a similar issue. At first I had the standard PHP memory limit error, then I just made sure I have -1 limit and shut down MySQL to save memory, because I have a soft 1G and burst 2G memory limit on my VPS server.
Unfortunately, that's not enough, I get:

Fatal error: Uncaught exception 'ErrorException' with message 'proc_open(): fork failed - 
Cannot allocate memory' in 
phar:///usr/local/bin/composer/vendor/symfony/console/Symfony/Component/Console/
Application.php:985

What I am trying to run is a simple composer install on a Laravel project with no extra dependencies:

        "require": {
                "laravel/framework": "4.1.*"
        },
[…]
        "scripts": {
                "post-install-cmd": [
                        "php artisan optimize"
                ],

Does adding those symfony I mentioned dependencies in your require section help?

Hm. That worked. Isn't the full dependency list built the same way no matter where the dependencies are in fact included?
Can I safely remove the symfony dependencies from the composer.json file after a successfull install to keep it clean?
(And thank you, @barryvd)

Apparently the requirements from the root composer.json are used to determine what package versions should be kept for consideration, but this doesn't happen for the dependencies of those packages. So when adding those symfony packages to your composer.json, to ignore everything < symfony 2.4 (the most), but when you let Laravel require them, it loads all versions in the (composer) history of symfony in memory, so it keeps growing every release..

@barryvdh There is a memory optimization for root dependencies, because we know the constraint exactly and so we can restrict the constraint for other packages based on the root requirements. Such optimization cannot be done based on your dependencies (well, we could in the case where you have an exact match constraint for a requirement, at the expense of making the error output even more confusing in case of conflict, but this is a very edge case anyway as you are encouraged to allow at least bug fix releases for your dependencies)

Yeah okay, it would be probably difficult because you have multiple options. But wouldn't it be possible to first loop through the possible dependancies and save all the requirements, and check based on that?
In the laravel example, which loads laravel/framework 4.1: load all versions that match and save the requirements. They all require symfony ~2.4 (or some older ~2.3). So now we know the minimum, continue loading all the matching symfony versions etc.
(No idea what is possible, just noticing that with the current solution, the memory size will grow with time. So what is now 512 MB will soon become 1G, 2G etc and the time to update will also increase..)

Solved for me with just commiting composer.lock

Just noticed that it is does seem a lot faster now then a few days ago. If I roll back to the latest release (I can't self-update to a specific commit), it goes from 10 secs (new release) to 45 secs (old release). Probably less memory to, so perhaps has to do with the changes in the replace functionality..

I can confirm this behavior, with an update from today, composer update took ~40 seconds and used at its peak 280 Mo while it was using more than 1Go and 2 minutes to complete before the update

It's working much better now

I'm limitted to 512mb and I have still been unable to find a fix.

@richiejenkins I can confirm HHVM does a fine job at this

ran into same issue , assigning more ram was the only way for me.

In addition to upping the php memory_limit setting to 1G, I had to make sure the VM I was running had more than 1G RAM allocated as well.

I've just faced the same problem (with Symfony project on PHP 5.3 if it matters)
and running php -dmemory_limit=1G composer.phar update didn't work.

My solution was changing memory_limit to -1 in /usr/local/etc/php/5.3/php.ini
and then just composer update.

Thanks to https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors

Lately I am also seeing allot of Cannot allocate memory error's. I didn't had any problems a few weeks a go installing new projects like Laravel.

I am running a local Vagrant box with 512MB allocated. When I watch my ram with watch -n 5 free mem -m, I start with:

             total       used       free     shared    buffers     cached
Mem:           490        338        151          5          6        100
-/+ buffers/cache:        231        258
Swap:            0          0          0

When I run composer update within the create-project folder that failed to install, due to lack of memory, I see the free mem (in this case 151MB) go lower until it reaches 0 and then composer will crash. Just wondering why it doesn't use the free buffers/cache space as well (258MB vs 151MB).

When I set my vagrant box to 1GB it will work, but 1GB is to much for my 4GB that's on Macbook Air. So I usually set it to 1GB, do whatever I need to do with composer, and then reload the vagrant box again with just 512MB. It is a pain however :/

@Ilyes512 I hear your pain, however, there are workarounds.

The memory required expands exponentially (well, fast) with the number of dependencies being checked and the number of versions each of those dependencies have. If your composer.json dependency list is very "liberal", and just asks for the latest of everything to be checked, then memory usage will grow until it busts the limits of your server.

The trick is to be more restrictive to specific versions of packages in composer.json You can look in composer.lock to see what versions of everything are installed now, and that may give you a better idea of what versions to require in composer.json Once you do this - and it may only be one or two packages, perhaps locking a package to version "1.2.x" instead of "1.x" - but that technique can get you over the memory issue.

Remember - it did work once, and now it runs out of memory. What has changed? The number of versions of all the dependent packages is what has changed. If your "update" command can be told to look at fewer combinations of these packages, then it will use substantially less memory.

Hope that helps :-)

@judgej Didn't know about the fact that being more specific will reduce memory consumption, thanks.

Most of my composer needs are just installing things like Laravel, Octobercms, phpunit, codeception. So packages that I havent altered at all. But I guess I will stick to the 1GB and go back to 512MB once done plan :)

So, I havent tried it yet, but I think if I provision my .lock file within my development Vagrant box I could get away with just 512MB for my VPS (for now).

As I noted here (#1898 (comment)), you could also supply some requirements from Laravel directly, that helped in my case. And just requiring ~4.1.28 instead of 4.1.x also helps.

And just requiring ~4.1.28 instead of 4.1.x also helps.

Maybe one of the devs can weigh in on this... is the memory reduction in the line above due to ~4.1.28 being converted into something like >=4.1.28? Is the ~ preferred over wildcard (.*|.x) matches?

I think just because you can skip all previous versions in your matches, so will probably be the same as >=4.1.28,<4.2, but shorter to write ;)

~4.1.28 is indeed equivalent to >=4.1.28,<4.2-dev, while 4.1.* is >=4.1-dev,<4.2-dev. this means that by using ~4.1.28, you exclude the first 28 4.1.x releases, as well as all pre-4.1 releases (alpha, beta and RC) if any

It is not a memory leak or HD issue. I don't exactly what it is, but I've now fixed the issue twice (two different machines) with these steps. Unsure if this helps, but I had the same problem and its is now resolved. I was getting this error [Composer\Downloader\TransportException]
The "https://getcomposer.org/version" file could not be downloaded: failed to open stream: Cannot allocate memory

I did the following. Unsure what fixed the problem, but here is what I did if it helps anyone:

1. force uninstalled brew (which btw si what I think is causing the problem in my case). I had to use the following command as my new was not uninstalling with --force or installing

rm -rf /usr/local/Cellar /usr/local/.git && brew cleanup

1. reinstalled brew new:
   ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
2. remove macports:
   sudo mv /opt/local ~/macports
3. I deleted mcrypt files from /usr/local/lib.
   /usr/local/lib/libecomlodr.dylib
   /usr/local/lib/libmcrypt.4.4.8.dylib
   /usr/local/lib/libmcrypt.la
4. deleted larval from my and composer
5. uninstalled MAMP
6. reinstalled MAMP - pointed document root to a new folder as I was getting errors with old directories
7. installed composer into new directory
   curl -sS https://getcomposer.org/installer | php
8. copied composer.phar to /usr/bin/ and /Applications/MAMP/bin/php/php5.5.10/bin/php (don't think you need it in php bin directory, but I went for overkill
9. deleted composer.phar from current directory
10. added the following paths to ~/.bash_profile

nano ~/.bash_profile

export PATH="/usr/local/bin:/usr/local/sbin:~/bin:$PATH"
export PATH="/usr/local/mysql/bin:$PATH"
alias composer='/usr/local/bin/composer.phar'
alias phpmamp='/Applications/MAMP/bin/php/php5.5.10/bin/php'
export PATH=/Applications/MAMP/bin/php/php5.5.10/bin:$PATH

1. reloaded bash profile
   source ~/.bash_profile
2. back into new created directory and loaded larval
   composer create-project laravel/laravel devbox --prefer-dist

and it worked fine.

1. I loaded facebook api, AWS, ardent in to .json file with require parameters and ran

composer update new/package

BTW, I find that is I run just composer update I get errors "Script php artisan clear-compiled handling the post-update-cmd event returned with an error"

but now everything is working again. Im sorry I can pinpoint what fixed it, but I think it was mcrypt / brew that said the errors stooped once I had composer.phar in both php and /usr/bin/ and when I updated my bash profile file.

Hope this helps

Having the same problem, started after I added PHPUnit to composer.json and ran update.

Increasing PHP mem to 1024M fixed it!

Disable xdebug while running the "composer update" command.
The xdebug cause this slowness for me.

@judgej Does being more specific help you in the case of your dependencies' requirements not being so specific? You can lock your own versions down in composer.json files you own but as for your external deps no help!

You can also put specific external dependencies in your own composer file if you want, just copy them from the external package and make them more specific.

Ah! Helpful thanks. Although it's a nightmare going through these versions and manually making sure they all work. Very slow process.

@drpheltright I usually just pick one of the main packages - a core laravel package for example - and lock that down. That is often enough to do the job. I can keep my eye on that package and raise its locked-down version number when new versions come out. Composer then handles any updated dependencies, but because the tree of versions it needs to check is then restricted, it involves a lot less memory. Composer is still handling the dependencies - I am not suggesting any of that process becomes manual.

To visualise it, if you ask composer to update everything to the latest available (php composer.phar update) then it has a massive tree of potentially tens of thousands of combinations of versions of everything that it needs to check, to determine which branch gives the latest list of compatible versions of all the packages involved. All we are doing is pruning off branches and telling composer to just go down some of those branches by kind of locking them in, and not others. It needs to keep all those branches in memory until it can make its decision, so if there are fewer to keep in memory, then there is less of a chance of running out of memory.

I also have problems with large memory consumption.
I personally don't understand why composer update and composer update twig/twig need the same amount of memory (in composer.json I just require symfony/symfony: dev-master). Shouldn't the second command take way less, because most dependencies are fixed to the currently installed versions? Shouldn't it just fetch all the possible Twig versions and check if they collide / match the requirements in composer.lock?

I was experiencing this issue as well today on the 1.0.0-alpha8 version, however I've since updated to the release on 08-11-14 and it seems to be resolved now.

Had the same problem as jkenneydaniel this morning, uninstalling 1.0.0-alpha8 (which was installed via Homebrew) to the manual cURL install version released on 08-11-14 solved the problem for me too.

I wonder, if composer code is based on C version of SAT solver (#1898 (comment)), would it be possible to make PECL binding of that same C library and avoid memory issues (the PECL extension could be optional, falling back to current implementation)

i believe current base library is https://github.com/openSUSE/libsolv
by it's description it seems to be independant and this composer repo format could be added there?

@jbonigomes That has bitten me once to. I assumed that brew would download the lastest , and it does! It just downloads the latest with a version.

To get the latest dev version just run composer self-update.

thanks @Ilyes512 will try it next time (y)

I had the same issue under OSX Lion, run the composer self-update command as @Ilyes512 suggests fixed the memory issue! Thanks

same question

Same problem

+1
on shared hosting you don't have option to increase memory size

Just adding this as a data point. Having only this in the composer.json

    "require-dev": {
        "guzzle/guzzle": "3.9.2"
    },

will cause

/usr/local/bin/composer: line 2:  9685 Segmentation fault      php -dmemory_limit=4G /usr/local/bin/composer.phar "$@"

real    4m12.120s
user    3m52.995s
sys 0m3.460s

Where as

    "require-dev": {
        "guzzlehttp/guzzle": "~5.0"
    },

results in a successful

real    0m9.633s
user    0m4.194s
sys 0m1.140s

I guess developers need to work on reducing the dependency graph, and packagist could do with a way of retiring packages which have been replaced (I guess that's the case with guzzlehttp vs guzzle).

I had to increase to 1024-2048MB for

{
    "require": {
        "drupal/drupal-extension": "dev-master"
    },
    "require-dev": {
        "igorw/composer-yaml": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true
}

I have problem. Composer use all of available ram(6GB) and 1GB Swap(max: 7GB) and freeze system... My linux lost stability and i must restart ...
I have Arch Linux with linux-ck kernel.

Please add information to disable XDebug(zend plugin)... After disabled, Composer work perfectly, without timeout, without too much eat RAM, out of memory and freezing system. ;)

thanks @TheAifam5!
you save my day!

@TheAifam5 How do you disable xdebug please ?

Thanks

Comment out any lines mentioning it in your php.ini file.

@joelrotelli you need just disable profiling feature in xdebug.
that's my debug config

zend_extension="xdebug.so"
xdebug.idekey=PHPSTORM
xdebug.profiler_enable=0
xdebug.profiler_enable_trigger=1
debug.remote_autostart=on
xdebug.remote_port=9001
xdebug.remote_enable=on
xdebug.remote_handler="dbgp"

OK thanks.

Passing to PHP5.4 solve my problem too.

so... if you used PHP Web Server, copy a php.ini from /etc/php to current directory(eg: your website). Run(if need, edit path and IP): php -S 127.0.0.1:80 -t "/home/theaifam5/www/public" -c ./php.ini
Edit copied php.ini, put all from @namreg .
Ok.. now server read local file php.ini because you specified which file.

So, now you must disable XDebug globally in /etc/php/php.ini. Replace line zend_extension="xdebug.so"to ;zend_extension="xdebug.so". If line not exist, check this path(exists on Arch Linux): /etc/php/conf.d/xdebug.ini and replace ;)

So. Globally XDebug is disabled, but for server have own configuration with enabled XDebug.. XDebug is enabled only for webserver because using own configuration.

If you using PHPStorm, add to your configration in PHP Built-in Web Server to Interpreter options: -c php-xdebug.ini.
Now PHPStorm start a webserver with XDebug, but all commands typed in console(everywhere, and in PHPStorm) starting PHP without XDebug..

:)

Yea, nice commit: 10f8e56

Shit not working(commit), no showing any warnings about XDebug ;p Bravo !

Same problem here

{
    "name": "laravel/laravel",
    "description": "The Laravel Framework.",
    "keywords": ["framework", "laravel"],
    "license": "MIT",
    "require": {
        "laravel/framework": "4.1.*",
        "phpunit/phpunit": "3.7.*",
        "barryvdh/laravel-debugbar": "1.*",
        "ammont/fa-slugify": "dev-master",
        "miladr/jalali": "dev-master",
        "intervention/image": "dev-master",
        "cviebrock/image-validator": "1.0.*"
    },
    "autoload": {
        "classmap": [
            "app/commands",
            "app/controllers",
            "app/models",
            "app/database/migrations",
            "app/database/seeds",
            "app/tests/TestCase.php",
            "app/Common.php"
        ]
    },
    "scripts": {
        "post-install-cmd": [
            "php artisan clear-compiled",
            "php artisan optimize"
        ],
        "post-update-cmd": [
            "php artisan clear-compiled",
            "php artisan optimize",
            "php artisan debugbar:publish"
        ],
        "post-create-project-cmd": [
            "php artisan key:generate"
        ]
    },
    "config": {
        "preferred-install": "dist"
    },
    "minimum-stability": "dev",
    "prefer-stable": true
}

same problem with php56 installed via homebrew with fpm and debug, it was ok with php shipped with yosemite

same here as @nvartolomei (vagrant/puphpet, php56, ubuntu14.04, windows)

@wiesson and others using laravel: try applying what @barryvdh suggested to me: #1898 (comment) — directly adding symfony subdependencies from laravel dramatically reduces the memory usage for this command. For me it worked like a charm.

I dunno why this needs so much memory but makes it a non starter on limited shared hosting environments ....

If we're supposed to only commit lock files to production, why, then, does composer not allow the installation of vendor packages without the json file?

And, again, this memory issue is killing me on shared hosting. Something needs to be done, as it's ridiculous to ask people to increase memory when they might not be able to.

@kevinm1 I think you've misunderstood the suggestion. The composer.json file must always been committed, but the lock file should also be committed as it is used in a composer install.

If the composer.lock file does not exist, then using composer install essentially runs an update, and saves the generated dependency map to the lock file. composer install then uses the composer.lock file to install the dependencies, but it also does some sanity checking to see if the composer.json file has changed.

Hope that helps.

Both files are needed.

You should run update only on dev machine (environment).

Problems like this are not always meant to be fixed by the way, hardware are better now so we can use more of it, or if you want you can rewrite entire composer in assembler .

On 19 nov. 2014, at 00:30, Kevin Major notifications@github.com wrote:

If we're supposed to only commit lock files to production, why, then, does composer not allow the installation of vendor packages without the json file?

And, again, this memory issue is killing me on shared hosting. Something needs to be done, as it's ridiculous to ask people to increase memory when they might not be able to.

—
Reply to this email directly or view it on GitHub.

Thanks to @nasr, disable xdebug fix this for me

Disable xdebug while running the "composer update" command.
The xdebug cause this slowness for me.

Recent patches improve memory usage slightly (0-50% reduction it seems depending on packages), but if you have large sets of packages it might still use a lot, and using a composer.lock to deploy to production (especially if prod has low memory like shared hosting) is still the best practice, not just for performance reasons.

I had this problem the other day. It turned out to have nothing to do with any real bug, it's just that the documentation on the composer website about how to set up composer globally on OSX using brew doesn't tell you that brew installs an old outdated version of composer, and that you have to update it with: composer selfupdate

After running composer selfupdate I didn't have this bug anymore. It took quite a bit of searching to figure that out. It would probably be easier, and maybe even reduce reports, if the composer getting started pageexplained this.

Wonderful catch, and so timely too! Thanks, @CaptainN. Your solution fixed the memory problem I ran into while installing Codesniffer.

Running

sudo composer selfupdate

Fixed it for me. I was having memory issues even when set to 5120000M!!!

==> /usr/local/bin/composer install hangs for me when running brew install wp-cli. selfupdatedoesn't seem to help.

I had same problem with:
composer create-project sylius/sylius-standard -s dev
solved for me with:
php -dmemory_limit=2G /usr/local/bin/composer.phar create-project sylius/sylius-standard -s dev

@CaptainN This worked perfectly.

I have same issue when I would install laravel 4.2 on my openshift gear.
Even I use a small gear that only has 512 MB RAM, this solved the issue
php -dmemory_limit=2G /var/lib/openshift/.cartridge_repository/redhat-php/0.0.23/usr/bin/composer.phar install

@Seldaek    for running composer install on production!
thank you man!

This solved my problem base on their Suggestion:
https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors

this worked for me as well: https://getcomposer.org/doc/articles/troubleshooting.md#proc-open-fork-failed-errors

Solve this problem.

Update PHP Version.
Ubuntu Terminal window:

1.sudo apt-get update

2.sudo apt-get install lamp-server^

That's all.

"You're on the wrong server" / "You don't have enough swap" / "Framework X is an edge-case" / "Problems like this are not always meant to be fixed"?? Whaaaat, seriously? Is everybody (with the brilliant exception of @stof and @judgej) turning into total apologists? 

The issue still remains: something is wrong, because it is insane that such an important and central tool such as the dependency manager, should gobble up gigglypiles of memory just in order to complete it's most basic tasks.

For reference: mem usage stays stable on my Yosemite Mac (PHP 5.5.20) but totally blows up on my Ubuntu Trusty deployment (PHP 5.5.9-1ubuntu4.9). None of the workarounds above has any effecty ~ it must be some triggering some leak in that older ("stable") version of PHP.

The issue still remains: something is wrong, because it is insane that such an important and central tool such as the dependency manager, should gobble up gigglypiles of memory just in order to complete it's most basic tasks.

Dependency resolution is not it's most basic task, on the contrary it is the most complex task. Being such a critical piece of the application, it is rather difficult to make improvements to. Servers generally have less memory than a development machine and it is recommended to do the update in your development environment and simply do an install from the lock on a server.

The solver is computing a solution for a graph theory equation. It basically does a tsort with an unbounded set discovering additional dependencies to consider for resolution throughout the process. The larger the graph the more memory required to compute the result. When you introduce potential nodes to the graph via discovery through a service, like packagist, that provides a large number of alternatives it increases the complexity and required memory/compute time. A traditional tsort algorithm has a complexity of O(log^2 n).

I would stress that my use of "basic" is not in the sense of "simplistic and of low complexity" - but in the meaning that this task is quintessential, and critical, to the goals of the program.

If we can conclude that there is nothing obviously wrong with the algo. implementation itself, then one way to start dealing with the problem could be to build up a dataset of known PHP version where bloat occurs and at the very least display a warning at execution time. Ideas?

It isn't a specific PHP version that causes bloat, it is a factor. Packages can have dependencies on different PHP versions. If you have a 5.3 installation of PHP you are eliminating candidates of packages that require php >5.4.

The largest factor is the dependencies themselves. For instance "symfony/symfony": "^2.7" is a much simpler constraint because it performs a replaces for all of it's subpackages with self.version... this eliminates a huge subset of possibilities because each sub package does not need to be evaluated.

The bloat generally occurs whenever you have a dependency somewhere down the line that has a large number of potential candidates. For instance, if you only require a subset of symfony packages they will need to be calculated individually to satisfy each constraint and dependency. If you drop that constraint down to ^2.3 you introduce a significant number of possibilities to evaluate.

The ideal goal of maintaining backwards compatibility with other open source packages, a good motivation IMO, is the main killer of the memory utilization of composer. When a new version is released that meets your constraint it needs to be evaluated.

"Packages can have dependencies on different PHP versions"

 Aaah darn, good point...

I turned off xdebug and still had problem. Turned off OpCode for cli and worked like a dream (and fast too).

I turned off xdebug and it works now.

Having it with HHVM too

Thanks @kureikain
Your solution solved my problem! 

I was having this issue. Updating composer (composer self-update) seems to have solved it for me...

i got this error when running composer.phar update. and i think it is similar to this issue, but i don't know how to fix it..

PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 144115188075867549 bytes) in phar:///bin/composer.phar/src/Composer/Util/RemoteFilesystem.php on line 179

this is my composer.json

{ "description" : "The CodeIgniter framework", "name" : "codeigniter/framework", "license": "MIT", "require": { "php": ">=5.2.4", "videlalvaro/php-amqplib": "2.5.*" }, "require-dev": { "mikey179/vfsStream": "1.1.*", "videlalvaro/php-amqplib": "2.5.*" } }

is it possible/normal for composer to allocate that much memory??

during an update, it can indeed use lots of memory

I might have missed some of the comments above but since this issue is still open I'll add my two coins.

My system:
VPS: 1Gb RAM, Ubuntu, ssd, no swap (I don't need it)

My situation:
I successfully updated composer to the latest version using self-update command. Then I tried to update Drush to the latest but received the memory error discussed in this issue.
I used the following command: composer global require drush/drush

I did this way because it's required by the official Drush documentation which is used by thousands so the suggestion to use composer on my local machine and then move to Git is not for me obviously.

Of course, I can temporary increase memory to 1G but folks... really... it's too much for one utility tool and it's probably not a PHP problem but a problem of the composer architecture.

So, maintainers please think about this when you're planning roadmap for future versions because I believe that no one utility tool deserves 1G of memory even if it's local machine memory.
Thanks.

@barryvdh Ain't your suggestion more like this, just a suggestion:

• Composer update
• Composer copies composer.json to composer.json.tmp
• Get all dependencies from root projects and add to require part in composer.json.tmp
• Internally run composer update again but then using the temporary file, which seems faster and less memory in your examples
• After update, remove composer.json.tmp file and write composer.lock file

这个问题好蛋疼

Fatal error: Uncaught exception 'ErrorException' with message 'proc_open(): fork failed - Cannot allocate memory' in phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:940

I'd consider this more than serious. I just tried to install Symfony CMF. I needed to increase memory limit to 2G in order to avoid constant out of memory issues. Come one guys! There is something seriously wrong with the composer. I'm managing dozens of dev machines and it was the first time for past few years that I had to increase it's memory over 1G and for what? A composer install?

With recent boom for small cloud dev machines this will become a serious issue.

@SiliconMind Be more specific about the version(s) you want to install. That is the key. Don't give composer the option to have to investigate every possible combination of every single package that is needed to install your framework; if you do, then it will be trying out many thousands of possible combinations to find the perfect latest compatible set of packages available, and it is that which eats memory.

Did you check the tipps mentioned at https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors ?

Also it would be very helpfull when you could create a blackfire profile of your run (after you increased php-memory-limit to a value high enough that it doesnt crash)

@judgej You know, I can handle myself a lot but I don't think that's the point. Consider someone less experienced that visits Symfony's CMF install instructions and sees this: composer create-project symfony-cmf/standard-edition cmf '~1.2'.

You can't expect that average John Developer will know composer inside out. And frankly, the versions for Symfony CMF are pretty precise. Maybe related packages are not specific enough, but you can't expect that all end users will now start to edit related composer.json files in hope of finding the source of the issue.

@staabm xdebug was disabled. It was vanilla debian install without any additional stuff. As for the profile - I'm sorry, I can't do this now.

I have been plagued by this issue for over two years now and the official response is still "add swap"? This should not be a problem and its ridiculous that it isn't resolved yet. If a memory issue such as this made it into anything I released I would lose my job.