14

I have a PHP project in which I load packages through Composer. I also run Continious Integration using Jenkins, on a dedicated CI server. Once every hour, Jenkins queries my repository for changes, and if present, if executes a test run.

First step of the testrun is making a fresh checkout of the repository, and performing a build of the application, using Phing. One of the steps of the build is performing an

composer install

Since Jenkins always works with a fresh checkout, composer will always fetch all packages on every test run, even if none of the packages have been changed since the previous run. This has a couple of disadvantages:

  • It takes a relativally long time to complete a test run (composer needs to fetch for example Zend Framework, which is rather large
  • It put unnecessary strain on the packagist server, if new packages are fetched every hour
  • If, for some reason, the composer install fails, so does my test run.

I was thinking of possibly storing the packages that composer fetches on a central spot at the CI server, so Jenkins would be able to access the packages at that location for every test run. Of course, now I have to rewrite part of my application to handle the fact that the vendor folder is in a different location when on the CI server. Secondly, I have to tell Jenkins to keep track of changes on the composer.lock file, to see if he needs to run composer anyway. I'm afraid none of those two things are really trivial.

Does anyone have any suggestions of a other/better way to do this, or is it the best option to just fetch all packages through composer on every test run. Admiditally, it's the best way to make sure you always use the correct packages, but it sortof feels like a waste of bandwith, certainly in later stages of development, when the list of packages will hardly change anymore.

4 Answers 4

5

One way to speed it up is to use composer install --prefer-dist which only downloads zips even for dev packages. This is preferred for unique builds since it skips the whole history of the project.

As for sparing packagist, don't worry about it too much, one build every hour isn't going to make a huge difference compared to all the open source libs that build on travis at every commit.

Sign up to request clarification or add additional context in comments.

Comments

4

One thing you could do is to store vendors in a location outside of project's workspace in jenkins so that it remains between the builds. You not necessarily need to change your application. Just update the build script so that it creates a symbolic link to the vendors location.

I use capifony for deployment and it uses this approach to keep the vendors between releases.

1 Comment

right, that way I only have to make sure that, in the build process, the composer.lock file is copied to that location, and do a composer install to check for new packages. And I might indeed be able to pull it of with only environment switches in the build file
0

One thing to note is that Composer caches packages that it downloads. So once they are downloaded the first time, they should work even if Packagist is down (not 100% sure), and network bandwidth spared (100% sure).

Second thing is: why are you running tests by doing a fresh checkout of the repository? It is entirely possible to keep a copy of your code in the workspace in Jenkins, and just make sure you wipe on every test run the caches, logs and other artifacts. This will speed up not only composer install, but also the git pulls, especially for big repos!

Side note: for our own Jenkins platform, where workspaces are not cleaned between tests, the main drawback we found with composer is the sheer amount of disk space taken by having the full vendor dir in each workspace. I tried to work around this by using symlinks and sharing the vendors (named based on hashes of composer.lock), but then composer autoloader had a bit of problems finding where to load classes from...

Comments

0

Steps to install zf2 project on Jenkins mkdir /path/to/your/project 1. Install the composer

curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer

Note: If the above fails due to permissions, run the mv line again with sudo. A quick copy-paste version including sudo:

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer
  1. create a composer.json file in the root directory of the project add all the pacakages you require

    { "name": "amarjitsingh", "description": "amarjitsingh", "license": "BSD-3-Clause", "keywords": [ "framework", "zf2" ], "homepage": "http://domain.com/", "require": { "php": ">=5.5", "zendframework/zendframework": "~2.5", "phpoffice/phpword": "dev-master", "doctrine/doctrine-orm-module": "0.7.0", "imagine/Imagine": "0.5.*", "zf-commons/zfc-user": "dev-master" }, "autoload" : { "psr.0" : "/module" }

    }

  2. run 'composer install' to install these packages.

  3. set up git on your machine if you are using ubuntu you can set up GIT using the folowing commands

    sudo apt-get update sudo apt-get install git

Set Up Git

git config --global user.name "Your Name"
git config --global user.email "[email protected]"

check the config list

git config --list

once you have setup GIT then c

cd /path/to/your/project

. once you have packes installed the create a '.gitignore' file in the dcument root and add 'vendor' inside it.

git init
git remote add origin https://[email protected]/username/zf2ci.git

apply below command to ADD, COMMIT, AND PUSH the files

git add .
git commit -m 'Initial commit with contributors'
git push -u origin master
git pull

using cloud you can use AWS . I am using digital ocean

1 create a droplet 2.name it as you wish , in mycase it is zf2ci 3. choose a package 4. choose the OS my cas eis Ubuntu 14.04 5. In applications tab choose LAMP 6 once you done with that you will get IP address, username root and password. 7. login the ip by using the putty 8. user root 9. password pass 10. once you get into it it will prompt to you to change the password 11. goto web root eg /var/www/html 12. install GIT 13. apt-get install git 14. clone the repo 15. git clone https://[email protected]/username/zf2ci.git 16. install composer on this machine

curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer

Note: If the above fails due to permissions, run the mv line again with sudo. A quick copy-paste version including sudo:

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer
  1. goto app path /var/ww/html/zf2ci
  2. run 'composer install --no-dev' we are installing it with no dev option becuasae we only install well tested code on app server

Step3

Create a Jenkins server 1. set up another droplet for Jenkins 2. image ubuntu 3.install Lamp install Jenkns

Installing Jenkins Before we can install Jenkins, we have to add the key and source list to apt. This is done in 2 steps, first we'll add the key.

1.1

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -

Secondly, we'll create a sources list for Jenkins.

1.2

echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list

1.3 Now, we only have to update apt's cache before we can install Jenkins.

apt-get update

1.4 As the cache has been updated we can proceed installing Jenkins. Note that Jenkins has a big bunch of dependencies, so it might take a few moments to install them all.

apt-get install jenkins

1.5 open the ip with port 8080 eg http://127.0.0.1:8080

1.6 install git on jenkins server

sudo apt-get update
sudo apt-get install git

1.7 install composer

curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer

1.8 enable user authentication

1.9 enable bitbucket plugin for Jenkins 1.9.1 Manage Jenkins->Manage Plugins->Bitbucket Plugin->download and install 1.9.2 create job create job-> project name(eg. zf2ci)-> source code management (git) provide ssh url([email protected]:username/zf2ci.git)-> branches to built (*/master) this is the branch where each time any user commits and merge the code with Master branch -Jenkins gets invoked

1.9.3 Build Triggers choose the option(build when a chnage is pushed) this will wok when we make a POST hook on bit bucket

1.9.4 Build->Execute shell

 composer install
 ./vendor/bin/phpunit ./tests

our tests sits intests dir

1.9.5 set a ssh key pair

login to jenkins Serevr through putty

su jenkins
    cd
    ls -la( check what is in the jenkins home directory)
    ssh-kegen -t rsa (dsa by default but choose rsa key ,it is faster)
    press enter(on path)
    press enter(leave the pass phrase empty , the whole point here is to avoid passwords in the automated jobs)
    pres enter



 cd .ssh

ls -la (you will find id_rsa.pub) file there

cat id_rsa.pub

(select all and copy the contents of the file)

1.9.6 goto bitbucket switch to the repo zf2ci goto settings click deployment keys->add key add label (jenkins) key*(paste the the contents of the id_rsa.pub)file here save key

summary
`zf2ci->settings->deployment keys->add key->type` label and paste id_rsa.pub key->save

1.9.7 register POST hook for repo

Settings->
Integrations->
Hooks->
POST(search for POST Hook)->
Add the url /IP of the Jenkins Server) (`172.62.235.100:8080/bitbucket-hook/`)
(the body of the post contanis information about the repository, branch, list of recent commits, user)

1.9.8 login to Jenkins server

su jenkinks
    cd
    cd .ssh
    git ls-remote -h ssh://[email protected]:username/zf2ci.git HEAD

1.9.9 save project on Jenkins 1.9.10 add the following command in the Execute Shell->command [rsync -y -vrzhe "ssh -o StrictMostKeyChecking=no" --exclude vendor/ . root@ipaddress:/var/www/html/zf2ci( of app server)]

ssh root@ipaddress<<EOF
    cd /var/www/html/zf2ci
    composer install --no-dev
    EOF

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.