How to manage per-build specific files during the Jenkins execution?

Question

I am getting started on adding continuous integration to an EC2 project using Jenkins. After the build execution I am deploying the build artifacts in the virtual machines.

Now having some problem with the intermediate files I need to maintain in Jenkins. During the deployment execution my glue ant task is running from the workspace/ folder. At this time I need to create several intermediate files that are this particular build specific.

How to maintain this files? I am aware of the Jenkins directory structure. (Jenkins documentation link on this)

Should I create these files under the builds/[BUILD_ID] folder.

However I do not find a way to get the this folders absolute path. (However I can deduct it from WORKSPACE variable which is available in Jenkins) I already checked Jenkins Set Environment Variables.

Answer 1

Jenkins can store artifacts for each build. You can specify the folders that you want to archive. (However, I simply find it easier to copy all of my required artifacts under a single folder. This way, I cam simply specify that one folder and not worry if I am missing anything.)

If you need these files for deployment based upon the build, I would store them as build artifacts in Jenkins. These are stored in http://$JENKINS_URL/$JOB_NAME/$BUILD_NUMBER/artifact/... where ... is the directory structure of your workspace. For example, I tend to build everything in my Java projects under target and store my artifacts under the target/archive folder (Makes it easier to save all of your artifacts if they're under a single folder) If I have a file foo.txt stored as an artifact, it's URL would be:

http://$JENKINS_URL/$JOB_NAME/$BUILD_NUMBER/artifact/target/archive/foo.txt

I can use wget or curl to pull down this artifact.

You may also decide (if you're using Ant anyway) to zip up or create a tarball of all of your artifacts in one easy to grasp file. That's what I do. Actually, there are usually three files in my target/archive directory:

• A zip file or tarball of all the artifacts I need for deployment (including scripts and intermediate files).
• A file called DEPLOYMENT_DIRECTIONS.txt which contains deployment directions.
• A file called deploy.sh that is my actual deployment file.

In my DEPLOYMNET_DIRECTIONS.txt file are these directions:

1). Log onto the Server

2). Execute the following curl command to download the deploy script:

    curl -o deploy.sh "@JENKINS_URL@job/@JOB_NAME@/@BUILD_NUMBER@/artifact/target/archive/deploy.sh"

    You may copy this and paste it right into the server where you're deploying.

3). Run the deploy.sh script.

    $ bash deploy.sh

Inside my build.xml is something like this:

    <copy todir="${archive.dir}">
        <fileset dir="${main.config.dir}">
            <include name="DEPLOYMENT_DIRECTIONS.txt"/>
            <include name="deploy.sh"/>
        </fileset>
        <filterset>
            <filter token="JOB_NAME"        value="${env.JOB_NAME}"/>
            <filter token="BUILD_NUMBER"    value="${env.BUILD_NUMBER}"/>
            <filter token="JENKINS_URL"     value="${env.JENKINS_URL}"/>
        </filterset>
    </copy>

The <filterset> replaces the @...@ variables with the environment variables from Jenkins itself. Now, my DEPLOYMENT_DIRECTIONS.txt look like this:

1). Log onto the Server

2). Execute the following curl command to download the deploy script:

    curl -o deploy.sh "http://jenkins.vegicorp.com/jenkins/server_app/15/artifact/target/archive/deploy.sh"

    You may copy this and paste it right into the server where you're deploying.

3). Run the deploy.sh script.

    $ bash deploy.sh

You can cut and paste that line from direction #2 right on the server.

A few notes:

• I use curl instead of wget because curl -o will overwrite an existing deploy.sh file. This way, I don't have to worry that wget will download the newest one deploy.sh.2 on me and someone accidentally runs the older deploy.sh from a previous build.
• I say base deploy.sh because I don't have to worry about paths, and setting the executable bit on the deploy.sh shell script. It's just easier to tell people to use bash deploy.sh than specify the chmod u+x, then do ./deploy.sh.

Hope this helps.

By the way, this was designed with three different comfort levels of groups:

• Group #1: Wants to log onto the machine where the deployment is taking place, and select the particular build they want to deploy. These groups follow the three steps in the DEPLOYMENT_DIRECTIONS.txt file in the build they want to manually deploy.
• Group #2: They still want to select the build to be deployed, but are comfortable with doing this directly from Jenkins -- never logging into the server. I use the Promoted Build Plugin to automate what's in the DEPLOYMENT_DIRECTIONS.txt file. That's pretty simple to do since it's just downloading deploy.sh and running it.
• Group #3: Wants to automatically deploy every build. We simply change the Promoted Build Plugin to run the promotion after every successful build.

Hope this helps.