FAQ

From ciportal

Contents

General

How can I get help?

You can contact the support through the ci-support@inria.fr mail address.


I have multiple projects to set up. Should I create multiple Jenkins projects in the portal or should I create a single Jenkins project with multiple jobs ?

Short answer: if these projects concern the same group of developers and/or if they have dependencies with each other, then you may prefer to create a single Jenkins project. In other cases you should create multiple projects.

Longer answer: there are multiple criteria you need to take into account:

  • privileges are granted in a per-project basis, not on a per-job basis. For example, you cannot have a public and a private job in the same Jenkins project.
  • Jenkins can handle dependencies between jobs (eg. a software that depends on libraries). Especially it can be configured trigger a new build if one dependency is updated.
  • slave nodes cannot be shared between projects. If you have to set up a very specific configuration for your different projects, then you will need to clone thes slave nodes.


Portal

What is the username to connect to the platform?

The username to log on the various platform is usually your email address (for the portal, the jenkins websites and the cloudstack front-end). To log on the virtual machine network, you have to use the uid provided on the front-end http://ci.inria.fr when click on your email address (upper-right) and "My account".


Why do I have to create a new account to log on the CI portal, while it would be simpler to log in with my INRIA iLDAP account?

Because all users are not INRIA users only, it is necessary for the CI platform to have its own LDAP. Synchronisation with iLDAP would be tricky. A single sign-on may be implemented in the future, but it's not considered to be a priority.


I work at INRIA (or a partner lab) but my account is identified as a "Guest"

When creating your account, you must use the same e-mail address as the one registered in the Inria LDAP directory.

If you do not know this address, you can make a query at this page: https://annuaire.inria.fr/


Jenkins

The list of available plugins is empty

In the plugin configuration page (Manage Jenkins -> Manage Plugins), go into the Advanced tab and click on the button Check now at the bottom right of the page.


Display CTest results in Jenkins

CTest results can be displayed using the xUnit plugin and the jenkins-ctest-plugin. To do so:

1. Install the xUnit plugin in Jenkins

2. Go to your execution slaves and get the jenkins-ctest-plugin:

su ci -c "cd && git clone https://github.com/aitzkora/jenkins-ctest-plugin.git" (Linux)

cd c:\builds
git clone https://github.com/aitzkora/jenkins-ctest-plugin.git (Windows)

3. Add a shell build step (Windows batch command on Windows) to your job in Jenkins to do the test and get the XML results (if your CMake build directory is "build"):

cd build
cmake -P /builds/jenkins-ctest-plugin/run-test-and-save.cmake (Linux)
cmake -P c:\builds\jenkins-ctest-plugin\run-test-and-save.cmake (Windows)

4. Add a "Publish xUnit Results" post-build step to your job in Jenkins and click "Add" / "Custom tools". Set the pattern to be "build/CTestResults.xml" (again, if "build" is your CMake build directory) and the custom stylesheet to "/builds/jenkins-ctest-plugin/ctest-to-junit.xsl" (if /builds is the ci user home as provided by the default templates).


I want to authorize anonymous to see my builds

Got to Manage Jenkins / Configure System / Authorization, set read for 'anonymous' (overall) in job's column.

I want to trigger a build remotely

  1. The first problem that comes is to find a way to use this API without using your own account. Indeed, it is a bad idea to share your credentials with anyone having access to this script. To solve this issue, you can use the Jenkins Build token root plugin. You can simply install it through the Jenkins plugin manager.
  2. Then, you can configure your job to accept remote build triggers by providing a TOKEN in the job configuration (Build Triggers>Trigger builds remotely (e.g., from scripts)>Authentication Token)
  3. From a remote script, you can simply invoke:
wget --no-check-certificate 'JENKINS_SERVER/buildByToken/build?job=NAME&token=SECRET'

where:

  • JENKINS_SERVER is the URL used to reach your jenkins server
  • NAME is the job name
  • SECRET is the token you specified in the job configuration

The same URL can be used when defining a POST hook in a SCM like Github or Bitbucket.

I want to automatically run a job after a commit (using SVN SCM)

  1. Go to job configuration for <YOUR_JOB>, click on "Trigger builds remotely (e.g., from scripts)" (FR "Déclencher les builds à distance") and set the token to a value: <TOKEN>. Don't forget to save your configuration.
  2. Click on your username <USER> in the left top corner in Jenkins windows, next "Configure..." et "Show API Token...", copy the value of <APITOKEN>.
  3. Now you can use wget command in your post-commit script, you can try it in a shell with wget.

To add a post-commit script on gforge.inria.fr, you have to connect using you SSH key to scm.gforge.inria.fr and put your lines in /svnroot/<GFORGEPROJECT>/hooks/post-commit or /gitroot/<GFORGEPROJECT>/<GFORGEPROJECT>.git/hooks/post-receive.

Replace <USER>, <APITOKEN>, <PROJECT> (the project unix name as provided in the portal), <YOUR_JOB> and <TOKEN> with your personnal values:

  • With wget 1.11 or upper, use this:
 wget --auth-no-challenge --http-user=<USER> --http-password=<APITOKEN>
      http://ci.inria.fr/<PROJECT>/job/<YOUR_JOB>/build?token=TOKEN
  • With wget 1.10.x:
 wget http://<USER>:<APITOKEN>@ci.inria.fr/<PROJECT>/job/<YOUR_JOB>/build?token=<TOKEN>


I want to automatically run a job after a commit (using Git SCM)

If you are using GIT plugin on Jenkins, you have a simplified method to run your job after commit :

  1. Go to job configuration for <YOUR_JOB>, click on "Poll SCM" (FR "Scrutation de l'outil de gestion de version") and set the value to: "# Leave empty. We don't poll periodically, but need polling enabled to let HTTP trigger work". Don't forget to save your configuration.
  2. Now you can use wget command in your post-commit Git script, you can try it in a shell with wget.

To add a post-commit script on gforge.inria.fr, you have to connect using you SSH key to scm.gforge.inria.fr and put the next wget command in /gitroot/<GFORGEPROJECT>/<GFORGEPROJECT>.git/hooks/post-receive.

 wget --auth-no-challenge --no-check-certificate 
    http://ci.inria.fr/<PROJECT>/git/notifyCommit?url=ssh://<GFORGE_JENKINS_USER>@scm.gforge.inria.fr//gitroot/<GFORGEPROJECT>/<GFORGEPROJECT>.git

A more complete script that support branches is post-receive. You just have to adapt the first lines (the PROJECT variable and the GFORGEPROJECT variable).

I want to grab file X from the Jenkins server / get the ip address of the Jenkins server

You can do lot of stuff using the groovy script. Go to "Manage Jenkins" / "Script console" and you will be able to type scripts. For instance, if you wish to execute the "hostname" command to grab the server host name, do:

 def command = """hostname"""
 def proc = command.execute()
 proc.waitFor()
 println "stdout: ${proc.in.text}"


I want to use subversion on a private project with public key authentication

1. Create a ssh key pair on the build slave by running « ssh-keygen »

 su ci -c ssh-keygen

Now the new key pair should be located in ~/.ssh/id_rsa.pub (public key) and ~/.ssh/id_rsa (private key). On Windows these two files should be located in %HOME_DIR%\.ssh

2. You should have an account with read rights on your private SVN repository, ideally (for security reasons) this account should be dedicated to your jenkins instance and only have access to the project you want to build. It is not recommended to use your personnal account.

3. Install the public key (~/.ssh/id_rsa.pub) on your subversion repository to allow the ci user to authenticate using his key (if you are using INRIA's forge, ssh keys are configured on this page: https://gforge.inria.fr/account/editsshkeys.php, wait 1 hour before the keys is actually deployed on the Inria's forge).

4. Configure your Jenkins job (in the web interface of Jenkins)

  • In the « Source Code Management » section, select « Subversion »
  • Fill the « Repository URL » box with the URL of your SVN repository (typically starting with svn+ssh://). Jenkins will report an error « No credential to try. Authentication failed (show details)

(Maybe you need to enter credential?) », this is normal, click on "credentials" and then select "SSH public key authentication". Enter the username of the ci account on the subversion repository and select the SSH key (the ~/.ssh/id_rsa file). Enter the passphrase if any.

  • Finish the configuration of your Jenkins job and save your changes.

I want to use a private GIT repository with public key authentication (this is the case for INRIA's forge)

1. Install the GIT plugin into your Jenkins instance

2. Install git on the build slave (Note: for windows slaves you should install the « Git for Windows » from http://msysgit.github.com/)

3. Create a ssh key pair on the slave by running « ssh-keygen » AS THE ci USER (Note: for windows slave this command should be run in « Git Bash » shell accessible in the Start menu).

 su ci -c ssh-keygen

Now the new key pair should be located in ~/.ssh/id_rsa.pub (public key) and ~/.ssh/id_rsa (private key). On Windows these two files should be located in C:\users\ci\.ssh

4. Your ci user on the build slave needs to have his « author » and « email » properties configured (even if you do not indend to do any commit)

 git config --global user.name "CI"
 git config --global user.email ci@dummyuser.ci.inria.fr

5. The ssh host key of your repository needs to be recorded in the known_hosts file of the ci user. To do that, just open a ssh session to the remote server and accept the server key. Thus if your repository is git+ssh://scm.gforge.inria.fr/gitroot/... then you will run:

 ssh scm.gforge.inria.fr
 
 The authenticity of host 'scm.gforge.inria.fr (131.254.249.52)' can't be established.
 RSA key fingerprint is 00:0c:90:8b:9d:b7:91:b8:ce:75:85:3b:c0:ee:73:07.
 Are you sure you want to continue connecting (yes/no)? yes
 Warning: Permanently added 'scm.gforge.inria.fr,131.254.249.52' (RSA) to the list of known hosts.
 Permission denied (publickey).

6. You should have an account with read rights on your private GIT repository, ideally (for security reasons) this account should be dedicated to your jenkins instance and only have access to the project you want to build. It is not recommended to use your personnal account.

7. Install the public key (~/.ssh/id_rsa.pub) on your git repository to allow the ci user to authenticate using his key (if you are using INRIA's forge, ssh keys are configured on this page: https://gforge.inria.fr/account/editsshkeys.php)

8. Configure your Jenkins job (in the web interface of Jenkins)

  • In the « Source Code Management » section, select « Git »
  • Fill the « Repository URL » box with the URL of your GIT repository (typically starting with git+ssh:// or ssh://). Jenkins will report an error « Failed to connect to repository », this is normal. Just ignore it and save your changes.

9. (Windows slaves only) Configure the HOME environment variable in your node:

  • Go to the configuration page of your node (menu Manage Jenkins -> Manage Nodes -> your node -> Configure)
  • In the « Node Properties » section, tick the « Environment variables » box
  • Click on the « Add » button to add a key/value pair
  • Set the name to « HOME » and the value to « C:\Users\ci » (this is the directory containing the .ssh/ configuration subdirectory)


I want to use Jenkins with a project hosted on Inria's GitLab server

Please refer to the GitLab tutorial to connect Jenkins with GitLab.

You can also find some good practices on this blog post from SED Sophia

We are not encouraging it, but if you prefer to go with gitlab's integrated CI pipelines

The GIT plugin freezes during the build (eg. when cloning the remote repository)

This is most likely due to an error in the configuration. The Jenkins plugin filters the standard error output of the git command, thus you are left with very little feedback.

Check the previous section of this FAQ to ensure that everything is well configured.


I want to use the output of a job as the input of another job, how can I do that?

You can that by installing the Copy Artifact Plugin in your Jenkins instance, then add a post-build action to your first job to archive the files to use as input on your second job. Finally, you add a "copy artifacts from another project" step in your second job.


Jenkins cannot send emails.

Please check :

  • either 'sender email address' in the 'email notification' category for Jenkins versions < 1.474,
  • or '' in the Jenkins Location category for newer versions.

It must be an email address from an inria domain (ex: jenkins@ci.inria.fr).


Jenkins closes SVN connection.

Caused by: org.tmatesoft.svn.core.SVNErrorMessage: svn: Sorry, this connection is closed.
     at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:101)
    at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector2.open(SVNSSHConnector2.java:153)

It's a well-known issue in SVN plug-in 1.37, upgrading to 1.45 or 1.50 resolves issue.


Update Jenkins connection method to old Windows slave

In May 2013, the procedure used by Jenkins to join Windows slaves has changed. This new configuration is automatic for new created slaves and is describe in the official documentation

However, old slaves require update for both Windows part on the slave and in Jenkins.

Before, make sure to suppress the old Windows Jenkins service on the slave: "sc delete <SERVICENAME>"


Update in Jenkins
  • Launch Method : Let Jenkins control this Windows slave as a Windows Service
  • Administrator User Name : .\<username>
  • Password : ****
  • Host : <IP addr>
  • Run service as : Use Administrator account give above


Update in Windows on the slave
  1. Start the Remote Registry
  2. Configure the firewall
  3. Add permission for the WBEM Scripting Locate
  4. Change/Add registry key LocalAccountFilterPolicy
  5. Allow administrator user to launch a service:
    • run the console secpol.msc
    • In Local Policies -> User Rights Assignment: add to user the right "Log on as a Service"

Jenkins failed to launch a Windows slave

  1. Check what you need to do on this Jenkins page Windows slaves fail to start via DCOM
  2. Configure the firewall, or even turn off the firewall (Control Panel / System and Security / Windows Firewall / Turn Windows Firewall on or off)
  3. Check the Jenkins service is starting when launching (Start menu and then type services.msc in the research bar). If Jenkins service is not there, install Jenkins as a Windows service


Jenkins failed to launch a linux slave via SSH (Server rejected the 1 private key(s) for ci)

Jenkins uses its SSH key to open a session on then Linux slave, to make it work the public key must be authorized on the slave.

  1. Get the public SSH key of your jenkins server. The key can be downloaded from the "Slaves" tab of your project in the web portal (https://ci.inria.fr/project/PROJECTNAME/slaves)
  2. Paste the content of the key (a line starting with ssh-XXXX) into the ~/.ssh/authorized_keys of user 'ci' on the slave. Be careful not to split the key over multiple lines, do not insert any newline character

I cannot find "Manage credentials" in the administration panel (and there is a helpless "Configure credentials" entry)

In version 2.0 of the credentials plugin, the "Manage credentials" entry was removed from the administration panel. You should follow the "Credentials" link from the main menu (on the left). Note that there are now multiple scopes for credentials, to add new credentials you should go into the "(global)" scope.

Note for French users : the credentials plugin is now translated. The French translation for "Credentials" is "Identifiants"

CloudStack

I can't connect to the CloudStack plateform : status said Invalid username or password

Maybe you have not enter the right login/password(Caps Lock, bad password). Most of the time, it is due to an invalid domain name, i.e. the third field in the form. Like it is explained in the next point the domain name is

   ci/the_name_of_my_project

it is NOT the name of the virtual machines!

What is the domain name to log on the CloudStack website?

The domain name to log on the CloudStack website is ci/projectname where projectname is the unixname of the project created on the continuous integration portal (http://ci.inria.fr)


How to open ports on the slaves?

In the CloudStack platfrom, go to the Network tab, select the Security Group Views in the Select view combobox on the top. Click on the default item and select the Ingress Rule You can now open new ports by entering new rules.


How can I connect to Windows? The remote desktop is not working...

First, make sure you have created a SSH tunnel (explained here for Linux/MacOS X and here for Windows). Second, the default Remote Desktop Client is not working properly most of the time in Mac OS X (maybe on some other systems too). Please try another client such as CoRD


Where to put big templates or ISOs for uploading in CloudStack?

Temporarily you can put these big files on scm.gforge.inria.fr. See some documentation here : http://siteadmin.gforge.inria.fr/FAQ.html#SCM

  • Copy file on GForge : scp <file> <username>@scm.gforge.inria.fr:/home/groups/<project>/incoming/<file>
  • In GForge go to Documents tab : https://gforge.inria.fr/docman/?group_id=<projectid>
  • Submit a new document, choose option uploaded file and your <file> in the list below.
  • Now the URL to download your file from CloudStack is : https://gforge.inria.fr/docman/view.php/<projectid>/<fileid>/<file>
  • Don't forget to put read access for anonymous to documents in tab Administration.


How to restore a slave from a snapshot?

The related documentation can be found here

How to increase (or decrease) the CPU/RAM allocated to a VM

  1. Stop your VM
  2. Log in the CloudStack user interface and select the instance (VM) you want to resize
  3. Hit the "Change Service Offering button" and select a new service offering (the meaning of each offering is the same as in the slave creation page).
    Offering.png
  4. Restart the VM (which may take a long time if the VM needs to be relocated to another physical machine)

Quota per projects

Here are the quotas for each ci project :

 . CPU limits : 40
 . Memory limits : 40Go
 . Primary Storage limits : 400Go   (Slaves storage - ROOT + DATA DISKS)
 . Secondary Storage limits : 400Go (Templates + Iso)

Each ci project has also a limit of 20 slaves.

If you go beyond this limit, you may have an error message of type : "Maximum number of resources of type '********' for domain id=xxxxx has been exceeded.", where type can be primary_storage, secondary_storage, ...

Slaves

General

How is defined a slave hostname?

The hostname of a slave in the cloud infrastructure (thus, from the SSH gateway) is : projectname-slavename.


Should I create one or multiple slaves ?

You will create multiple slaves if your software needs to support multiple environments, like:

  • different operating sysems (Linux, Windows, ...)
  • different linux distributions (Ubuntu, Debian, Fedora, ...)
  • different x86 CPU (32 bits or 64 bits)

Thus you have to create one slave for each configuration you want to support.


ssh root@my-slave.ci says it cannot find my-slave.ci. What is this .ci domain anyway ?

Indeed .ci is not a real internet domain name.

The procedure to connect to the slaves is a little cumbersome because they lie in a private network (and it requires to open a tunnel through the ci-ssh.inria.fr frontend).

To make it easy, we recommend to set up a proxy command in your ssh configuration file. Once this configuration is done, ssh root@my-slave.ci should bring you to the right place.


What is the public IP address of my slave ?

The slaves lie in a private network. When connecting to the Internet their private address is translated as 128.93.193.1 (cloud-gw.inria.fr).


cat /proc/cpuinfo (or Windows) gives me information that are not consistent with the information seen on the Web Portal, what is right?

Actually, the service offer give twice more timeslots to a 2GHz virtual machine that to a 1GHz virtual machine (so it is not a processor speed per se). The displayed frequency about the CPU on the virtual machine is the actual host CPU frequency (and not the allocated CPU timeslot).


My slave crashed. Even after a reboot I cannot connect to it. What can I do ?

Through the CloudStack interface you can have access to the console of your slave and fix what is preventing it to boot.

Once on the CloudStack dashboard:

  1. go to the Instances tab
  2. select the slave you want to debug
  3. click on the >_ button

Then a new window will open and give access to the console of your slave, just like if your were sitting in front of it.

CloudStack-Console.png

I want to share some data trough NFS between some slaves. How could I do ?

We detail the method considering that slaves and master are ubuntu. Adapt the installation of nfs client and server with respect to your distributions.

  • first install a server on one of your slaves
  apt-get install nfs-kernel-server
  • add a line to the /etc/exports file and restart nfs server
  echo "/builds/toshare xx.xx.xx.xx (rw,nohide,insecure,no_subtree_check,async) ">>/etc/exports
  /etc/init.d/nfs-kernel-server restart
  • install a nfs client on a client
  apt-get install nfs-common
  • edit the ip rules on cloudstack : in the tab Network, in "select view" field : select "security groups" and click on "default" and select "Ingress Rule" tab

network tab in cloudstack

  then add a TCP rule with on port 2049 with CIDR containing the ip of the slave / 32
  Start PORT : 2049
  End PORT : 2049
  CIDR : xx.xx.xx.xx / 32 (where xx.xx.xx.xx is the ip adress of your slave)   
  • add a mount line in the /etc/fstab on the slave
  server_name:/builds/toshare /builds/tomount/ nfs rw,user  0 0
  • mount the file system on the slave
 mount /builds/tomount
Jenkins's agent fails to start: "Slave JVM has not reported exit code. Is it still running?"

This happens with old versions of Java. If java -version displays 1.6 (or lower) then you should upgrade your JVM.

Known problem with GitHub connexions

Anonymous requests are limited at 60 reqs/hour per IP (only one for all the Virtual machines of the entire CI). Authenticated requests are limited at 5000 reqs/hour per user (user = authenticated GitHub user).

As a lot of project are using anonymous connection to GitHub, sometimes connection are closed from the GitHub servers. This can lead to errors like:

  SEVERE: Unable to connect to GitHub repo

or:

  $ brew search qglviewer
  Error: GitHub API rate limit exceeded for 128.93.193.1. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

or even:

  $ git clone https://github.com/blabla/blabla.git
  Cloning into 'blabla'...
  remote: Counting objects: 41841, done.
  remote: Compressing objects: 100% (51/51), done.
  Receiving objects: 22% (9382/41841), 42.39 MiB | 51.00 KiB/s 
  $

The best recommended practice is to use git plugin configured to use http notifications from github hooks.

If you are using github to clone a third party library for your project, it is advised to clone through https autentication : https://help.github.com/articles/which-remote-url-should-i-use/

Linux

What is the password for the templated Linux slaves?

The default root password is password (except for Ubuntu 16.04 where there is no root account but ci user is in the sudoers).

The default password of ci (the user running the jenkins jobs) is ci.

Mac

What is the password for the templated Mac slaves?

The default password of ci (the user running the jenkins jobs) is ci (ci user is in the sudoers).

Windows

What is the password for the templated Windows slave?

The default administrator account created is ci with the password ci

How to change a password via rdesktop

Do not hit Ctrl-Alt-Del (really!)

To change the user password, select: StartWindows SecurityChange Password


How to set up a build in the MinGW/MSYS shell?
#!c:\mingw\msys\1.0\bin\sh -login
cd "/c/builds/workspace/$JOB_NAME"


I added an extra hard drive to the VM but it is not detected by the OS

You may have a problem with the SCSI driver.

  1. go to the device manager (Right click on My Computer -> Manage -> Device Manager)
  2. select the SCSI controller
  3. install the latest viostor driver from Redhad (https://fedoraproject.org/wiki/Windows_Virtio_Drivers#Direct_download). Be careful to select the right driver (windows version & architecture) or windows will crash very badly (do not even use automatic detection, select the driver manually from the right folder).
I upgraded the Virtio Storage Driver (viostor.sys) and windows is definitely crashed

Ok, you installed the virtio driver and got the blue screen of death. Now the VM can no longer boot, the "Safe Mode" fails, the "Automatic Startup Repair" fails. Where do you want to go today?

You need to remove the viostor driver manually.


How to install a Unix environment and a sshd daemon

You can install MSYS2 (https://sourceforge.net/p/msys2/wiki/MSYS2%20introduction/) and follow the instructions to install SSH server sshd running with cygrunsrvd are here : https://gist.github.com/samhocevar/00eec26d9e9988d080ac.

Do not forget to create /etc/passwd file before running the installation script.

This solution is the robustest, but the environment of ssh connection is Unix.

How to install a Windows daemon based on OpenSSH

You can install OpenSSH for Windows, which is in beta development by Microsoft.

The instructions to install Win32 OpenSSH are here : https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

Grid'5000

For people working on Grid'5000, it is possible to use Grid'5000's API from ci.inria.fr. In particular, people with software using specific hardware features that are not available on cloudstack slaves (typically Infiniband networks, specific processor generations or topologies, ...) are welcome to use Grid'5000 as part of their continuous integration process. For this, they are allowed to request the creation of a specific user whose password might be shared between project members as part of CI tasks. They should request that this user is made member of the 'agents' group in its affiliation so that kind of usage can be tracked. Special care must be taken that the associated password does not end up in any version control software.