Gerrit updated to 2.12.2 / Jenkins updated to 1.652

I’ve updated Gerrit from 2.11.5 to the latest release 2.12.2. These are the user-visible highlights:

  • New Submit Whole Topic setting: All changes belonging to the same topic will be submitted at the same time. Currently disabled because it’s still experimental but I will enable this once it’s considered stable.
  • Support for GPG Keys and signed pushes. You can add your GPG key in Gerrit and git push –signed to use this. This should work right now – but doesn’t for me at least. If you have more success, let me know. :)
  • New search operators, e. g. author:, committer:, commentby: and a few others.
  • Your preferences for editing and diff presentation can now be configured in your user settings.
  • Gerrit’s in-line editor has now support for Emacs and Vim key maps.
  • There are several new API calls available for those using their own Gerrit clients (yes, I’m looking at you, Kylie McClain! ;-) ).

You can find the complete release notes for the Gerrit versions here:

Gerrit 2.11.6
Gerrit 2.11.7
Gerrit 2.12
Gerrit 2.12.1
Gerrit 2.12.2

As for Jenkins, I’ve updated it to 1.652 as well. Nothing spectacular there but some bug fixes in the backend mostly; including two security fixes.

The full changelog can be found here.

P. S.: If you’re from Germany, specifically from Baden-Württemberg, Rheinland-Pfalz or Sachsen-Anhalt: STOP READING THIS AND GO TO CAST YOUR VOTE. I did.

P. P. S.: If you (want to) vote for the AfD or other fascist parties: Please let me know. I like to know my enemies.


Gerrit updated to 2.11.3 / Jenkins updated to 1.626

I’ve just updated Gerrit to the latest release 2.11.3. These are the user-visible highlights:

  • When you choose a user (e. g. to add a reviewer) inactive accounts are not suggested anymore.
  • If you use side-by-side diffs (why ever would you?!), their performance has been improved
  • If your browser supports the JavaScript clipboard API (e. g. Chromium does) that’s preferred over the old Flash widget.
  • Quite a few bug fixes.

You can find the complete release notes for Gerrit 2.11.3 here.

As for Jenkins, I’ve updated it from 1.623 to 1.626 as well. Nothing spectacular there but some bug fixes in the backend mostly.

The full changelog can be found here.

Gerrit updated to 2.11.2 / Jenkins updated to 1.620

This morning, I’ve updated Gerrit to the latest release 2.11.2. These are the user-visible highlights:

  • Automatic suggestions in the search box work again.
  • Several issues that could potentially cause data loss have been fixed.
  • Newer jgit version

You can find the complete release notes for Gerrit 2.11.2 here.

As for Jenkins, I’ve updated it from 1.617 to 1.620 as well. Lots of bugfixes were implemented the most interesting of which concerned the console (log) output that could get truncated.

The full changelog can be found here.


Quick information: Jenkins’ build scripts are in git and Gerrit

Just a small update today: You can find my build bash scripts for Jenkins in git and Gerrit now.



git clone



Please submit patches via Gerrit.

Server update: Gerrit, Jenkins & friends (some downtime today)

As some of you will have noticed, my server has had some performance issues lately. Testing patches with Jenkins could take a long time, building stages was taking up to 20 hours (!) instead of the normal two and Gerrit itself could be slow at times as well.

This was due to aging hardware (that server was assembled in 2010), one horribly performing hard disk and constant over-heating of the machine. I’ve changed a few things over the last few days and performance is much better now but still not to my satisfaction.

Today, I’ve finally had enough of squeezing performance out of rotting hardware and simply ordered a new server. Functionally, it will be identical to good old galileo but on beefier hardware, especially with much more RAM (48 GB instead of 12 GB) which is essential for my uses.

While I’m writing this, I’m copying over the entire galileo to the new server which will hopefully be finished in a few hours from now. Once that’s done, I’m going to

  • shut down all non-essential services on galileo (that includes Gerrit, Jenkins, the bots and lots of other stuff),
  • re-sync the data to the new server,
  • change the DNS records,
  • fire up all current services on the new server,
  • rejoice!

(N. B.: Should you think I’ve forgotten something important here, please comment while you still can. :-) )

This update will, thus, cause some downtime (there was some earlier today already when I was preparing the migration). So don’t be alarmed if all services (including my IRC bouncer :-) ) are gone for a while – we’ll be back!

Ideally, you’ll only even read this post when all is over. :-)

Stages – some changes

As most of you know from my earlier stage building post, our stages for amd64 and x86 are being built automatically using Jenkins. And that won’t change.


Recently, there has been a discussion about adding Emacs to the stages in addition to vim. The reaction was overwhelmingly negative; even more so than I had expected.




  • I’m the one who spends his time and actual money on building stages, deploying them to, dealing with fall-out from changes, etc.,
  • there’s no real argument against adding Emacs but “it’s not POSIX!” and “stages should be minimal!”, “only having Vim is a nice filter against the kind of user we don’t really want anyway”, “use the editor outside the stage”, etc.
  • people are told to use e4r, which stands for “editor for retards” and is neither Emacs nor vim and, thus, doesn’t fix the problem,


… this is what I’ve changed effective by May, 7th 2014


  • I’ve added Emacs to the stages I build unilaterally (the first stage with Emacs included will be built on May, 8th 2014),
  • I’m not going to change the stages set because that would influence any stages not built by me,
  • I’ve stopped deploying the stages I build to (the last stage I deployed there is of May, 7th 2014),
  • The stages I build will always be located at (where they’ve been since I started building them),
  • I’ve added the link to the stages on my server to the install guide as an alternative.


In consequence…


  • if it is that big a deal that Emacs will be included in my stages, someone else will have to build official stages manually,
  • if it’s not that big a deal, our core devs can say so on our mailinglist and I’ll gladly revert things to how they were – just with Emacs in the stages set.


Miscellaneous Updates to Gerrit, Jenkins & friends

Over the last few weeks, I’ve made a few changes to Exherbo’s Gerrit and Jenkins installation:

  • Build artifacts are stored in Jenkins:

    If a build is unsuccessful, the Paludis build log, config.log (if it exists) and the cave-resolve command used are stored. Since build artifacts are associated with a project (i. e. the repository) and not individual builds, the files have the build number prepended in their file name:

    If a build is successful, only the cave-resolve command and the detected dependencies are archived:


    This change is effective immediately for all exheres repositories in Jenkins. If you have further suggestions of files to archive, please let me know.


  • Due to the recent OpenSSL Heartbleed bug issue, I’ve re-issued the SSL certificates on my machines on April, 11th 2014. If you have a password login on galileo, now is the time to change it. For, the host key fingerprints are as follows:

1024 a8:8c:7d:87:3d:a6:61:78:8d:59:f5:52:d7:b1:3c:34 (DSA)
256 f1:08:b0:12:66:1c:72:af:2d:d8:c7:15:ca:13:f6:f2 (ECDSA)
256 68:1c:27:e2:8e:b5:42:62:98:8a:f3:04:45:c1:60:f9 (ED25519)
2048 37:38:6e:81:4a:53:24:4d:b0:fb:c7:3e:f0:1d:63:8c (RSA1)
2048 06:57:ff:b7:e6:6f:c8:31:76:c8:9a:7c:37:d7:f5:47 (RSA)

The web server’s certifacte has the following fingerprints:
  • SHA-256 Fingerprint:
    85 3D 47 8B A5 44 76 A0 09 96 9F 15 4A 9A F8 C5 1C 95 D1 1D 92 4E 78 06 0D F6 E5 7B 1B C5 B2 07
  • SHA-1 Fingerprint:
    0D E8 77 4F 74 67 AB 61 12 97 A5 57 84 1F B4 51 C1 1D D4 19


  • Gerrit was updated to version 2.8.4. There are quite a few bug fixes in the release notes but probably most important for those of you who experienced issues with the side-by-side diff view in the new change screen are the fixes they applied to it.


There’s more to come but that’s going into another post. :-)

Re-testing patches in Gerrit with Jenkins

A recent request concerned the possibility to re-test patches patches in Gerrit with Jenkins. I’ve now added a very simple switch for that:

In any given given change (but your own!), you can add a comment with “+1 Retest” set as a “verdict”. On the old change screen (really, change to the new one, it’s much nicer), it looks like this:

Retest: Old change screen


On the new change screen, it looks like this:

Retest: New change screen

It’s really, truly easy. You can even do it multiple time – just uncheck the “Retest” checkbox (new screen) or set “0 Retest” (old screen) first and then set it again.

This works for any change as long as you are a registered user – any change but your own. Unfortunately, the Jenkins’ Gerrit-Trigger plugin simply ignores the “CommentAdded” event completely for ones own changes. If you want those re-tested, just amend your commit (no need to actually change anything; the commit timestamp will be updated) and Jenkins will see the patch’s new revision and re-run the test.


On a side note, I’d like to apologise for the instability for about 1,5 days earlier this week. There were some really nasty hardware/software issue that I had to sort out. This is done now, though, and things are running smoothly again.


Automated stage building with Jenkins

For quite some time now, I’ve been working on the automatic creation of Exherbo stages for AMD64 (x86_64) and X86 using Jenkins.

Till now, Exherbo’s stages were created manually by one of us Exherbo core developers and usually basically “on-demand” or when they were major changes. This finally changed yesterday.

Using the jobs stage_amd64 and stage_x86 respectively, the stages…

  • are built using a modified stage building script – each stage is build from scratch. The latest stage gets unpacked and used to build a new one. I call this: Implicit pro-active quality assurance. No pbins are used in the entire process. Everything is fresh and new, just for you.
  • the sources are kept right next to the stages at You can see right there what sources were used.
  • the XZ archive is added to the job (the most recent ten stage archives are kept on their respective job). Like this, you can always check the exact build log for the stage you want and rest assured both are right there at your fingertips.
  • next, the successfully built stage is deployed to is undergoing maintenance but you really want your Exherbo? Download a stage from this alternative location, sync your repositories from Gerrit and you’re unstoppable!
  • and finally automatically transferred to our traditional where some “extra magic” is performed by Jenkins that creates the “current” symlinks, updates the checksum files, etc.

And this happens every single day. The entire process is completely scalable as well – once the cross stuff is production-ready, I’m going to build ARM stages as well and it will be done by simply copying a Jenkins job and changing a few lines. A milestone for Exherbo.

We just made major changes? Wait a few hours, get them in your stage for your new Exherbo installation.
You somehow got a broken stage? Use an older one, compare them, go nuts with them. There are always at least 10 stages per architecture to choose from.
And if a stage fails to build? Well, the next day it will be fixed because we’re watching over the entire process.

Isn’t this incredibly awesome?

And you can even play a round of Bullshit Bingo using this very post! ;-)


Jenkins: Build-testing for every single commit

As of yesterday, I’ve drastically increased the number of testing machines in Jenkins (from two to 21!). This enables me to poll for changes in git every five minutes and automatically start a testing run for every single commit – regardless of it entering via Gerrit or not. (Note: This is in addition to the Gerrit trigger I’ve been using from day one on, not replacing it.)

Here’s an example from today:

Apart from looking at Jenkins directly, you can join the #exherbo-bots channel where the jenkins-exherbo bot announces all test results. Here’s an example for that:

[03.01.2014 13:42:59] <irker657> Exherbo: pipping arbor:master * 76ec9309669d / packages/sys-apps/grep/grep-2.15.exheres-0 packages/sys-apps/grep/grep-2.16.exheres-0: grep: Bump to 2.16
[03.01.2014 13:46:50] <jenkins-exherbo> Project arbor build #67:SUCCESS in 1 min 5 sec:

I hope you’ll find this useful and will actually look at the results since it will find out if you missed a dependency. And it potentially shows you automagic dependencies.