The Great Merge II: Exherbo’s multi-arch/cross implementation has been merged to master

Exherbo’s multi-arch/cross implementation1 has been merged to master.

You WILL have to migrate to this new setup in order to be able to
install new software and updates.

You’ll find the migration guide at 2. Make sure to follow it VERY
closely because getting things wrong does have the potential to break
your system. (You’ll of course be able to access your data via a rescue
system if everything goes wrong. We’re changing the base system but are
NOT touching user data.)

If you don’t want to migrate right now, you shouldn’t update or install
ANYTHING anymore until you are ready to do so.

The last commit prior to the migration has been tagged as “pre-cross” in
all official repositories so that you can stick with that if need be.
If you wish to stay on the pre-cross revision, you need to set
“sync_options = –revision=pre-cross” in the repository’s configuration

If you do migrate (which you should), you’ll find that some packages
might not yet install. As always, you’re fully expected to help with
migrating exheres to the new base system which basically involves
undoing our former multi-lib approach. You’ll find more details at 3
and you’re, of course, encouraged to look at recent commits that mention
“multiarch” or “cross” for tips on converting other packages.

Furthermore, we have no control over 3rd-party/unofficial repositories.
While quite a few of them feature a “cross” branch right now, they might
not have been merged when the official repositories were.
Thus, you might have to add “sync_options = –branch=cross” (without the
quotation marks) to the respective repository’s configuration until the
maintainer has merged the cross branch into master.

Cross/multi-arch: Get going! Merge date: TOMORROW, Sunday, April 5th 2015

I’d like to remind everyone that we are going to merge all the cross branches to their respective master branches

TOMORROW, Sunday, April 5th 2015!

Whatever you’re doing Exherbo-wise, please DROP IT for the moment and help us get more packages ready!

Migrating exheres to the new base system isn’t (most of the time) complicated: It basically involves undoing our former multi-lib approach. You’ll find more details here and you’re, of course, encouraged to look at recent commits that mention “multiarch” or “cross” for tips on converting other packages.

You may, of course, test cross-compilation to something but what’s REALLY important is that the package works on a NATIVE system, e. g. boost does work natively but fails at cross-compilation currently. For now, that’s ok.

If you want to convert packages in a third-party repository that doesn’t have a cross branch yet, please get in touch with me – I can create cross branches for EVERY repository in Gerrit.

To get started:

So, go for it!

(And always remember: My statistics will record your fabulous work for generations to come who will sing the praise of the immortal heroes who made cross work! ;-) )

(Questions? -> #exherbo)

Exherbo: Not perfect…

… but we welcome everyone:






Gerrit: Update to 2.10-rc1, GitHub, OpenID & other stuff

As you will have noticed, after a hard crash and failed efforts to restore the existing Gerrit, it had to be re-installed from scratch. With it, some changes had to be made:


  • OpenID is no longer available for authentication as Google is deprecating it.
  • GitHub is now used for authentication. So you must have an account there.
  • cgit is integrated into Gerrit and can be used stand-alone as well.
  • All repositories had to be re-imported. If you have a repository in Gerrit, please contact zlin (via IRC or email) or myself (via IRC or email) to get your permissions set up again.


I’ve updated my post Gerrit Code Review for Exherbo as well.


P. S.: Due to being ill, I might not be available.

Quick information: Jenkins’ build scripts are in git and Gerrit

Just a small update today: You can find my build bash scripts for Jenkins in git and Gerrit now.



git clone



Please submit patches via Gerrit.

Server update: Gerrit, Jenkins & friends (some downtime today)

As some of you will have noticed, my server has had some performance issues lately. Testing patches with Jenkins could take a long time, building stages was taking up to 20 hours (!) instead of the normal two and Gerrit itself could be slow at times as well.

This was due to aging hardware (that server was assembled in 2010), one horribly performing hard disk and constant over-heating of the machine. I’ve changed a few things over the last few days and performance is much better now but still not to my satisfaction.

Today, I’ve finally had enough of squeezing performance out of rotting hardware and simply ordered a new server. Functionally, it will be identical to good old galileo but on beefier hardware, especially with much more RAM (48 GB instead of 12 GB) which is essential for my uses.

While I’m writing this, I’m copying over the entire galileo to the new server which will hopefully be finished in a few hours from now. Once that’s done, I’m going to

  • shut down all non-essential services on galileo (that includes Gerrit, Jenkins, the bots and lots of other stuff),
  • re-sync the data to the new server,
  • change the DNS records,
  • fire up all current services on the new server,
  • rejoice!

(N. B.: Should you think I’ve forgotten something important here, please comment while you still can. :-) )

This update will, thus, cause some downtime (there was some earlier today already when I was preparing the migration). So don’t be alarmed if all services (including my IRC bouncer :-) ) are gone for a while – we’ll be back!

Ideally, you’ll only even read this post when all is over. :-)

Stages – some changes

As most of you know from my earlier stage building post, our stages for amd64 and x86 are being built automatically using Jenkins. And that won’t change.


Recently, there has been a discussion about adding Emacs to the stages in addition to vim. The reaction was overwhelmingly negative; even more so than I had expected.




  • I’m the one who spends his time and actual money on building stages, deploying them to, dealing with fall-out from changes, etc.,
  • there’s no real argument against adding Emacs but “it’s not POSIX!” and “stages should be minimal!”, “only having Vim is a nice filter against the kind of user we don’t really want anyway”, “use the editor outside the stage”, etc.
  • people are told to use e4r, which stands for “editor for retards” and is neither Emacs nor vim and, thus, doesn’t fix the problem,


… this is what I’ve changed effective by May, 7th 2014


  • I’ve added Emacs to the stages I build unilaterally (the first stage with Emacs included will be built on May, 8th 2014),
  • I’m not going to change the stages set because that would influence any stages not built by me,
  • I’ve stopped deploying the stages I build to (the last stage I deployed there is of May, 7th 2014),
  • The stages I build will always be located at (where they’ve been since I started building them),
  • I’ve added the link to the stages on my server to the install guide as an alternative.


In consequence…


  • if it is that big a deal that Emacs will be included in my stages, someone else will have to build official stages manually,
  • if it’s not that big a deal, our core devs can say so on our mailinglist and I’ll gladly revert things to how they were – just with Emacs in the stages set.


Miscellaneous Updates to Gerrit, Jenkins & friends

Over the last few weeks, I’ve made a few changes to Exherbo’s Gerrit and Jenkins installation:

  • Build artifacts are stored in Jenkins:

    If a build is unsuccessful, the Paludis build log, config.log (if it exists) and the cave-resolve command used are stored. Since build artifacts are associated with a project (i. e. the repository) and not individual builds, the files have the build number prepended in their file name:

    If a build is successful, only the cave-resolve command and the detected dependencies are archived:


    This change is effective immediately for all exheres repositories in Jenkins. If you have further suggestions of files to archive, please let me know.


  • Due to the recent OpenSSL Heartbleed bug issue, I’ve re-issued the SSL certificates on my machines on April, 11th 2014. If you have a password login on galileo, now is the time to change it. For, the host key fingerprints are as follows:

1024 a8:8c:7d:87:3d:a6:61:78:8d:59:f5:52:d7:b1:3c:34 (DSA)
256 f1:08:b0:12:66:1c:72:af:2d:d8:c7:15:ca:13:f6:f2 (ECDSA)
256 68:1c:27:e2:8e:b5:42:62:98:8a:f3:04:45:c1:60:f9 (ED25519)
2048 37:38:6e:81:4a:53:24:4d:b0:fb:c7:3e:f0:1d:63:8c (RSA1)
2048 06:57:ff:b7:e6:6f:c8:31:76:c8:9a:7c:37:d7:f5:47 (RSA)

The web server’s certifacte has the following fingerprints:
  • SHA-256 Fingerprint:
    85 3D 47 8B A5 44 76 A0 09 96 9F 15 4A 9A F8 C5 1C 95 D1 1D 92 4E 78 06 0D F6 E5 7B 1B C5 B2 07
  • SHA-1 Fingerprint:
    0D E8 77 4F 74 67 AB 61 12 97 A5 57 84 1F B4 51 C1 1D D4 19


  • Gerrit was updated to version 2.8.4. There are quite a few bug fixes in the release notes but probably most important for those of you who experienced issues with the side-by-side diff view in the new change screen are the fixes they applied to it.


There’s more to come but that’s going into another post. :-)

Re-testing patches in Gerrit with Jenkins

A recent request concerned the possibility to re-test patches patches in Gerrit with Jenkins. I’ve now added a very simple switch for that:

In any given given change (but your own!), you can add a comment with “+1 Retest” set as a “verdict”. On the old change screen (really, change to the new one, it’s much nicer), it looks like this:

Retest: Old change screen


On the new change screen, it looks like this:

Retest: New change screen

It’s really, truly easy. You can even do it multiple time – just uncheck the “Retest” checkbox (new screen) or set “0 Retest” (old screen) first and then set it again.

This works for any change as long as you are a registered user – any change but your own. Unfortunately, the Jenkins’ Gerrit-Trigger plugin simply ignores the “CommentAdded” event completely for ones own changes. If you want those re-tested, just amend your commit (no need to actually change anything; the commit timestamp will be updated) and Jenkins will see the patch’s new revision and re-run the test.


On a side note, I’d like to apologise for the instability for about 1,5 days earlier this week. There were some really nasty hardware/software issue that I had to sort out. This is done now, though, and things are running smoothly again.


Automated stage building with Jenkins

For quite some time now, I’ve been working on the automatic creation of Exherbo stages for AMD64 (x86_64) and X86 using Jenkins.

Till now, Exherbo’s stages were created manually by one of us Exherbo core developers and usually basically “on-demand” or when they were major changes. This finally changed yesterday.

Using the jobs stage_amd64 and stage_x86 respectively, the stages…

  • are built using a modified stage building script – each stage is build from scratch. The latest stage gets unpacked and used to build a new one. I call this: Implicit pro-active quality assurance. No pbins are used in the entire process. Everything is fresh and new, just for you.
  • the sources are kept right next to the stages at You can see right there what sources were used.
  • the XZ archive is added to the job (the most recent ten stage archives are kept on their respective job). Like this, you can always check the exact build log for the stage you want and rest assured both are right there at your fingertips.
  • next, the successfully built stage is deployed to is undergoing maintenance but you really want your Exherbo? Download a stage from this alternative location, sync your repositories from Gerrit and you’re unstoppable!
  • and finally automatically transferred to our traditional where some “extra magic” is performed by Jenkins that creates the “current” symlinks, updates the checksum files, etc.

And this happens every single day. The entire process is completely scalable as well – once the cross stuff is production-ready, I’m going to build ARM stages as well and it will be done by simply copying a Jenkins job and changing a few lines. A milestone for Exherbo.

We just made major changes? Wait a few hours, get them in your stage for your new Exherbo installation.
You somehow got a broken stage? Use an older one, compare them, go nuts with them. There are always at least 10 stages per architecture to choose from.
And if a stage fails to build? Well, the next day it will be fixed because we’re watching over the entire process.

Isn’t this incredibly awesome?

And you can even play a round of Bullshit Bingo using this very post! ;-)