Version bump to systemd-43 / Move to /usr

(This is the same as the news item but I want this to get maximum exposure.)

Read ALL of this, it’s important to everyone using systemd.

Up to systemd[=42] we installed boot-critical components to / and others to /usr. This split was causing issues with respect to tmpfiles, intrinsic dependencies and dependencies on stuff on /usr.

systemd[=43] finally removes this split and installs everything but udev and pam stuff to /usr.

This won’t matter much to you if you don’t have /usr split from / (it should not be split; cf. http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken).

Even if you don’t have /usr != /, you need to update all packages that install to /${LIBDIR}/systemd/system because that got moved, too, of course. I’ve rev-bumped all packages, that install their own custom systemd units but even after you’ve updated those, you’ll still have some in /${LIBDIR}/systemd/system. Find out which package they belong to (use cave owner) and re-install them.

Should you forget to do so, you might end up in systemd’s emergency mode. If that happens, don’t panic. Get your network connection up and continue updating/re-installing. You’ll live, I promise.

There might be orphaned systemd units left behind. Check those on your own and decide if you need to move them to /etc/systemd/system. If you do, don’t forget to systemctl disable and then enable them.

You’ll also have some broken symlinks in /etc/systemd/system pointing to /${LIBDIR}/systemd/system. To fix those, all you have to do is disable and re-enable the respective unit. Here’s how to do it quickly and easily:

for link in $(find -L /etc/systemd/system -type l); do
systemctl disable $(basename ${link});
systemctl enable $(basename ${link});
done

Final sanity checks:

1. Is /${LIBDIR/systemd gone? If so, carry on; if not, you missed a step. Go back and find out which one.

2. No broken symlinks in /etc/systemd/system anymore? (“find -L /etc/systemd/system -type l” doesn’t output anything) If so, carry on. Otherwise, you missed a step. Go back and find out which one.

If you do NOT have /usr separated from /, you’re done now and it should be safe to reboot if you so desire.

If you do have /usr separated from /, you’ll have to use an initramfs (preferrably created by dracut) for booting from systemd[=43] onwards.

The first step to using an initramfs is enabling CONFIG_BLK_DEV_INITRD in your kernel, recompiling and installing it. If you want to switch from a custom initramfs to dracut, don’t forget to empty CONFIG_INITRAMFS_SOURCE in your kernel configuration either if you have been using it before.

If you want to use dracut (sys-boot/dracut[>=14]), install it and add

add_dracutmodules+=”98usrmount”

to /etc/dracut.conf. If you have some weird configuration, you might need to add further dracut or kernel modules. In general, though, dracut is going to pick up everything you’ll need to boot.

Now run dracut to create your shiny new initramfs:

dracut -H <initramfs filename incl. path> <kernel version>

e. g.

dracut -H /boot/init-3.2.5.gz 3.2.5-00001-gf74dd96

-H (or –hostonly) tells dracut to build an initramfs for the machine it’s running on. Leaving it out should create a HUGE generic initramfs that should bascially be able to boot everything. In reality, leaving -H out usually builds something that doesn’t boot anything.

Next, update grub’s config so that it includes a root= parameter for the kernel command line and your new initramfs, e. g.:

(for grub-0.9x’s menu.lst)
title Exherbo Linux
root (hd0,1)
kernel /kernel-3.2.5-00001-gf74dd96 root=/dev/primary/uselv
initrd /init-3.2.5.gz

or

(for grub-1.9x’s grub.cfg)
menuentry “Exherbo Linux” {
set root=(hd0,1)
linux /kernel-3.2.5-00001-gf74dd96 root=/dev/primary/uselv
initrd /init-3.2.5.gz
}

Do NOT forget the root= parameter. It’s essential.

(Of course, you need to adjust paths and filenames to your setup but if I need to tell you that, you shouldn’t be using Exherbo in the first place.)

If you’re using a custom initramfs, you must make sure that you mount /usr as early as possible but definitely before systemd (/sbin/init) starts. If you’re rolling your own initramfs, you should know how to accomplish that.

After you’ve updated your grub configuration, systemd is updated and your kernel is ready, too, say a little prayer ;-) and reboot.

Best regards, Wulf

HowTo: systemd on Exherbo

This comes up all too often, so here’s a HowTo for systemd on Exherbo:

  • You have to run a Linux kernel >=2.6.39. The new kernel is only needed at runtime, not for building systemd.
  • You should run a Linux kernel >=3.8. The new kernel is only needed at runtime, not for building systemd.
  • Kernel options for systemd: cf. systemd’s README, here’s an excerpt:

CONFIG_DEVTMPFS
CONFIG_CGROUPS (it’s OK to disable all controllers)
CONFIG_INOTIFY_USER
CONFIG_SIGNALFD
CONFIG_TIMERFD
CONFIG_EPOLL
CONFIG_NET
CONFIG_SYSFS

Linux kernel >= 3.8 for Smack support

Udev will fail to work with the legacy layout:
CONFIG_SYSFS_DEPRECATED=n

Legacy hotplug slows down the system and confuses udev:
CONFIG_UEVENT_HELPER_PATH=””

Userspace firmware loading is deprecated, will go away, and
sometimes causes problems:
CONFIG_FW_LOADER_USER_HELPER=n

Some udev rules and virtualization detection relies on it:
CONFIG_DMIID

Mount and bind mount handling might require it:
CONFIG_FHANDLE

Optional but strongly recommended:
CONFIG_IPV6
CONFIG_AUTOFS4_FS
CONFIG_TMPFS_POSIX_ACL
CONFIG_TMPFS_XATTR
CONFIG_SECCOMP

For systemd-bootchart a kernel with procfs support and several
proc output options enabled is required:
CONFIG_PROC_FS
CONFIG_SCHEDSTATS
CONFIG_SCHED_DEBUG

For UEFI systems:

CONFIG_EFI_VARS
CONFIG_EFI_PARTITION

Furthermore:
CONFIG_FANOTIFY=y (only used for readahead stuff which is not enabled by default.)

CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y (only used for readahead stuff which is not enabled by default.)

 

  • Set the “systemd” option globally in /etc/paludis/options.conf: */* systemd
  • Install systemd: cave resolve -x sys-apps/systemd (Read what cave tells you. If in doubt, read Paludis’ documentation.)
  • Reinstall every package with the new option set: cave resolve world -cx
  • Switch to systemd as your init system: eclectic init set systemd
  • Set the desired hostname in /etc/hostname.
  • Optional: Edit /etc/vconsole.conf to your liking. (If you delete it, empty it or comment out everything, systemd will use the kernel’s defaults.)
  • Optional: Edit /etc/machine-info to your liking.
  • Read Lennart’s blog post about the other configuration files.
  • Install a Linux kernel >=2.6.39. (see above for kernel options, etc.)
  • Reboot.

After that reboot, you’ll be in a console with a minimal set of services started, hopefully ready to log in. Log in as root (the keyboard layout is set to US in vconsole.conf (see above) by default!). Then you can enable whatever services (found in /lib/systemd/system) you like, suggested ones are:

  • dhcpcd.service or NetworkManager.service
  • sshd.socket (it doesn’t start? Missing host keys? man sshd or http://tinyurl.com/24jwxjd)

As an extremely simple and limited alternative to NetworkManager.service, there’s network.service and network.conf which get installed if you set the “simple-net” option for systemd. network.service only allows for static network setups with IPv4.

Alternatively, you can use dhcpcd.service.

If I were you, I’d not enable your display manager’s service (either kdm.service, gdm.service, xdm.service or slim.service) until your basic system has at least booted properly once and you can reach your system using ssh because in case things go wrong, it’s easier not to have to wrestle with a GUI.

To actually enable a service, run “systemctl enable <foo.service>”. More details can be found in systemd’s man page.

If you need help, it’s available in #exherbo, as usual, but if you didn’t read this before asking, grumpy me will bite your head off unless you prove you read this by saying “I have furuncles on my arse.”. Yes, I’m being serious.

 

FAQ section:

  • “How/where do you specify extra modules to be loaded?” – You put the module name into /etc/modules-load.d/foo.conf and it will get loaded. Unless systemd-udev has already loaded it for you. Check that first.
  • “My hostname is set to something funny, e. g. ’08’!” – If you’re using NetworkManager, you need to set your hostname in /etc/NetworkManager/NetworkManager.conf, too.
  • “I’m getting messages about failing services, e. g. dev-hugepages.mount or sys-kernel-debug.automount. What’s up with that?” – You can either enable the corresponding kernel options, delete the symlink (e. g. /lib/systemd/system/basic.target.wants/sys-kernel-debug.automount) or just ignore those messages. They’re harmless.
  • “When sshd.socket is enabled, every closed ssh connection leaves a failed service around, e. g. sshd@192…:55140.service.” – Harmless as well. There are no ressources used by those so ignore them. (This should be fixed anyway.)
  • “Where can I learn more about the usual administration tasks? – Read Lennart’s series of blog posts about systemd for administrators: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8
  • “How do I debug problems with systemd?” – Read this page http://fedoraproject.org/wiki/How_to_debug_Systemd_problems
  • “I’m completely lost. What do I do?” – Please remember there’s always a friend around. It’s called “man”. ;-)

systemd and the Linux kernel

This comes up all too often, so here’s a HowTo for systemd on Exherbo:

  • You have to run a Linux kernel >=2.6.39. The new kernel is only needed at runtime, not for building systemd.
  • You should run a Linux kernel >=3.0. The new kernel is only needed at runtime, not for building systemd.
  • Kernel options for systemd: In your kernel config, enable autofs4, devtmpfs and cgroups. Do not enable autofs3. Here’s what I’m using (I enable more kernel options than strictly necessary, though.):

CONFIG_DEVTMPFS=y (Strictly required!)

CONFIG_DEVTMPFS_MOUNT=y (unless you're using an initramfs that's mounting it for you, e. g. one created by Dracut)

# CONFIG_AUTOFS_FS is not set (Strictly required!)

CONFIG_AUTOFS4_FS=y (Strictly required!)

CONFIG_CGROUPS=y (Strictly required!)

# CONFIG_CGROUP_DEBUG is not set

CONFIG_CGROUP_NS=y

CONFIG_CGROUP_FREEZER=y

CONFIG_CGROUP_DEVICE=y

CONFIG_CGROUP_CPUACCT=y

# CONFIG_CGROUP_MEM_RES_CTLR is not set

CONFIG_CGROUP_SCHED=y

CONFIG_BLK_CGROUP=y

# CONFIG_DEBUG_BLK_CGROUP is not set

CONFIG_FANOTIFY=y (only used for readahead stuff which is not enabled by default.)

CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y (only used for readahead stuff which is not enabled by default.)

systemd in Exherbo – what’s happened so far…

It has been quite a while since I last wrote something about my work on systemd in Exherbo, so here’s an update:

What has been accomplished so far:

  • The Exherbo patches are done. Do NOT try to submit them upstream yet, though. I’ll take care of that when the time is ripe.
  • Lots of services are done.
  • You can boot and run most systems using systemd now.
  • I’ve built new amd64 and x86 stages without any init system so you can start out without the baselayout-1/sysvinit cruft.
  • The installation guide has been updated.

Every systemd service is implemented natively and we’re not using anything from baselayout-1 or sysvinit anymore. Instead, all the important stuff has been moved to skeleton-filesystem-layout. systemd’s dependencies have been updated accordingly.

Thus, for people using systemd, baselayout-1 and sysvinit are now obsolete. YAY!

What still needs to be done:

  • Improve existing service definitions for systemd.
  • Create socket definitions for several of the existing service definitions. (And new ones, of course.)
  • Create systemd service files for missing services.

Rules for new service files:

  • Please make sure they’re implemented natively. I won’t accept non-native service files unless you can convince me there’s definitely no other solution.
  • If you convinced me, scripts go to /usr/${LIBDIR}/systemd.
  • EnvironmentFiles (configuration) go to /etc/conf.d and end in .conf. We do NOT create a configuration file for every single service but create (grep-able) logical units, e. g. now-obsolete font@.service and keymap@.service used to use console.conf).
  • You can reference environment variables from configuration files in service. If you have to quote the values in the configuration file, you need to use $FOO; if you don’t quote them (preferred), you use ${FOO}. This is probably a bug (and known to upstream) but for now that’s how it is.
  • Services and their (potentially) accompanying files must not collide with baselayout-1.

Requirements:

  • You have to run Linux kernel >=2.6.36-rc1 (I’m using 2.6.36-rc6; latest NVidia-Drivers work fine and there are patches for the VMWare modules available.).

How to get started with systemd:

Read this.

Conclusion:

Since both systemd and its exheres have now reached an acceptable degree of stability, I don’t intend on breaking things anymore as I’ve done over the last months from time to time.

In fact, systemd is so usable these days, I’m writing this on a systemd-initialised system! This means as well that I can live without baselayout-1 and sysvinit. YAY! :-)

systemd in Exherbo – Rules of Engagement

As you may have noticed, I’ve recently added systemd to ::philantrop for use in Exherbo. I’m writing this to

  • warn you that I will break systemd (and consequently your boot process) until further notice recklessly, repeatedly and without prior warning to anyone
  • make clear what I intend to do with systemd in Exherbo
  • make a plan for myself.

What I want to do first is get a feeling for systemd and see if it might have the potential to replace baselayout-1 (bl-1) and, at least for myself, be used instead of the init-system-that-is-not-to-be. ;)

As I really want to replace bl-1, I’m not going to go the Gentoo way of simply adding a handful of pseudo-units that essentially just call the openrc init scripts. If you want that, you’re on your own and I won’t accept patches that do that. Instead, I’m aiming for:

  • a full native set of systemd units not tainted by anything else
  • a minimal set of non-native configuration files (what we currently have in /etc/conf.d; I don’t think it will be possible to avoid them completely but I will if I can)
  • staying as near to upstream as possible and I’ll try to submit my patches upstream even though the DISTRO_PORTING instructions don’t exactly make inclusion seem likely
  • the units included in the systemd package will at most get you to some kind of login (either graphical or console)
  • all additional units for services like sshd should eventually be added to their respective packages, possibly using a “systemd” option.

The process to make systemd really usable in Exherbo will be a slow one. One that I expect to take till autumn this year because:

  • I’m really busy at work,
  • this summer seems to become a damn hot one again and I spend quite some time after work in our pool,
  • I’ll be on holidays in France for most of July.

The steps I intend to take:

  • finalise the Exherbo patches for systemd (90% done, ETA: Mid June)
  • create and enhance a basic set of units for booting (5% done)
  • create units for other services

How you can help:

  • Remind me of the stuff I’ve forgotten due to a power outage here. :-)
  • Make yourself acquainted with systemd units.
  • Once I’ve pushed the Exherbo patches, test systemd and
  • submit patches for the missing units. :-)
  • Leave comments here or on the dev mailinglist so that I can consider your input, comments and concerns.