GRUB2 Editor v0.6.4

After 15 months, GRUB2 Editor received a new release today. Highlights of this release are: GRUB submenu support (introduced in GRUB 2.00), LibQApt 2.x support (for removing old entries), proper check for memtest entries and various [bug] fixes. All in all, updating to the latest version is highly recommended.

kcm-grub2

Submenu support in GRUB2 Editor

Changelog:

v.0.6.4 (12/10/2013)
——————–
*ADDED: Preliminary submenu support.
*ADDED: Preliminary LibQApt 2.x support.
*FIXED: Manage some ImageMagick crashes.
*FIXED: Improved security by detecting GRUB paths at compile-time.
*FIXED: Fixed the encoding issues in the helper.
*FIXED: Check for the memtest script in the helper.

Available Translations:
Catalan (ca), Catalan (Valencian) (ca@valencia), Czech (cs), Danish (da), German (de), Greek (el), Spanish (es), Estonian (et), Finnish (fi), French (fr), Irish Gaelic (ga), Galician (gl), Hungarian (hu), Italian (it), Lithuanian (lt), Norwegian Bokmål (nb), Dutch (nl), Punjabi/Panjabi (pa), Polish (pl), Portuguese (pt), Brazilian Portuguese (pt_BR), Romanian (ro), Russian (ru), Slovak (sk), Slovenian (sl), Swedish (sv), Turkish (tr), Ukrainian (uk), Chinese Traditional (zh_TW).

I would like to thank all KDE translators who were involved. Their work is very much appreciated!

Packages:
There are packages for most major distributions (i.e. Arch Linux, Debian, Fedora, openSUSE, Ubuntu). Please take a look at the Downloads page.

Gentoo-Fu: Optimizing the build process

This is another Gentoo post, not strictly related to KDE, but a follow-up to my previous post, building KDE branches (such as 4.9 and master). Ccache can greatly reduce the build time for these constantly updated packages, but here I’ll present some other tips & tricks. These optimizations are applied system-wide and will therefore improve the build process for all packages, not just live/KDE-specific.

These optimizations are backed up by testing data which I myself performed on my system (Intel Core i5 3570K @ 3.40 GHz, 8GB DDR3 RAM @ 1333Mhz) in a proper environment (each run was performed after a cold boot to avoid any unwanted caching by the OS). The following optimizations were tested:

  • Moving the build directory to RAM (tmpfs)
  • Quiet build (redirect build output to log files)
  • Building multiple packages at once
  • Performing debug-less builds

Another possible optimization is setting emerge’s niceness to a more appropriate level, but I didn’t get around to testing it.

Building from RAM & Quiet Builds

The first test was timing the build process for kdelibs (without ccache and after a cold boot as explained above). It includes the first two optimizations in order to create a nice comparison table (HDD/RAM building and --quiet-build=y/n). Furthermore this test was conducted both in a non-accelerated KMS console (booting in single-user mode) and in an accelerated X-console (Konsole in a KDE session).

Before dumping the results, I will explain how the used optimizations were achieved:

Moving the build directory to RAM (tmpfs):

This is easily done with the following command:

mount tmpfs /var/tmp/portage/ -t tmpfs -o uid=250,gid=250,mode=0775,size=75%

To make it permanent add the following line in /etc/fstab:

tmpfs /var/tmp/portage tmpfs uid=250,gid=250,mode=0775,size=75% 0 0

So we actually mount a tmpfs instance over /var/tmp/portage (portage’s default build path), assign proper read/write privileges to the ‘portage’ user and group (preserving the original mode bits) and set a maximum size of 75% of your RAM. As a side note, tmpfs mounts act as a stack: when you umount this tmpfs instance, the original /var/tmp/portage contents will be revealed underneath (more information in this excellent article). Setting the appropriate permissions is quite straight-forward; I have just retained the original values. Lastly, setting a size of 75% of your RAM ups the default value of 50% (useful for packages with a large code base – i.e. chromium). You really shouldn’t set ‘size=0′ (i.e. disable size restrictions): a memory-hungry build will irreversibly hog and crash your machine instead of getting killed for running out of memory (more information in the above article and in the kernel’s following file: Documentation/filesystems/tmpfs.txt). This isn’t very cool: I did this with a chromium debug build on RAM and the kernel eventually crashed!

I would also advise you to enable the “fail-clean” FEATURE in your /etc/make.conf to free valuable space in your RAM when a build fails.

Quiet build (redirect build output to log files):

You can enable quiet builds by appending --quiet-build=y to emerge’s arguments. By default it is set to ‘n’. To make it permanent, just append it to the EMERGE_DEFAULT_OPTS variable in /etc/make.conf.

Before proceeding, you should also set PORT_LOGDIR="/var/log/portage" in /etc/make.conf to redirect the build output to log files in this directory, otherwise the build output is simply lost. This way you lose no functionality, you can always monitor the build output from another virtual console.

Lastly, the “clean-logs” FEATURE will periodically delete old build logs (older than one week by default).

So, having explained the optimizations used, I will present the test data. The test was run only once in each configuration but is quite representative:

Non-accelerated console:

HDD build: 10m 3.707s

RAM build: 9m 52.246s

HDD+quiet build: 8m 35.364s

RAM+quiet build: 8m 22.819s

Accelerated console:

HDD build: 8m 47.189

RAM build: 8m 33.321s

HDD+quiet build: 8m 35.187s

RAM+quiet build: 8m 23.945s

Obviously the non-accelerated console lags considerably when printing the build output. Other than that, a RAM build saves you ~12s in this case, plus considerably less HDD spinning noises. A quiet build gives another 10s speedup. It’s really not that great a speedup, but both optimizations are easily applied, so why not have them?

Building multiple packages at once

For this test, I timed the build process for amarok, k3b, kdevelop and ktorrent in a single run. Same conditions as above, with RAM+quiet build enabled in a KMS console. This test was quite straightforward: the only thing changing was the number of parallel jobs that portage was processing (-j emerge argument):

1 parallel job (-j1): 9m 9.950s

2 parallel jobs (-j2): X 8m 9.904s

3 parallel jobs (-j3): 8m 7.315s

Unlimited(=4) parallel jobs (-j): X 8m 10.261s

X denotes a build failure due to the compiler crashing. So, although it brought some nice speed improvements, I wouldn’t recommend running multiple jobs due to instability. I guess this is more oriented towards distcc/multiprocessor-powered systems.

Performing debug-less builds

This is not really an optimization; I included it mainly for comparison to the debug builds above and to lay out the procedure of enabling per-package debug-less builds. This can be really useful for certain packages (such as chromium) which fail to build in-RAM due to memory shortage (debug builds are extremely memory-consuming).

So the only test done here was building amarok, k3b, kdevelop and ktorrent (as above), with RAM+quiet build enabled in a KMS console, one parallel job, after disabling debug symbols generation (explained below). The result is 7m 47.164s. I actually expected this to be lower, but it seems that debug builds are not that time-consuming after all.

To enable debug symbol generation (extensive article here) I have “-ggdb” in my CFLAGS variable in /etc/make.conf and “compressdebug splitdebug” in the FEATURES variable. This generates debug symbols for all installed packages (located under /usr/lib/debug).

To disable debug symbol generation you have to make a per-package configuration (a list of excluded packages from the debug configuration mentioned above). Create /etc/portage/package.env and add the following line (chromium used as an example):

www-client/chromium nodebug.conf

We inform portage that the www-client/chromium package will be installed with the configuration variables listed in /etc/portage/env/nodebug.conf. The contents of this file are the following:

CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"
FEATURES="${FEATURES} -compressdebug -splitdebug"

CFLAGS here is exactly the same as in /etc/make.conf, excluding “-ggdb”. So, you get the idea. Another good use of per-package environment variables, related to this post, is specifying a different build directory (which is not on RAM). Instructions here.

Conclusion

All in all, no optimization will yield more than a few seconds speedup. However, after running all these tests, I have permanently enabled RAM+quiet builds. Performing the build from RAM makes way more sense: you keep HDD activity at a minimum so that other I/O processes can run in parallel without considerable lagging, you get a lot less noise, plus a small boost in the build times. Same goes for quiet builds which bring only advantages (you can always monitor the build output from another virtual console as explained above).

Gentoo-Fu: Building KDE branches

Being a happy Gentoo user myself for about half a year, I thought I would share some tips from my personal experiences on this great distro. It’s nothing you cannot already google for; these are just some ideas/motives to further dig into for yourselves. Mayhaps I will write another post or two about Gentoo. If I ever decide to overcome my laziness :) Take it easy with this post, it’s a bit lengthy, but to quote Blaise Pascal: “I haven’t had time to make it shorter yet“.

Gentoo being a source-based distribution allows for some very cool stuff like building from an upstream git branch. You can find ebuilds for KDE branches 4.9 and master (as of 17.08.2012), which can vastly help you with bug triaging/fixing. Bug triaging is as easy as updating your system from this branch and trying to reproduce bugs (the procedure is fully automated thanks to Portage’s Moo Powers – “emerge --moo” – and the Gentoo Developers). Bug fixing is as easy as writing a patch and applying it using Portage’s excellent patching abilities. I actually *fixed* a bug like this recently (Bug #297209), being too lazy to manually pull and compile the source code. Sure, a manual setup is way more flexible, but doesn’t come without quite some hassles.

These ebuilds are available from overlays [1] and there is a full-blown guide on how to install KDE from these ebuilds [2]. I won’t go into much detail here, these guides are more than enough; I will simply mention what the situation is like on my system. Actually I will simply quote my system updating process (it slowly builds into a ritual the more you delve into Gentoo) and more details will pop up in time:

# layman -S; emerge --sync
# emerge -uDNavbk --with-bdeps=y @world
# FEATURES="ccache" smart-live-rebuild -E -- -av1b
# emerge -ca; emerge -Pa; revdep-rebuild -- -av1b
# eclean-dist -dn; eclean-pkg; emaint -f all

So, let’s go through this and don’t forget: man pages are your friend.

# layman -S; emerge --sync

Updates the overlays and portage’s cache. Nothing fancy here. This is your “sudo apt-get update” friend, masqueraded. Or “sudo pacman -Sy”. Whatever; all distros share some core principles.

# emerge -uDNavbk --with-bdeps=y @world

Now, this is a beast of command line options. All over the net you will find references to -uDNav (I think it has almost become Gentoo’s “sudo apt-get upgrade”), but I have spiced it with some extra options. I will go shortly through these but “man emerge” is the way to go.

-u is shorthand for --upgrade. It pulls the latest versions of the packages specified – here we specified the @world set which is laid out in your world file [3]. In other words, these are your explicitly installed packages (all the other installed packages are dependencies to these).

-D is shorthand for --deep. It takes into consideration the whole dependency tree of the packages specified, not just their immediate dependencies. Without this you would only be updating the packages specified in the @world set along with their immediate dependencies.

-N is shorthand for --newuse. It takes into consideration packages that may not have new versions but whose USE flags have changed since last compilation. After meddling with USE in /etc/make.conf or /etc/portage/package.use, I always issue this update command to rebuild packages as necessary.

-a is shorthand for --ask. Always include this option (or -p/--pretend instead), otherwise Portage will move on without confirmation. You don’t want that.

-v is shorthand for --verbose. A little verbosity is always nice :)

-b is shorthand for --buildpkg. Just before installing a package, this option creates a binary package out of it and stores it in /usr/portage/packages/. It may hurt a bit your compilation times and your disk’s free space, but it comes in really handy if you move back and forth KDE’s branches like me. When I was testing my patch in multiple branches (4.8, 4.9, master) recently, the effort+time to switch between them was minimal. But this requires the next option:

-k is shorthand for --usepkg. Practically it instructs Portage to use binary packages created with -b, to [greatly] shorten compilation times.

–with-bdeps=y updates the build dependecies too. These are normally not included, Portage needs to be explictly told to do so, and I use it merely because I want a fully updated system (improves my Gentoo Feng-Shui).

So practically -uDNav is the de facto update command in Gentoo and -bk allows for using binary packages (which is really great: it never hurts to have some binary packages laying around, you never know when you’ll need them – especially useful when downgrading).

# FEATURES="ccache" smart-live-rebuild -E -- -av1b

This is where the fancy stuff happen. Ebuilds from the kde overlay pull the source code straight from KDE’s Git and Subversion repositories, but have a fixed version (e.g. the ebuilds from the 4.9 branch all have a version of 4.9.49.9999). This is not really cool because their version remains the same all the time and the update command I mentioned above won’t update these ebuilds. You either have to manually re-emerge these packages (after manually monitoring the source code repositories for commits - zero coolness points) or install a tool to monitor the remote git repositories for commits since the last time you compiled and update the packages as necessary. So, you need smart-live-rebuild (available in Gentoo’s official repos).

It will record the commit from which you built last time and in the future won’t rebuild this package unless a newer commit has occured. Extra coolness points. But it wouldn’t be that cool if it wasn’t combined with the preceding FEATURES=”ccache” (there is an official tutorial about configuring ccache for your reading pleasure [4]). I have configured it with a 5GB cache over here but I only use it for live ebuilds which need frequent re-compiling. ccache will slow down the first compilation a bit but future compilations will be blazing fast, which is the key to maintaining a fully updated installation from a KDE branch.

Command line options explained:

-E is shorthand for --no-erraneous-merge. Simply don’t emerge packages for which the update has failed.

terminates the smart-live-rebuild options. Any remaining options will be directly passed on to emerge.

-av1b we have gone through this. -1 stands for --oneshot and it won’t append the packages emerged to your world file [3]. Since these packages are already installed, you only wish to update them. -1 is definitely not necessary here as smart-live-rebuild explicitly includes --oneshot on its own. I include it for symmetry with the next command.

On the other hand, smart-live-rebuild is not that smart. KDE’s infrastructure often contains multiple packages in a single repository and a change to one will make smart-live-rebuild rebuild all the packages from this repository. For instance a change to ksmserver (which is a package contained in the kde-workspace Git repo) will rebuild *all* packages residing in this repo. Still, better than nothing :)

# emerge -ca; emerge -Pa; revdep-rebuild -- -av1b

This is my post-update command. It checks for unneeded packages and removes them as necessary. -c is shorthand for --depclean. -P on the other hand stands for --prune and is of similar use to –depclean, but to be honest, it hasn’t occured to me yet that –prune detects an unneeded package that –depclean hasn’t already detected. I include it for Feng-Shui reasons (it’s a fast operation so no big deal).

revdep-rebuild is another must-have tool for any sensible Gentoo user; it detects broken references to libraries (i.e. when updating to newer versions of libraries) and rebuilds packages as needed. Its command line options are explained above. Make a habit of running revdep-rebuild after every update command in order to avoid system breakage!

# eclean-dist -dn; eclean-pkg; emaint -f all

Lastly, my system maintainance command. It removes old dist files (a.k.a. source code tarballs used for emerging) along with old binary packages. Default behavior is to keep the files from packages whose version still exists in the Portage tree (as newer versions roll out, old versions are removed from the Portage tree). This can take a hell of a time for dist files because it checks the entire Portage tree several times! -dn greatly reduces this time by only keeping installed versions. This is great for a rebuild but no good for downgrading. But since you have created binary packages for previous versions, you don’t have to recompile in order to downgrade! So, problem effectively solved ;)

emaint checks for all kinds of problems in your installation and tries to -fix them. Nifty small utility.

[1] Gentoo Overlays: Users’ Guide

[2] Gentoo KDE Guide

[3] /var/lib/portage/world

[4] Caching Compilation

GRUB2 Editor v0.5.8

GRUB2 Editor received a new minor release today, mainly to update the openSUSE Build Service packages. The changelog is indeed tiny:

v.0.5.8 (18/06/2012)
——————–
*ADDED: Install on the boot sector of a partition instead of the MBR.
*FIXED: Fix installation detection in Gentoo.
*FIXED: Fix translation loading.

Actually it’s a release straight from git, no planning had been made. The changes are minimum since v0.5.5.

Translations included in the release: ca cs da de el es et fr ga hu lt nb nl pl pt pt_BR ru sv uk zh_TW

As always you can download it from this page.

GRUB2/BURG Integration in KDE

No, you’re not having a déjà vu, this is my second post about GRUB2/BURG support for KDM which shows up in PlanetKDE. I am sorry to abuse the planet for publicity, but I have seen quite a bit of confusion as to how KDE should be configured in order to interface with GRUB2/BURG and I would like to get some things straight.

What follows is a step-by-step guide describing how to configure KDE in order to automatically select a GRUB2/BURG entry other than the default when rebooting:

Shutdown Dialog

If you are a GRUB2 user skip the BURG instructions and vice versa.

Step 1. Inform GRUB2/BURG that you plan to use this feature:

GRUB2 instructions:

Open /etc/default/grub in the text editor of your preference (as root) and add the following:

GRUB_DEFAULT=saved

In case a GRUB_DEFAULT option already exists, just set it to “saved”.

BURG instructions:

Edit /etc/default/burg instead. The variable name is the same (GRUB_DEFAULT).

Step 2. Update your GRUB2/BURG menu file:

GRUB2 instructions:

The menu configuration file must contain set default=”${saved_entry}”. You are highly discouraged to manually edit this file, so execute the following command (as root):

grub-mkconfig -o /boot/grub/grub.cfg

Please note that some distributions (i.e. Fedora, Gentoo, Mandriva, openSUSE) use a slightly different naming scheme for GRUB2. So you’d have to adjust the above to the following:

grub2-mkconfig -o /boot/grub2/grub.cfg

BURG instructions:

BURG users should execute the following command (as root):

burg-mkconfig -o /boot/burg/burg.cfg

Step 3. Inform KDE that you are using GRUB2/BURG:

GRUB2 instructions:

Select “Grub2″ as your bootloader under “System Settings > Login Screen > Shutdown > Boot manager”.

BURG instructions:

For KDE >= 4.8.0:

Select “Burg” as your bootloader under “System Settings > Login Screen > Shutdown > Boot manager”.

For KDE < 4.8.0:

Select “Grub2″ as your bootloader under “System Settings > Login Screen > Shutdown > Boot manager”.

Step 4. Create symlinks:

GRUB2 instructions:

This step only applies to Fedora/Gentoo/Mandriva/openSUSE (or any other distribution which uses /boot/grub2 instead of /boot/grub)! If this is not the case for you, skip it.

For KDE >= 4.8.3:

You don’t have to do anything! You may safely skip this step.

For KDE < 4.8.3:

Execute the following commands (as root):

mkdir /boot/grub
ln -s /boot/grub2/grub.cfg /boot/grub/grub.cfg
ln -s /sbin/grub2-reboot /sbin/grub-reboot

In the above commands replace /sbin/ with wherever the grub2-reboot binary is installed. You can easily determine this with “which grub2-reboot”.

BURG instructions:

For KDE >= 4.8.0:

You don’t have to do anything! You may safely skip this step.

For KDE < 4.8.0:

Execute the following commands (as root):

mkdir /boot/grub
ln -s /boot/burg/burg.cfg /boot/grub/grub.cfg
ln -s /sbin/burg-reboot /sbin/grub-reboot

Same rule as above applies here: if the burg-reboot binary is not located under /sbin/, locate it with “which burg-reboot”.

Step 5. Reboot!

That’s it, you’re done! Now either reboot or restart KDM (something like “/etc/init.d/kdm restart”) and you’re all set ;-)

As a side note, GRUB2 Editor (supports BURG) automatically adjusts your configuration to the above (i.e. it automatically performs steps 1 and 2).

Feel free to link to this post in linux forums as a reference. All of the above refer to KDE >= 4.7.

GRUB2 Editor v0.5.5

GRUB2 Editor received another bugfix release today. This is mostly a boring release: no fancy new features so you may as well skip reading.

Changelog:

v.0.5.5 (09/01/2012)

——————–

*ADDED: Experimental BURG support (only commands shared with GRUB2).
*ADDED: Option to disable Plymouth (Advanced -> Linux Kernel Arguments).
*FIXED: Fedora/openSUSE: the GRUB installation is now properly detected.
*FIXED: Encoding issue (UTF-8 encoding is now supported).
*FIXED: A QPackageKit backend bug where previous results were returned.

Please note that BURG “experimental” support is barely tested. The only testing I performed was whether it crashes or not.

Available Translations:

Catalan, Czech, Danish, German, Greek, Spanish, Estonian, French, Irish [Gaelic], Hungarian, Lithuanian, Dutch, Panjabi/Punjabi, Polish, Portuguese, Brazilian Portuguese, Russian, Swedish, Ukrainian, Traditional Chinese.

I would like to thank all KDE translators who were involved. Their work is very much appreciated!

Packages:

There are packages for most major distributions. Please take a look at the Downloads page.

GRUB2 Bootloader Editor v0.5.0

GRUB2 Bootloader Editor version 0.5.0 was just released. Highlights of this release are:

Easily recover GRUB2 (from a Live CD):

Installing(=Recovering) GRUB

Are you unable to boot your Linux installation because you accidentally installed Windows after Linux? No problem! Boot from a Live CD of your Linux distribution, install the GRUB2 Editor using one of the many provided packages and restore GRUB in just a couple of seconds. It’s really easy! (More info)

Disable memtest entries:

Disable memtest entries

You may now further customise your GRUB2 menu by selecting whether or not memtest entries will be generated.

Translations:

In this release you will also find translations for the following languages, thanks to the corresponding KDE translation teams:

  • Danish
  • German
  • Estonian
  • Hungarian
  • Dutch
  • Portuguese
  • Portuguese (Brazil)
  • Swedish
  • Ukrainian

Other features:

Other new features include:

  • Warn the user when trying to uninstall the current kernel.
  • Manually find missing configuration files which are needed.
  • Recover initial settings (undoes all changes performed by this tool).

I estimate that the next release will be in about 2 months from now, due to restraining schedule.

As always you may get binary packages for most major distributions (Arch, Fedora, Mandriva, openSUSE, Ubuntu) from our KDE-Apps.org page. Enjoy :-D

Changelog Quote:

v.0.5.0 (09/05/2011)
——————–
*ADDED: Recover GRUB2 (from a Live CD).
*ADDED: Option to toggle generation of memtest entries.
*ADDED: Manually find missing configuration files which are needed.
*ADDED: Restore default settings.
*FIXED: Warn the user when trying to uninstall the current kernel.

*I18N: Added 9 new translations: Danish, German, Estonian, Hungarian, Dutch, Portuguese, Portuguese (Brazil), Swedish, Ukrainian.

Recovering GRUB (The Easy Way)

It’s well past midnight over here so I’ll try to keep it short. I will present you a new feature of the GRUB2 Editor: recovering GRUB (from a Live CD). A new feature that hopefully many user-friendly distributions will adore because it bypasses the intimidating console when trying to recover GRUB, for example when installing Windows after your Linux distribution; an action that will erase GRUB from the MBR(=Master Boot Record) and render your Linux setup unbootable. I’ll let some screenshots do the talking:

Live CD Welcome Screen

Recover Screen

Installing(=Recovering) GRUB

Job Done!

So, firstly you are presented with two options when launching the GRUB Editor from a Live CD (screenshot #1); choosing the recovery option provides you with a list of detected partitions along with some information to help you determine where your previous installation resides (screenshot #2), picking a partition and Applying will initiate the installation process (screenshot #3) which will hopefully lead you to screenshot #4.

This post was published in hoping for feedback before making the next release (which will happen anytime soon). So any suggestions/insults are more than welcome :D

PS: Screenshots and testing done on a Kubuntu 10.10 Maverick Meerkat Live CD.