{Snamell(it)}

  • {Snamell(it)}
  • Blog
  • About
  • Location
  • Contact Us

Monthly Archives: July 2011

July 22, 2011 · pti

Disable authentication for global proxy settings on Ubuntu

2011-07-22 Fri 16:33
Figure out proxy settings Linux

Ubuntu has a Network Proxy chooser which allows you to select a location (a la MacOSX). This works well enough except that the UI is a bit counter-intuitive (in my humble opinion)which causes me to regularly nuke some predefined setting inadvertently. This is not a big deal though.

However for update manager (and several other tools) to pick up the new proxy settings you need to push the settings down to the system level. This takes 2 times typing your password. Now, this IS a big deal.

When I go back and forth between work and home I have to change this at least 2 times per day. Also it irks me that a detail setting like the proxy is not auto-detected and I need to login to change this ‘system’ setting. My laptop is essentially a single user system and I do not see switching the proxy as a serious security issue, even with 3 kids running around the home.

To come back to auto-detection, while this works fine at work, it fails to figure out that at home that there is a direct connection to the Internet. I can probably fix this by replacing my aging wireless router with my Time Capsule as the Internet gateway router, but I prefer to have the Time Capsule close to my desk.

In any case the Network proxy shows 2 times the authentication dialog box. A particularly nice feature (Is this new in Natty?) is that the dialog shows for which DBUS setting access is being asked.

The first dialog asks access to com.ubuntu.systemservice.setProxy. This response is configured in the file /usr/share/polkit-1/actions/com.ubuntu.systemservice.policy. This is a very readable XML file which contains a section for the setProxy action. I feel no reservation in allowing unchecked access to the setProxy. Although this might make a man-in-the-middle attack easier someone with the sophistication to pull this off, does not need to doctor my PC to do it.

<action id="com.ubuntu.systemservice.setproxy">
  <description>Set current global proxy</description>
  <message>System policy prevents setting proxy settings</message>
  <defaults>
    <!-- PTI : original settings
    <allow_inactive>no</allow_inactive>
    <allow_active>auth_admin_keep</allow_active>
    -->
    <allow_inactive>yes</allow_inactive>
    <allow_active>yes</allow_active>
  </defaults>
</action>

Retrying, and indeed one of the authentication requests dropped.

Note that the action was configured with auth_admin_keep which according to the docs would mean we should be authenticated for some time,so I would not expect the second authentication I am getting. Must be a subtlety which escapes me at the moment.

The second action is more problematic since the set-system om the system gconf settings is much less fine-grained than setProxy and can potentially cause more damage to the system.

<action id="org.gnome.gconf.defaults.set-system">
  <description gettext-domain="GConf2">Change GConf system values</description>
  <message gettext-domain="GConf2">Privileges are required to change GConf system values</message>
  <defaults>
    <allow_inactive>no</allow_inactive>
    <!-- PTI: Original setting
    <allow_active>auth_admin</allow_active>
    -->
    <allow_active>yes</allow_active>
  </defaults>
</action>

After relaxing this second method, I can finally easily switch proxies between Locations.

There are several things bugging me:

  1. The set-system method really is too wide in scope.
  2. There should be a more elegant way that modifying files under /usr/share
  3. My system should actually switch location unaided.

For the time being, I fixed a frustration and learned something in the process. The result is not yet fully satisfactory, but that will improve over time. **

  • Tutorial explaining editing static files
  • pklocalauthority man page
  • Install Updates as Unprivileged User

Of course there is a more elegant solution than editing files under /usr/share folder.

Everything is explained in the man pages pklocalauthority and PolicyKit.conf (among others).

But that’s for another day…

Posted in linux, Ubuntu | Tags: authentication, proxy, Ubuntu | Leave a comment |
July 20, 2011 · pti

Making dotFiles visible on the Mac

Dotfiles, can’t live with ’em, can’t live without ’em.

Dot files are the term for folders and files starting with a ‘.’ so they do not show upwhen using plain ls.

Tha Mac has a cool keycode to toggle visibility of dotfiles in the File Open/Save dialog, but this does not work in the finder for one reason or another.

In practice this meant I had to deal with dotFiles and dotDriectories I found on the net some incantation to force the setting for the finder to show/hide the dotfiles. Upon restarting the finder the widows will reopen with the updated setting.

I found on the net some snippets of sh script (but I forgot where and cannot immediately retrieve it), and I immediately dumped them in my ~/bin folder.

~/bin/hide-dotfiles :

#!/bin/sh
defaults write com.apple.finder AppleShowAllFiles -bool FALSE
killall Finder

~/bin/show-dotfile :

#!/bin/sh
defaults write com.apple.finder AppleShowAllFiles -bool TRUE
killall Finder

So in order to toggle the setting in the graphical finderyou drop to the commandline set the flag in the desired state and restart the finders. Works nice…

One remark : if you suddenly see strange icons on your desktop, then you probably still have the dotFiles visible selected. hide-dotfiles will clean your desktop up again.

*

Posted in MacOSX | Leave a comment |
July 14, 2011 · pti

Building debian packages in a cleanroom

Overview and Goals

We build our solutions mostly on Ubuntu Natty and we deploy to Debian (currently lenny). One problem we face is that Debian has a slow release cycle and the packages are dated. Before a new release is approved and deployed to our target servers it can still take many months causing us to have to use up to 3 year old technology.

So we are often faced to ‘backport’ packages or debianize existing packages if we want to use the current releases.

In the past we had different build servers for the target architectures. However this is a heavy solution and scales poorly. It also makes upgrading to the next release that much heavier.

So we need a system for building debian packages that is :

  1. Fully automated
  2. Target multiple distributions Debian (lenny,squeeze) and Ubuntu(natty, maverick)
  3. Build on development machines(a) and Jenkins/Hudson CI servers(b)
  4. easily configurable
  5. memorizable process

The goal is to make packages for internal consumption, and the process outlined here falls short of the community standards.

Enter pbuilder

Of course we are not the first or only one with this issue. In fact we are laggards and there are excellent articles on the ‘net to help us with these goals.

e.g.

  • PBuilder User Manual
  • Pbuilder Tricks on the Debian Wiki
  • PBuilder How To over at the Ubuntu Wiki

The pbuilder program create a clean room environment of a freshly installed empty debian or ubuntu distro, chroot into it and starts building based on the project metadata, mostly from the debian/control file.

It does this by unpacking a preconfigured base image of the selectable target , installing the build dependencies, building the package in the cleanroom, moving the artifacts to the hosting machine and cleaning everything up again. And it does this actually surprisingly fast. This clearly satisfies goals 1 and 2 (and half of 3 if we assume a developer has full control over his laptop).

The pbuilder is configured through commandline options, which are clear and friendly enough but you end up with commandlines of several lines long which are impossible to type in a shell and are a maintenance nightmare in build scripts (clearly conflicts with point 5). Also in the ideal world we would be able to retarget a build without touching the checked out files, e.g. with environment variable (see goals 3 and 4).

Configuring pbuilder

On the Pbuilders Tricks page I found a big smart shell script to use as the pbuilder configuration file ~/.pbuilderrc.

# Codenames for Debian suites according to their alias. Update these when
# needed.
UNSTABLE_CODENAME="sid"
TESTING_CODENAME="wheezy"
STABLE_CODENAME="squeeze"
OLDSTABLE_CODENAME="lenny"
STABLE_BACKPORTS_SUITE="$STABLE_CODENAME-backports"

# List of Debian suites.
DEBIAN_SUITES=($UNSTABLE_CODENAME $TESTING_CODENAME $STABLE_CODENAME $OLDSTABLE_CODENAME
    "unstable" "testing" "stable" "oldstable")

# List of Ubuntu suites. Update these when needed.
UBUNTU_SUITES=("natty" "maverick" "jaunty" "intrepid" "hardy" "gutsy")

# Mirrors to use. Update these to your preferred mirror.
DEBIAN_MIRROR="ftp.be.debian.org"
UBUNTU_MIRROR="mirrors.kernel.org"

# Optionally use the changelog of a package to determine the suite to use if
# none set.
if [ -z "${DIST}" ] && [ -r "debian/changelog" ]; then
    DIST=$(dpkg-parsechangelog | awk '/^Distribution: / {print $2}')
    # Use the unstable suite for Debian experimental packages.
    if [ "${DIST}" == "experimental" ]; then
        DIST="unstable"
    fi
fi

# Optionally set a default distribution if none is used. Note that you can set
# your own default (i.e. ${DIST:="unstable"}).
: ${DIST:="$(lsb_release --short --codename)"}

# Optionally set the architecture to the host architecture if none set. Note
# that you can set your own default (i.e. ${ARCH:="i386"}).
: ${ARCH:="$(dpkg --print-architecture)"}

NAME="$DIST"
if [ -n "${ARCH}" ]; then
    NAME="$NAME-$ARCH"
    DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}")
fi

BASETGZ="/var/cache/pbuilder/$NAME-base.tgz"
DISTRIBUTION="$DIST"
BUILDRESULT="/var/cache/pbuilder/$NAME/result/"
APTCACHE="/var/cache/pbuilder/$NAME/aptcache/"
BUILDPLACE="/var/cache/pbuilder/build/"

# make sure folders exist
mkdir -p $BUILDRESULT
mkdir -p $APTCACHE

echo "Target : $BUILDRESULT" >>/tmp/dist

if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then

    OTHERMIRROR="deb file:///var/cache/pbuilder/$NAME/result ./"
    BINDMOUNTS="/var/cache/pbuilder/$NAME/result"
    HOOKDIR="/var/cache/pbuilder/$NAME/hooks"
    EXTRAPACKAGES="apt-utils"
    # Debian configuration
    MIRRORSITE="http://$DEBIAN_MIRROR/debian/"
    COMPONENTS="main contrib non-free"
    DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS[@]}" "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg")
    if $(echo "$STABLE_CODENAME stable" | grep -q $DIST); then
        EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring"
        OTHERMIRROR="$OTHERMIRROR | deb http://www.backports.org/debian $STABLE_BACKPORTS_SUITE $COMPONENTS"
    fi
elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then
    # Ubuntu configuration
    MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/"
    COMPONENTS="main restricted universe multiverse"
    DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS[@]}" "--keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg")
else
    echo "Unknown distribution: $DIST"
    exit 1
fi

I just updated the distribution names to the current situation and added the directory where the packages are collected as a repository so subsequent builds can use these packages as dependencies. I also specified the keyrings to use for Debian and Ubuntu and made sure the expected folders are created to mount them in the clean room.

I created this in my account on my development laptop and added a symbolic link in ~root/.pbuilderrc to this file so I can update it from my desktop environment and do not have to get my brain all twisted up to try to remember with which configuration I am busy in my shell, sudo, su –, …

THe way the script works is that the configuration adapts itself to the content of the DIST and ARCH environment variables. So to configure lenny-amd64 as target is sufficient to do

~ > export DIST=lenny
~ > export ARCH=amd64

This approach is also perfect Jenkins or Hudson to determine the target build from checked out sources, since it can be specified in the build recipe. (satisfies goals 3b, 4 and 5)

Since we have to run these programs using sudo we must make sure the environment variables are passed by sudo. We can do this in the Defaults line of the /etc/sudoers file with the envkeep instruction.

...
Defaults  env_reset,env_keep="DIST ARCH http_proxy ftp_proxy 
 https_proxy no_proxy"
... snip ...
# Cmnd alias specification
Cmnd_Alias PBUILDER=/usr/sbin/pbuilder, /usr/bin/pdebuild
... snip to end of file ...
# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

pti     ALL=(ALL) NOPASSWD: PBUILDER
jenkins ALL=(ALL) NOPASSWD: PBUILDER
#includedir /etc/sudoers.d

You add the DIST and ARCH variables there. I also included the environment variables for proxying so I can easily switch between environment on my laptop and these changes propagate to sudo (which is also useful for plain apt-get, by the way).

I also added a line to show how to make the tools available for a user without having to give their password. This is not needed for interactive work, but very much so for the user as which the CI server is running (in our case jenkins). Note that the definition should be after the group definitions, otherwise these take precedence and jenkins has to provide his password (read: is hanging during the build).

Creating the target base images

The heavy lifting is now done. Let’s create an base.tgz for lenny-amd64.

~ > export DIST=lenny
~ > export ARCH=amd64
~ > sudo pbuilder create

Now go and have a cup of coffee (or read some emails).

Rinse and repeat for the other target platforms.

Backporting existing packages

In theory backporting would be as simple as

~  ᐅ cd tmp
~/tmp  ᐅ apt-get source mongodb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Need to get 1,316 kB of source archives.
Get:1 http://be.archive.ubuntu.com/ubuntu/ natty/universe mongodb 1:1.6.3-1ubuntu2 (dsc) [2,276 B]
Get:2 http://be.archive.ubuntu.com/ubuntu/ natty/universe mongodb 1:1.6.3-1ubuntu2 (tar) [1,285 kB]
Get:3 http://be.archive.ubuntu.com/ubuntu/ natty/universe mongodb 1:1.6.3-1ubuntu2 (diff) [29.0 kB]
Fetched 1,316 kB in 1s (679 kB/s)
gpgv: Signature made Thu 17 Mar 2011 11:49:37 PM CET using RSA key ID D5946E0F
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./mongodb_1.6.3-1ubuntu2.dsc
dpkg-source: info: extracting mongodb in mongodb-1.6.3
dpkg-source: info: unpacking mongodb_1.6.3.orig.tar.gz
dpkg-source: info: unpacking mongodb_1.6.3-1ubuntu2.debian.tar.gz
dpkg-source: info: applying debian-changes-1:1.6.3-1
dpkg-source: info: applying build-process-remove-rpath
dpkg-source: info: applying mozjs185
~/tmp  ᐅ DIST=lenny ARCH=amd64 sudo pbuilder build mongodb_1.6.3-1ubuntu2.dsc
I: using fakeroot in build.
I: Current time: Thu Jul 14 14:28:17 CEST 2011
I: pbuilder-time-stamp: 1310646497
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/lenny-amd64-base.tgz]
...

and you should get a nice set of debian packages in */var/cache/pbuilder/lenny-amd64.

In practice you will often end up with errors like :

... snip ...
The following packages have unmet dependencies:
  pbuilder-satisfydepends-dummy: Depends: xulrunner-dev (>= 2.0~) but it is not installable
The following actions will resolve these dependencies:

Remove the following packages:
pbuilder-satisfydepends-dummy

Score is -9850

Writing extended state information... Done
... snip ...
I: cleaning the build env
I: removing directory /var/cache/pbuilder/build//6279 and its subdirectories

In these case you have to walk the dependency tree till you find the leafs, and walk back up the branches to the trunk. Note also that chances are that unless you target machines which only serve a very specific purpose, you might end up with packages which are uninstallable since you pull out the rug from other installed packages. However we have the principle to use 1 virtual host to deliver 1 service, hence there are very little packages deployed to them and nothing complicated like desktop environments.

Simple leaf packages often build without a hitch:

~/tmp  ᐅ DIST=lenny sudo pbuilder build libevent_1.4.13-stable-1.dsc
I: using fakeroot in build.
I: Current time: Thu Jul 14 14:44:00 CEST 2011
I: pbuilder-time-stamp: 1310647440
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/lenny-amd64-base.tgz]
I: creating local configuration
I: copying local configuration
I: mounting /proc filesystem
I: mounting /dev/pts filesystem
I: Mounting /var/cache/pbuilder/ccache
... snip ...
dpkg-genchanges: including full source code in upload
dpkg-buildpackage: full upload (original source is included)
W: no hooks of type B found -- ignoring
I: Copying back the cached apt archive contents
I: unmounting /var/cache/pbuilder/lenny-amd64/result filesystem
I: unmounting /var/cache/pbuilder/ccache filesystem
I: unmounting dev/pts filesystem
I: unmounting proc filesystem
I: cleaning the build env
I: removing directory /var/cache/pbuilder/build//15214 and its subdirectories
I: Current time: Thu Jul 14 14:49:57 CEST 2011
I: pbuilder-time-stamp: 1310647797
~/tmp  ᐅ ls -al /var/cache/pbuilder/lenny-amd64/result
total 5260
drwxr-xr-x 2 root root    4096 2011-07-13 19:47 .
drwxr-xr-x 5 root root    4096 2011-07-13 19:13 ..
-rw-r--r-- 1 pti  pti     2853 2011-07-14 14:49 libevent_1.4.13-stable-1_amd64.changes
-rw-r--r-- 1 pti  pti     9129 2011-07-14 14:49 libevent_1.4.13-stable-1.diff.gz
-rw-r--r-- 1 pti  pti      907 2011-07-14 14:49 libevent_1.4.13-stable-1.dsc
-rw-r--r-- 1 pti  pti   499603 2009-12-05 23:04 libevent_1.4.13-stable.orig.tar.gz
-rw-r--r-- 1 pti  pti    61956 2011-07-14 14:49 libevent-1.4-2_1.4.13-stable-1_amd64.deb
-rw-r--r-- 1 pti  pti    31262 2011-07-14 14:49 libevent-core-1.4-2_1.4.13-stable-1_amd64.deb
-rw-r--r-- 1 pti  pti   172950 2011-07-14 14:49 libevent-dev_1.4.13-stable-1_amd64.deb
-rw-r--r-- 1 pti  pti    51588 2011-07-14 14:49 libevent-extra-1.4-2_1.4.13-stable-1_amd64.deb
-rw-r--r-- 1 root root    9051 2011-07-14 14:48 Packages
~/tmp  ᐅ

Using pdebuild for building packages

Many of our packages are debianized and can be build using debuild.

I use here the Ubuntu sources of tokyocabinet as an example (which uses the libevent package we just built, btw):

~/tmp/tokyocabinet-1.4.37  ᐅ DIST=lenny ARCH=amd64 pdebuild
...snip...
 dpkg-genchanges  >../tokyocabinet_1.4.37-6ubuntu1_amd64.changes
dpkg-genchanges: not including original source code in upload
dpkg-buildpackage: binary and diff upload (original source NOT included)
W: no hooks of type B found -- ignoring
I: Copying back the cached apt archive contents
I: unmounting /var/cache/pbuilder/lenny-amd64/result filesystem
I: unmounting /var/cache/pbuilder/ccache filesystem
I: unmounting dev/pts filesystem
I: unmounting proc filesystem
I: cleaning the build env
I: removing directory /var/cache/pbuilder/build//4199 and its subdirectories
I: Current time: Thu Jul 14 15:05:27 CEST 2011
I: pbuilder-time-stamp: 1310648727
~/tmp/tokyocabinet-1.4.37  ᐅ ls /var/cache/pbuilder/lenny-amd64/result
...snip...
tokyocabinet_1.4.37-6ubuntu1_amd64.changes
tokyocabinet_1.4.37-6ubuntu1.debian.tar.gz
tokyocabinet_1.4.37-6ubuntu1.dsc
tokyocabinet_1.4.37.orig.tar.gz
tokyocabinet-bin_1.4.37-6ubuntu1_amd64.deb
tokyocabinet-doc_1.4.37-6ubuntu1_all.deb

Sometimes the dependencies break on the version of debhelpers. This version is added conservatively by the dh* scripts and often is overly conservative. Many packages build just fine with older versions of the debhelpers. ** *

Setting up automated build

To set this up on the build server we have to replicate the steps above

  1. Create the ~/.pbuilderrc file
  2. Symbolic link to this file in ~root/.pbuilderrc
  3. Allow jenkins to use sudo for building packages
  4. Create a jenkins job to (re)build the packages
  5. Create jobs to build the packages

* *

Posted in build, linux, Ubuntu | Leave a comment |
July 4, 2011 · pti

Enable real idletime for Org Mode on Ubuntu

Org Mode can use the idle time to correct time tracking entries. On the Mac this works on the idle time of the computer, on other platforms it uses the idle time of emacs. Hence if you do a significant task in another program it will be idle for org-mode.

There is a little program delivered with org-mode sources to estimate the “real” idle time based on the information used by screensavers. Unfortunaltely it is in source code and needs to be compiled first. Everything is actually well documented, just scattered.

Let’s see where that little program is located

$ locate x11idle
/home/pti/.emacs.d/vendor/org-mode/UTILITIES/x11idle.c

This conveniently shows the location of the source file. If you might find a x11info file (without .c) in a bin directory then it is already installed and you’re done.

If you get no results and there is some remark that the database is too old then it is time to refresh it using sudo updatedb. Then the lookup should find the file.

Now go to the folder where the file is located. In my case :

$ cd ~/.emacs.d/vendor/org-mode/UTILITIES

We need to make sure we have the necessary X screensaver libraries for compiling :

$ sudo apt-get install lib-xss-de
sudo apt-get install libxss-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  x11proto-scrnsaver-dev
The following NEW packages will be installed:
  libxss-dev x11proto-scrnsaver-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 34.1 kB of archives.
After this operation, 225 kB of additional disk space will be used.
Do you want to continue [Y/n]?
Get:1 http://be.archive.ubuntu.com/ubuntu/ natty/main x11proto-scrnsaver-dev all 1.2.1-1 [21.3 kB]
Get:2 http://be.archive.ubuntu.com/ubuntu/ natty/main libxss-dev amd64 1:1.2.1-1 [12.9 kB]
Fetched 34.1 kB in 0s (200 kB/s)
Selecting previously deselected package x11proto-scrnsaver-dev.
(Reading database ... 567603 files and directories currently installed.)
Unpacking x11proto-scrnsaver-dev (from .../x11proto-scrnsaver-dev_1.2.1-1_all.deb) ...
Selecting previously deselected package libxss-dev.
Unpacking libxss-dev (from .../libxss-dev_1%3a1.2.1-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up x11proto-scrnsaver-dev (1.2.1-1) ...
Setting up libxss-dev (1:1.2.1-1) ...

Now we can compile x11idle. We need to specify to link against the X11 and X Screensaver libraries (Xss). I immediately place the resulting executable file in ~/bin/x11idle. Alternatively you could use the /usr/local/bin folder, but first compile to a temporary location like /tmp and move it there : sudo mv /tmp/x11idle /usr/local/bin.

$ gcc -o ~/bin/x11idle -lX11 -lXss x11idle.c
x11idle.c: In function ‘main’:
x11idle.c:19:5: warning: format ‘%u’ expects type ‘unsigned int’, but argument 2 has type ‘long unsigned int’
$ x11idle
22
$ x11idle
20

I tried to run it using the normal path and got 22 and 20 seconds idle time : it works !!!

The orgmode code will now find and use it.

For more background info see

  • Blogpost about getting idle time in linux.
  • The message with the patch proposal for including it with org-mode

2011-07-04 Mon 11:11
Track down reason for Dangling Time References

Posted in Uncategorized | Leave a comment |

Pages

  • {Snamell(it)}
  • Blog
  • About
  • Location
  • Contact Us

Archives

  • July 2013
  • March 2012
  • July 2011
  • June 2011
  • November 2008
  • September 2008
  • August 2008
  • July 2008
  • June 2008

Categories

  • build (2)
  • Java Stuff (6)
  • linux (5)
  • MacOSX (1)
  • Personal (1)
    • Huiswerk (1)
    • Kinderen (1)
  • Plone (1)
  • Project Management (2)
  • Ubuntu (6)
  • Uncategorized (3)

WordPress

  • Log in
  • WordPress

Subscribe

  • Entries (RSS)
  • Comments (RSS)
© My Website