elsdoerfer.name

I love logcheck, but I consistently find I need to use a number of custom rules to keep the noise down. Managing those rules in a central source control repository makes sense if you need to maintain multiple servers, but logcheck makes this difficult by requiring your rule files to be placed in the same directories as the builtin default rules.

To work around this issue I have written a small Python script that will symlink the rule files from your repository into the appropriate logcheck directories. The git repository includes the custom rules I am personally currently using, but you can of course easily replace those with your own.

While setting up a new server recently, I spent a bit of time doing some basic benchmarks, and I thought it couldn't hurt just putting them out there. It's really just the raw data I collected, without analysis or commentary. Also, at this point, if a piece of information isn't listed here, I likely don't have it any more.

The hardware is a Intel Xeon L5410A 4x 2.44 Ghz with 16 GB Ram (unless otherwise noted). The ext3 filesystem was used for all tests.

Ubuntu Hardy, 8GB RAM

time git clone [linux-kernel]

2075 KiB/s

real 5m43.632s
user 2m49.540s
sys 0m7.720s

time tar -czf linux.tar.gz linux-git/

real 0m38.317s
user 0m37.460s
sys 0m1.580s

Ubuntu Jaunty, 8GB RAM, on LVM

time git clone [linux-kernel]

2072 KiB/s

real 5m39.452s
user 2m49.040s
sys 0m7.660s

time tar -czf linux.tar.gz linux-git/

real 0m37.786s
user 0m36.140s
sys 0m1.650s

(repeat)

time git clone [linux-kernel]

2061 KiB/s

real      5m36.951s
user     2m48.430s
sys       0m7.980s

time tar -czf linux.tar.gz linux-git/

real       0m37.812s
user      0m36.100s
sys        0m1.700s

bonnie++

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer      32G 47133  73 50981  13 25721   4 39083  57 54106   5 368.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 34523  60 494559  99  1708   2 14833  23 +++++ +++  1132   2
elsdoerfer,32G,47133,73,50981,13,25721,4,39083,57,54106,5,368.9,1,256,34523,60,494559,99,1708,2,14833,23,+++++,+++,1132,2

iperf

939 MBit/s

(repeat)

iperf

931 MBit/s

./nbench

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97) 

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          1108.5  :      28.43  :       9.34
STRING SORT         :          246.16  :     109.99  :      17.02
BITFIELD            :      4.5177e+08  :      77.49  :      16.19
FP EMULATION        :          241.92  :     116.08  :      26.79
FOURIER             :           25098  :      28.54  :      16.03
ASSIGNMENT          :           33.36  :     126.94  :      32.93
IDEA                :          6721.3  :     102.80  :      30.52
HUFFMAN             :          2206.8  :      61.19  :      19.54
NEURAL NET          :            49.6  :      79.68  :      33.52
LU DECOMPOSITION    :          1563.5  :      81.00  :      58.49
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 80.788
FLOATING-POINT INDEX: 56.897
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 4 CPU GenuineIntel Intel(R) Xeon(R) CPU           L5410  @ 2.33GHz 2333MHz
L2 Cache            : 6144 KB
OS                  : Linux 2.6.28-11-server
C compiler          : gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4)
libc                : libc-2.9.so
MEMORY INDEX        : 20.857
INTEGER INDEX       : 19.652
FLOATING-POINT INDEX: 31.557
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Ubuntu Hardy, 512MB RAM, on LVM (read-ahead 8192)

bonnie++ -s 1g -n 2

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer       1G 54938  82 59756  15 38555   8 68071  92 119171  11 674.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  2 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
elsdoerfer,1G,54938,82,59756,15,38555,8,68071,92,119171,11,674.0,1,2,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

bonnie++ -s 1g -n 256

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer       1G 57688  87 60005  15 36878   7 70054  94 116765  11 697.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 39843  66 446376 100  1778   2 14549  22 +++++ +++  1130   2
elsdoerfer,1G,57688,87,60005,15,36878,7,70054,94,116765,11,697.2,1,256,39843,66,446376,100,1778,2,14549,22,+++++,+++,1130,2

Ubuntu Hardy, 512MB RAM, on LVM (read-ahead 8192)

bonnie++ -s 1g -n 2

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer       1G 61444  93 60359  16 27047   5 38587  58 63769   6 674.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  2 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
elsdoerfer,1G,61444,93,60359,16,27047,5,38587,58,63769,6,674.7,1,2,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

bonnie++ -s 1g -n 256

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer       1G 50989  95 61399  15 26720   5 32579  59 63642   6 703.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 39059  66 439526 100  1763   2 17369  27 +++++ +++  1203   2
elsdoerfer,1G,50989,95,61399,15,26720,5,32579,59,63642,6,703.4,1,256,39059,66,439526,100,1763,2,17369,27,+++++,+++,1203,2

Ubuntu Hardy, 512MB RAM, on LVM (read-ahead 16384)

bonnie++

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer       1G 55044  90 60922  16 36693   8 47231  75 124815  15 685.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  4 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
elsdoerfer,1G,55044,90,60922,16,36693,8,47231,75,124815,15,685.4,1,4,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Ubuntu Hardy, 512MB RAM, on LVM (read-ahead 4096)

bonnie++ -s 1g -n 4

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
elsdoerfer       1G 58692  90 63051  17 31598   6 62978  88 92682   9 691.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  4 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
elsdoerfer,1G,58692,90,63051,17,31598,6,62978,88,92682,9,691.7,1,4,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,++

Ubuntu Hardy, KVM VM (non-virtio, 2GB RAM)

time git clone [linux-kernel]

2058 KiB/s

real 5m52.794s
user 2m50.330s
sys  0m17.260s

Ubuntu Hardy, KVM VM (virtio, 2GB RAM)

time git clone [linux-kernel]

2059 KiB/s

real 5m52.455s
user 2m50.430s
sys 0m14.470s

time tar -czf linux.tar.gz linux-git/

real   0m40.305s
user  0m37.910s
sys    0m2.080s

iperf

850 Mbit/s

Ubuntu Hardy, KVM VM (virtio disk, pcnet network, 2GB RAM)

iperf

73 Mbit/s

Ubuntu Hardy, KVM VM (virtio, 8GB RAM, 1CPU, 35GB disk)

bonnie++ -s 32g -n 256

Version 1.03b       ------Sequential Output------ --Sequential Input- --Random-
                               -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm1             32G 30618  50 30184   5 21731   4 35322  56 40786   5 280.2   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 41823  50 +++++ +++ 11834  11 44915  53 +++++ +++  4866   5
vm1  32G  30618,50    30184,5          21731,4    35322,56    40786,5    280.2,2    256 41823,50,+++++,+++,11834,11,44915,53,+++++,+++,4866,5

(repeat)

bonnie++ -s 4g -n 16

Version 1.03b       ------Sequential Output------ --Sequential Input- --Random-
                        -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
vm1              4G 33773  56 48037   8 45177  16 59915  96 349013  85 12613 100
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
vm1,4G,33773,56,48037,8,45177,16,59915,96,349013,85,12612.9,100,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

./nbench

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          1139.8  :      29.23  :       9.60
STRING SORT         :          231.36  :     103.38  :      16.00
BITFIELD            :      4.3024e+08  :      73.80  :      15.42
FP EMULATION        :             248  :     119.00  :      27.46
FOURIER             :           24781  :      28.18  :      15.83
ASSIGNMENT          :          32.415  :     123.35  :      31.99
IDEA                :          5663.5  :      86.62  :      25.72
HUFFMAN             :          2433.2  :      67.47  :      21.55
NEURAL NET          :           49.58  :      79.65  :      33.50
LU DECOMPOSITION    :            1559  :      80.77  :      58.32
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 78.957
FLOATING-POINT INDEX: 56.595
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : GenuineIntel QEMU Virtual CPU version 0.9.1 2332MHz
L2 Cache            : 2048 KB
OS                  : Linux 2.6.24-23-server
C compiler          : gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu3)
libc                : libc-2.7.so
MEMORY INDEX        : 19.909
INTEGER INDEX       : 19.550
FLOATING-POINT INDEX: 31.390
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gc

Ubuntu Hardy, OpenVZ VE (1GB disk, 256 RAM) inside KVM VM (virtio, 8GB RAM, 1CPU, 35GB disk)

./nbench

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          1142.3  :      29.29  :       9.62
STRING SORT         :          231.68  :     103.52  :      16.02
BITFIELD            :      4.3035e+08  :      73.82  :      15.42
FP EMULATION        :          247.92  :     118.96  :      27.45
FOURIER             :           24853  :      28.27  :      15.88
ASSIGNMENT          :          32.414  :     123.34  :      31.99
IDEA                :          5665.2  :      86.65  :      25.73
HUFFMAN             :          2436.1  :      67.55  :      21.57
NEURAL NET          :
CPU:NNET--error in opening file!

Debian Lenny, 16GB RAM, on LVM (read-align 256)

iperf

921 Mbits/sec

bonnie++ -u michael -s 1g -n 8

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 59378  88 66050  14 24926   4 59592  77 69149   6 719.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  8 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
darjeeling,1G,59378,88,66050,14,24926,4,59592,77,69149,6,719.3,1,8,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

bonnie++ -u michael -s 1g -n 256

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 59848  89 65199  14 24752   4 61072  80 69264   5 727.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 42443  60 378919 100  1999   2 28995  40 468931 100  1370   2
darjeeling,1G,59848,89,65199,14,24752,4,61072,80,69264,5,727.4,1,256,42443,60,378919,100,1999,2,28995,40,468931,100,1370,2

./nbnench

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          1099.4  :      28.19  :       9.26
STRING SORT         :          247.12  :     110.42  :      17.09
BITFIELD            :      4.5086e+08  :      77.34  :      16.15
FP EMULATION        :          242.64  :     116.43  :      26.87
FOURIER             :           24699  :      28.09  :      15.78
ASSIGNMENT          :          33.014  :     125.62  :      32.58
IDEA                :          6714.6  :     102.70  :      30.49
HUFFMAN             :          2174.8  :      60.31  :      19.26
NEURAL NET          :           49.02  :      78.75  :      33.12
LU DECOMPOSITION    :          1559.9  :      80.81  :      58.35
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 80.450
FLOATING-POINT INDEX: 56.330
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : 4 CPU GenuineIntel Intel(R) Xeon(R) CPU           L5410  @ 2.33GHz 2333MHz
L2 Cache            : 6144 KB
OS                  : Linux 2.6.26-2-openvz-amd64
C compiler          : gcc version 4.3.2 (Debian 4.3.2-1.1)
libc                : libc-2.7.so
MEMORY INDEX        : 20.798
INTEGER INDEX       : 19.550
FLOATING-POINT INDEX: 31.243
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

Debian Lenny, 512MB RAM, on LVM (read-align 8192)

bonnie++ -u michael -s 1g -n 256

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 60006  89 65137  14 29067   5 64637  84 78755   6 697.6   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 43515  63 379181 100  2018   3 26261  38 469528 100  1234   1
darjeeling,1G,60006,89,65137,14,29067,5,64637,84,78755,6,697.6,1,256,43515,63,379181,100,2018,3,26261,38,469528,100,1234,1

Debian Lenny, 512MB RAM, disk direct (read-align 256)

bonnie++ -u michael -s 1g -n 256

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 42348  63 45139   9 18894   2 42236  55 46424   4 639.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 38219  54 378907 100  1901   2  8710  12 470516 100  1182   1
darjeeling,1G,42348,63,45139,9,18894,2,42236,55,46424,4,639.7,1,256,38219,54,378907,100,1901,2,8710,12,470516,100,1182,1

Debian Lenny, 512MB RAM, disk direct (read-align 8192)

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 32064  48 45588   9 19811   3 45751  60 51435   4 647.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 43643  62 378565  99  1858   2  8396  11 467667  99  1124   1
darjeeling,1G,32064,48,45588,9,19811,3,45751,60,51435,4,647.7,1,256,43643,62,378565,99,1858,2,8396,11,467667,99,1124,1

(repeat)

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 31010  51 44288   9 20165   4 46325  62 53544   4 626.5   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 44501  62 378833 100  1837   2  8413  11 468219 100  1158   1
darjeeling,1G,31010,51,44288,9,20165,4,46325,62,53544,4,626.5,1,256,44501,62,378833,100,1837,2,8413,11,468219,100,1158,1

(repeat)

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
darjeeling       1G 31876  49 45912   9 21013   3 42870  56 42369   3 653.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                256 41862  59 377359 100  1818   2  9454  13 469059  99  1179   1
darjeeling,1G,31876,49,45912,9,21013,3,42870,56,42369,3,653.1,1,256,41862,59,377359,100,1818,2,9454,13,469059,99,1179,1

Having recently written my first Widget for Android, here are some of the things I learned in the process.

Use a thread, not just a service

In the blog post introducing the AppWidget framework, you are encouraged to use a service to perform your widget updates if you are doing anything that might take a little longer, in order to avoid Application Not Responding (ANR) timeouts. However, this will usually not be enough. Since your service callbacks also run in your application's main thread, you'll still be in danger of triggering ANR as soon as your process is asked to do other things while still being busy with your first widget update. For example, your widget's broadcast receiver might be triggered again for another instance of your widget, or the user might start interacting.

The solution is to have your service start a separate thread. For an example, see Jeffrey Sharkey's android-sky Widget.

Stop your service when you're done

When your done with updating (if your using a thread, it'll be when your thread has finished), make sure you stop your service. Nobody seems to be doing this, including the example code in the official blog post, nor sample projects by Android team members. As a result, so many widgets have their process run in the background all the time, not allowing Android to kill it to reclaim memory. Considering that RAM isn't exactly a plentiful resource on devices like the G1, I would have thought this were especially important. Unless your updating multiple times an hour, I would strongly suggest that stopping your service and letting Android free up your process when necessary is the right thing to do.

Try to work around the bugs in Android 1.5 and the Launcher's Widget implementation

It turns out there are quite a few of them, and they all seem to revolve around issues with deleting a widget. In particular:

• AppWidgetProvider fails to handle delete intents correctly. As a result, if you are subclassing it, your onDelete() method will not be called. The solution is to override onReceive() and handling delete intents yourself. Here's a code snippet.
• In any case, if a user deletes an instance of your widget, the 1.5 default home screen will still keep sending update requests for the already deleted widget's id, until the user adds a new instance of your widget (apparently, some internal bundle of widget ids is not reset on delete). This seems like this could result in a considerable waste of resources, or even impact your functionality, with you sending out widget updates that nobody is going to see. Note: I'm not sure if those obsolete widget ids are cleared out on reboot (probably). You might also not have this problem if you're using custom scheduling instead of the default update facility.
  The best one can do here appears to be keeping an internal list of deleted widgets (widgets for which you - hopefully - received a delete intent), and then just skipping those updates.
• Finally, there seems to be another issue with phantom widgets when a configure activity is used and the user cancels it. In this case also, you'll continue to receive updates for the widget (which won't be shown anywhere), except it seems even worse, because apparently this time around, widget will really exist internally, rather than just some internal state not being update correctly (so even a reboot won't help here).
  The work around seems similar: keep information around that indicates whether a widget as been configured successfully, and don't bother updating widgets that haven't.

Android's layout editor  for Eclipse is much improved in the 1.5 SDK vs. the old 1.0/1.1 versions. Unfortunately, I didn't realize that until yesterday, because the new controls didn't show up for me. Instead, Eclipse just said:

Eclipse is loading framework information and the Layout library from the SDK folder.  file.xml will refresh automatically once the process is finished.

Of course, the process never finished. Because I never had any use for the old version, at first I didn't even know something was wrong.

There seem to be a handful of people out there with the same problem, although none of the suggested solutions worked for me. Until I found this Spanish site. It turns I should have uninstalled the old Eclipse plugin before installing the 1.5 SDK one. In my plugin directory (~/.eclipse/plugins), there were still .jar files from the old version around (named 0.8, apparently). Simple removing those files and restarting Eclipse did the trick for me.

A while ago, I set up Apps2SD on my ADP, and while I am not sure what, I must have done something wrong, because afterwards, Android refused to boot properly, claiming that the file system permissions of pretty much all installed packages didn't match up:

Package x.y.z has mismatched uid: 10089 on disk, 10079 in settings

Here's the Python script I used to apply the file system permissions that Android expected. It iterates over the packages defined in your devices /data/system/packages.xml. Use at your own risk.

"""Parse Android's /data/system/packages.xml file and spits out
shell code to fix UIDs.

This helps you fix "Package x.y.z has mismatched uid: 10089 on disk, 10079
in settings" errors.
"""

from xml.dom import minidom

xmldoc = minidom.parse('packages.xml')

packages = xmldoc.getElementsByTagName('package')
ignored = []
for package in packages:
    try:
        userId = package.attributes['userId'].value
        is_shared = False
    except KeyError:
        userId = package.attributes['sharedUserId'].value
        is_shared = True

    # do not touch permissions of shared apks (they userid always seems to be 1000)
    if not is_shared:
        print "busybox chown %s:%s %s" % (userId, userId, package.attributes['codePath'].value) 

    for subdir in ('', 'databases', 'shared_prefs'):  # note we don't touch lib/
        print "busybox chown %s %s:%s /data/data/%s/%s" % (
            '-R' if subdir else '', userId, userId, package.attributes['name'].value, subdir)

While upgrading my G1 to the new 1.5 JF image, the system wouldn't come back up after applying the update.zip file (yes, it's normal that the first reboot takes a while, but this was too long).

adb logcat was usuable at that point, and kept looping messages like:

D/AndroidRuntime(   66): >>>>>>>>>>>>>> AndroidRuntime START <<<<<<<<<<<<<<
D/AndroidRuntime(   66): CheckJNI is OFF
D/dalvikvm(   66): DexOpt: incorrect opt magic number (0xff ff ff ff)
D/dalvikvm(   66): Stale deps in cache file; removing and retrying
W/dalvikvm(   66): partial write in inflate (8152 vs 32768)
E/dalvikvm(   66): Unable to extract+optimize DEX from '/system/framework/framework.jar'
D/dalvikvm(   66): Failed on '/system/framework/framework.jar' (boot=1)
D/dalvikvm(   66): VM cleaning up
D/dalvikvm(   66): LinearAlloc 0x0 used 4100 of 4194304 (0%)
W/dalvikvm(   66): JNI_CreateJavaVM failed
E/AndroidRuntime(   66): JNI_CreateJavaVM failed

Turns out that the 8MB free space I had on my data partition were not enough for the update. Removing a couple larger apps through the shell let the boot process continue (though in the end, I used a backup to start a new, "clean" update).  All in all, the update needed an additional 22 MB on my phone. I'm not sure if this is JF specific and the actual 1.5 image is smaller, but, if you run into trouble, this might be the reason.

Now that the patent application has gone through, I'd like to see every text editor and IDE implement this feature:

For the first five who do, I'm going to waive any and all claims to royalties. So what is everybody waiting for?

Seriously though, if anybody is aware of an editor that supports this, I'd love to know.

When designing a layout in XML, Android supports a number of different metrics. From your Java code though, methods like setPadding or setTextSize only take absolute pixel values. It took me a while to find a solution, but apparently the (slightly too cumbersome, I feel) way to to this is through defining a Dimension Resource.

For example, create a resource file in ./res/values/dimens.xml and add this item:

<dimen name=table_padding>10dp</dimen>

Then, from within your activity you would do:

int tablePadding = getResources().getDimensionPixelSize(R.dimen.table_padding);

I wonder whether the conversion code used by getDimension/getDimensionPixelSize is exposed somewhere.

As I mentioned on Twitter a couple days ago (Yes, I've finally surrendered), I was surprised to find that Python's urllib/urllib2 refused to open the unicode url I gave it. Then I realized I didn't actually understand how precisily the non-ascii url stuff even worked, so I decided to change that.

Apparently, a URI is by definition restricted to (a subset of) ASCII characters (or maybe it consists of just bytes and does not have a concept of characters altogether, I couldn't quite make out the official stance). To enable a wider set of characters, IRIs where introduced in RFC 3987. IRIs by definition can contain unicode characters, and the RFC describes how an IRI has to be converted to an equivalent ASCII-only URI.

Therefore, to open a IRI (i.e. a unicode address,) in urllib, we first have to go through this conversion process. Essentially, two things need to be done:

• The domain name needs to be IDNA-encoded, also known as Punycode. Python, since 2.3, supports both an idna and a punycode codec. The latter is the base algorithm, the former knows about domain syntax and makes sure each label (i.e. subdomain) is handled separatly, as it should.
• The path and querystring components need to be UTF8-quoted, i.e. need to use the percent-encoding, with each octet being considered UTF8-encoded. Firefox also encodes a username/password portion in the same way.

With that in mind, the whole thing already seemed a lot more straightforward. I had a couple additional requirements, though:

• The function doing the  IRI => URI conversion should support being generically plugged into an urlopen() call, i.e. since urlopen() doesn't actually require a url, but also handles for example filesystem paths, the converter needed to be able to deal with those non-urls as well, without corrupting them.
• It needed to be able to handle URLs from "out in the wild", some of which may already be quoted (and should therefore not be quoted again).

The result currently looks like this:

def asciify_url(url, force_quote=False):
    r"""Attempts to make a unicode url usuable with ``urllib/urllib2``.

    More specifically, it attempts to convert the unicode object ``url``,
    which is meant to represent a IRI, to an unicode object that,
    containing only ASCII characters, is a valid URI. This involves:

        * IDNA/Puny-encoding the domain name.
        * UTF8-quoting the path and querystring parts.

    See also RFC 3987.
    """
    assert type(url) == unicode

    parts = urlparse.urlsplit(url)
    if not parts.scheme or not parts.netloc:
        # apparently not an url
        return url

    # idna-encode domain
    hostname = parts.hostname.encode('idna')

    # UTF8-quote the other parts. We check each part individually if
    # if needs to be quoted - that should catch some additional user
    # errors, say for example an umlaut in the username even though
    # the path *is* already quoted.
    def quote(s, safe):
        s = s or ''
        # Triggers on non-ascii characters - another option would be:
        #     urllib.quote(s.replace('%', '')) != s.replace('%', '')
        # which would trigger on all %-characters, e.g. "&".
        if s.encode('ascii', 'replace') != s or force_quote:
            return urllib.quote(s.encode('utf8'), safe=safe)
        return s
    username = quote(parts.username, '')
    password = quote(parts.password, safe='')
    path = quote(parts.path, safe='/')
    query = quote(parts.query, safe='&=')

    # put everything back together
    netloc = hostname
    if username or password:
        netloc = '@' + netloc
        if password:
            netloc = ':' + password + netloc
        netloc = username + netloc
    if parts.port:
        netloc += ':' + str(parts.port)
    return urlparse.urlunsplit([
        parts.scheme, netloc, path, query, parts.fragment])

A version with more extensive comments and doctests is part of the FeedPlatform code.

I subsequently found out that there is at the very least one other existing implementation of this, in httplib2. While that one doesn't avoid double-quoting and won't leave non-urls alone (my own specific requirements), the latter enables it to support partial urls. It also uses a custom quote function written after the spec, rather than relying on urllib.quote, which is interesting. I wonder what the practical differences are there. Finally, it has a bug where an auth-portion in the domain will lead to invalid IDNA-encoding, but that should be rare anyway.

It might further be noteworthy that the SVN version of FeedParser also applies IDNA-encoding, but does so on the full string given, which corrupts the URL it if there are non-ascii characters in any non-domain part.

Getting Xapian installed on a fresh Debian lenny/testing machine was a bit tougher than usual. I kept getting this message while trying to compile the Python bindings:

..._xapian.so: undefined symbol '_ZTIN6Xapian13PostingSourceE'

It turns out that lenny ships with a new version of aptitude, one which uses Xapian for search (which is actually pretty cool news). As a result, an older version of Xapian is already installed in /usr/lib and for some reason, the self-compiled version in /usr/local/lib was not being used. In the end, it was apparently a simple cache issue (make sense - you'd assume that the local/-version would take precedence by default, after all), and simply running ldconfig once made it work.

I wasted a couple of hours trying to solve this. On the plus side, I also learned some new things about Linux, so I guess it's allright