Building Android projects using Python 0

I’ve hated Ant since the first time I laid eyes on it. So I’m glad I finally took the time to understand the Android build process in detail, and sat down to reproduce it in Python.

A while ago, I wrote about how to build multiple versions of an Android app (like a free and a donate version). That was a long post. Compare it to this:

project = AndroidProject('AndroidManifest.xml', sdk_dir='/opt/android')
for version in ('com.foo.app.free, 'com.foo.app.donate'):
    apk = project.build("%s.apk" % version, package_name=version)
    apk.sign('keystore', 'alias', 'name')
    apk.align()

Using the library, you can write your build scripts in plain Python, or with a Python-based build tool (personally, I’m using fabric). See for example the buildfile for Android-Autostarts.

The Github page has more usage samples.

Jailbreaking AppleTV 2 / iOS 4.2.1 0

I finally managed to make this work and install XBMC. Oh what a frustrating process that was. I must have run into every single one of the various problems people have reported. Error 21, Error 1600, Error 1602, no SSH after jailbreak, no network after jailbreak. Trying with both Seas0nPass and PwnageTool/tetheredboot. Endless restores and switching between HDMI and Micro-USB cable. At one point, I had SSH, but the XBMC apt install seemingly crashed the device and it wouldn’t reboot.

The forums are full with all kinds of variations on the default instructions that people say worked for them, but it all seems mostly random. I have no clear answers either, but here are some things that appeared to make a difference for me:

  • It’s hard to tell what’s going on with the device and what mode it currently is in, but I had the most success when using the interactive instructions included in PwnageTool to enter DFU mode. I think it’s because without those, I tended to press MENU/PLAY for less then the 7 seconds recommended, and PwnageTool forced me to do it right. It seems like those 7 seconds might actually be close to the minimum required, and when in doubt, you should err on the side of holding the buttons for longer than shorter. What’s confusing is that it seems the AppleTV can appear in iTunes even before the 7 seconds are over, which might make you release the buttons too early.
  • The restore process never worked for me while the power cable was plugged in.
  • Most other problems just mysteriously disappeared after a while after restoring the original firmware and attempting the jailbreak again.
  • The no-networking problem went away after switching from a firmware built with PwnageTool to one built with Seas0nPass. However, I previously had the same PwnageTool firmware running with a working network as well, and the internet is full of reports of the problem appearing after reboots or waking from sleep, so I can probably look forward to the issue appearing again as well.

Convert VirtualBox machines to VMWare Workstation 0

  1. In VirtualBox, use File -> Export Appliance
  2. Fix the .ovf file.
  3. Use ovftool to convert: ovftool –diskMode=monolithicFlat export.ovf new-machine.vmx

On Games 0

Most big games today use story and character to explore themes we have explored before in other media that – simply put – use story and character to better effect. Demon’s Souls, Left 4 Dead and Captain Forever all explore the way we experience and perceive our world as individuals and collectively through our culture. And they do so using the uniquely meaningful properties of interactive games.
These are games that have something to say that can’t be said in other media.
These are the games that are pushing human culture forward.

Clint Hocking, Edge 221 (December 2010), p118

Python argparse: Combine nargs=* with subparsers 0

Say you set up argparse with a nargs=’*’ or nargs=’+’ argument first, followed by a subparser to handle a command:

parser = argparse.ArgumentParser()
parser.add_argument('--items', nargs='+', default=[])
subparsers = parser.add_subparsers()
subparser.add_parser('foo')
subparser.add_parser('bar')

The usage would look like this:

usage: script.py [--items ITEM [ITEM ...]] {foo,bar}

This is actually somewhat problematic. If you were to parse the arguments “–items one two foo”, argparse will assume that foo is an item, and complain about the lack of a command (error: too few arguments).

A workaround is letting the user break out of the nargs-based argument by given a single “-” character. This can easily be done with:

parser.add_argument('-', dest='__dummy',
    action="store_true", help=argparse.SUPPRESS)

Now, the following will work: “–items one two – foo”.

I think that two dashes (“–”) are more common for this purpose (with a single dash usually referring to stdin/stdout), but unfortunately, argparse doesn’t seem to support using two.

Edit: I’m a dummy. argparse already has support for “–” to break nargs built in. It’s not 100% the same, as it will force everything that follows to be considered a positional argument (whereas argparse in theory would support multiple sets of positional arguments, I think), but for most cases, relying on the builtin “–” is the right choice.

Reconstruct VM if VirtualBox.xml, $vm.xml is lost 0

If you are using snapshots, this isn’t an easy feat. You need to reconstruct the snapshot tree, with only a bunch of UUIDs to go on. Fortunately, each (differential) VDI file stores the UUID of it’s parent VDI. The following Python script parses a set of VDI files and tries to rebuild the tree:

http://github.com/miracle2k/linuxutils/blob/master/reconstruct-vdi-tree.py

It can output the XML code that goes into VirtualBox.xml directly. If your VM’s configuration file is also gone, the easiest way to go about seems to be:

  • Create a new machine with your base VDI.
  • Create snapshots matching the tree given by this script.
  • Edit the VM-specific XML file, and VirtualBox.xml, changing the UUIDs of the dummy snapshots with the UUIDs given to you by this script.

Merkur Bank HBCI in Hibiscus 0

Als Benutzerkennung muss der ursprüngliche numerische Online-Key verwenden werden, nicht der selbst konfigurierte Alias. Die HBCI-Version muss explicit auf FinTS 3.0 gestellt werden, via automatischer Erkennung scheint es nicht zu funktionieren. Die HBCI URL ist hbci11.fiducia.de/cgi-bin/hbciservlet.

Booting Ubuntu Karmic and Windows 7 from a GPT partition using BIOS 4

Those are some notes taken while trying to get this setup running. I’ve actually been running this for a long time, and so the post has have been in draft status far too long. I needed to finish it up before my memory would fail me.

The roadblocks here are mainly related to weak support for GPT. Windows up to at least version 7 can only boot from GPT when EFI is also used, and some of the Linux tools also have the one or other quirk when dealing with GPT (note that I was using Karmic; Lucid may not have some of the problems).

  • Create your GPT based partitions using gparted from an Ubuntu Live CD.
    • Make sure your Windows partition is first, or at least one of the first three.
    • Create a 1MB partition with the bios_grub flag. This is what the guided Ubuntu installer would do, and it’s required by grub to boot from GPT. gparted apparently enforces a minimum partition size of 8 megabytes, which should be find too, if wholly unnecessary, but I used parted to get to just one megabyte. I choose ext2 as the filesystem type, but I don’t think it matters.
  • In case you want to encrypt your root-filesystem: Make sure you have /boot on a separate, unencrypted partition.
  • Use gptsync to write out a Hybrid-MBR based on your GPT. This is required to make Windows boot. This is also why Windows needs to be one of the first three partitions. Further GPT partitions will not fit into the MBR. Note that for all I can tell, once Windows is booted, you can access your full GPT disc just fine.
    • If gptsync has problems with “unknown” partitions, try with a newer .deb from the Debian repositories.
    • Important: At this point, don’t touch your GPT partition table except with tools that can properly deal with Hybrid-MBRs. Unfortunately, that doesn’t include gparted, which will clear out the Hybrid-MBR. In this case, you have to run gptsync again.
  • Use fdisk to change partition type id of your Windows 7 partition to HPFS/NTFS (hex 7). If it hasn’t the proper id, Windows setup will refuse to install on it. When I tried to fix that by recreating the partition through the Windows installer, it ended up with slightly different LBA bounds than before (i.e. resulting in mismatch to the GPT table). So, do this now.
  • Install Windows.
  • Install Karmic. grub2 should pick up the Windows installation automatically (this is done by checking for a Boot/BCD file. If your Windows install isn’t found, you might want to look at /usr/lib/os-probes/ to find out why).

If you ever need to rewrite grub2, maybe cause you reinstalled Windows, you can boot into an Ubuntu Live CD and do:

# If you are using encryption, open the root partition
$ cryptsetup ...
# If you are using LVM, it might be necessary to make the partition available
$ lvchange ...

# Setup a chroot
$ mount $ROOT_PARTITION /mnt
$ mount $BOOT_PARTITION /mnt/boot
$ mount -o bind /dev /mnt/dev
$ mount -t proc proc /mnt/proc
$ mount -t sysfs sys /mnt/sys
$ chroot /mnt

# Update grub2's data (runs the os-prober, for example)
$ update-grub2
# Install the actual boot loader code
$ grub2-install /your/disk

Android: Build multiple versions of a project 7

This is a question that comes up once in a while.

For example, say you’d like to offer a full and a demo version, which isn’t really uncommon. Now, the Android Market enforces that each application package needs to use a unique package name, so you can’t just recompile your .apk with a DEMO=true variable set. That probably sounds easier than it is.

The package name is specified inside your AndroidManifest.xml file, but if you change it, you’ll probably notice two things:

  1. The activities etc. you declared in your manifest are flagged as not found, because you specified them in relative form (.MainActivity); those references are now based on the new package name, but the namespace of your Java classes hasn’t changed. That’s easy enough to fix, just reference all components with their full name.
  2. The R.java class that Android generates for you will be generated using the application namespace. Since that has no changed, so has the location of the R class. The imports all across your app still reference the old name though. This is the main stumble block.

What I used to do is have two git branches, developing the normal (full) version on the master branch, and having a second demo-packaging branch with the adjustments necessary to build the demo version, including the updated imports. That branch would be rebased against master whenever I release was in order.

However, I now got tired dealing with the merge conflicts that would inevitably occur whenever I had to change the import sections during developing the master branch, which was basically every time.

So I decided to try my luck with using Android’s ant-based build system, and if necessary, automate everything with search&replace. Fortunately, it turns out that this isn’t really necessary, for the most part: aapt, which is used to generate the R.java, takes a –custom-package option. Using that, you can compile your demo version in org.example.myapp.demo, and still have the R class generated into org.example.myapp.R, where all your imports point to.

Unfortunately, the option is pretty new – it only seems available in API Level 7, i.e. the 2.1 SDK, so you’ll need to build against that (of course, you’re app can still work on older versions).

Here’s a short step to step description of what I did:

  1. Start with the default Android build.xml file. If you don’t have one yet because you created your project through Eclipse, you can just run android create project with the appropriate options in a temporary directory, and copy the build.xml and build.properties files.
  2. As per the comments in this file, add import=false to the <setup> tag in this file, and copy the contents of {SDK_DIR}/platforms/android-2.1/templates/android_rules.xml below it. Unfortunately, there doesn’t seem to be a good way to customize this more granular fashion. Actually, copying only individual targets that you’d like to modify does work, but now you’re depending on implementation details of the rules file, which probably isn’t the better option.
  3. Look for the -resource-src target and add two new arguments to the exec call:
                <arg value="--custom-package" />
                <arg value="${application.namespace}" />

    Define application.namesace in your build.properties file. Set it to the namespace that is shared across all different versions, and that you use in your code when referencing your R class.

  4. You should now be able to build different versions of your app by simply changing the namespace declared in the manifest. Still, that can be automated. Personally, I have removed AndroidManifest.xml itself from version control, and am generating a copy with the appropriate namespace based on a template, with ant asking me about what version I want to build:
    <target name="-query-build-type">
        <input
            message="What version should I build?"
            validargs="default,donate"
            addproperty="opt.version" />
        <condition property="opt.package" value="${application.namespace}.d" else="${application.namespace}">
                <equals arg1="${opt.version}" arg2="donate" />
        </condition>
    </target>
    
    <target name="copy-templates">
        <filter token="__PACKAGE_NAME__" value="${opt.package}" />
        <delete file="AndroidManifest.xml" quiet="true" />
            <copy file="AndroidManifest.xml.template" tofile="AndroidManifest.xml" filtering="true" />
    </target>
    

    I’ve added -query-build-type as the first dependency to the compile target, and copy-templates as a dependency before the -resource-src target.

  5. In closing, I do need to complain about who utterly inappropriate XML is as a language for build files that humans are going to write by hand. I don’t get why people keep using that approach.

Update 2011-02-28: aapt now seems to have (not sure in which version this was introduced) a –rename-manifest-package option which can be used to greatly simplify that second step: No more search & replace.

Update 2011-03-01: Have a look at my new approach to do this, by not using ant at all for the build.

UserVoice alternatives which aren’t crazy expensive 3

I really like UserVoice – unfortunately, the pricing is totally off. I don’t even think I’m asking for that much. All I would like to do is use it as part of a mobile Android app. Which means, there would either need to be a mobile version (there isn’t), or an API that would allow me to build a mobile client (Commercial API access starts at $289/month).

Of course, there are a few other essential features which you need to pay for, like the ability to export your data, starting at $19. Which I would happily do, except if I wanted to use it for five different projects, I have to pay that amount for five different accounts.

The obvious alternative is GetSatisfaction. While they also don’t seem to have a mobile version, and they also claim that “Commercial API Access” is available in the $289 plan only, you do have the ability to register apps for an API Key in their account management, and projects like Satisfaction Remote Component make me hopeful that there is some kind of API available for less.

Unfortunately, I can’t really stand the service in general. There’s too many features I don’t care for complicating the UI, and it requires registration from users before they can vote. Even the registration is confusing: Instead of simply letting me create a channel, I’m being told something about employee verification, needing to wait four business days and stuff I need to do with my website.

Two decent alternatives seem to be crowdsound and UserEcho, but again, no mobile site or API, though crowdsound seems to be working on one.

On the open source side, I’ve found OpenEcho, which seems dead though, and of course there’s the option of using an StackOverflow-clone – which would still require to build a mobile version myself.

For now, I think I’m going to sit out another round and see if things look better in a couple months.

Update 2011-05-08: More options are:

-->