Static linking on Mac OS X

Sometimes you want to distribute a program without requiring users to install a long chain of dependencies. Typically we can do this by building a statically linked binary; for example, by passing the -static flag when linking with gcc. Apple, however, does not support statically linked binaries, so this approach won’t work on Mac OS X. Apple does, however, support statically linked libraries. This means we can get pretty close to what we want: we’ll statically link user-space libraries into our binary, but we won’t statically link with system libraries or the kernel.

To make this approach work, we need to hide all of the user-space shared libraries during the linking phase of our build process. The Apple linker always prefers shared libraries when it can find them, but will revert to static libraries in the absence of a shared library. I’ve yet to find a command-line parameter to change this behavior, but a brute-force solution is available. Essentially, we want to rename every shared library in /usr/local such that it no longer ends in .dylib, link our program, then put all of the shared libraries back the way we found them. I wrote this Python 3 script to automate the process. To temporarily hide the shared libraries on your system, run it like this:

$ ./hide-shared-libs.py -d /usr/local --hide


To restore your shared libraries, use the following command:

$ ./hide-shared-libs.py -d /usr/local --restore


Of course, this assumes that you have static versions of all of the libraries you intend to link with. By default, not all software will install static libraries. I use Homebrew to build software on Mac OS X, and have needed to add --enable-static to the configure parameters for a few packages. Once you rebuild with this parameter, however, static linking should work as expected.

Say “Hola” to mDNS-SD in Java

Service discovery in Java is not as simple as it should be. Apple’s reference implementation, Bonjour, is ironically very difficult to setup on Mac OS X (for JavaFX at least). Alternative pure-Java implementations exist, but I’ve yet to find one that identifies services quickly, offers configurable logging, and is robust enough to tolerate the mDNS responses generated by a 3rd-generation Apple TV (seriously, this kept crashing one of the services I tried). While the system-wide service design provided by Bonjour and Avahi is ideal for reducing network traffic via caching, it also adds an additional software dependency that is outside the control of an app developer.

Enter Hola, a dead-simple, pure-Java implementation of Multicast DNS Service Discovery (mDNS-SD). Zeroconf networking promises incredible ease-of-use for end users, but to encourage adoption that simplicity should extend to developers as well. Thus, Hola has been designed with low requirements and a (hopefully!) intuitive API. Want to find every TiVo on your local network? Here’s the code to do it:

try {
    Service service = Service.fromName("_tivo-mindrpc._tcp");
    Query query = Query.createFor(service, Domain.LOCAL);
    List<Instance> instances = query.runOnce();
    instances.stream().forEach(System.out::println);
} catch (UnknownHostException e) {
    logger.error("Unknown host: ", e);
} catch (IOException e) {
    logger.error("IO error: ", e);
}

Each returned Instance will have a user-visible name, a list of IP addresses, a port number, and a set of attributes:

String userVisibleName = instance.getName();
List<InetAddress> addresses = instance.getAddresses();
int port = instance.getPort();
if (instance.hasAttribute("platform")) {
    String platform = instance.lookupAttribute("platform");
}

And that’s it. To search for different services, just change the String passed to Service.fromName(). Right now only synchronous operation is supported, but I plan to add an asynchronous listener in the next release. As for requirements, Hola depends on nothing beyond Java 8 and SLF4J.

Development is hosted on GitHub at https://github.com/fflewddur/hola and pre-compiled JARs are available. If you’re looking for Java-based service discovery, give Hola a try. As always, comments and bug reports are welcome.

QuickTuring: a fast Java implementation of the Turing stream cipher

I recently needed to use Qualcomm’s Turing stream cipher for a Java project. Some Googling found a Java port of Qualcomm’s C++ reference implementation, which behaves almost exactly as expected: its output is always correct, but its runtime is significantly slower than the C++ implementation. As in, an order of magnitude slower. If this were the late ’90s, such a performance difference between C++ and Java might be expected… but it isn’t, so it’s not.

Digging into the code revealed several areas that required sequential execution, thus preventing HotSpot’s JIT-compiler from really going to town in its optimizations. Qualcomm’s implementation, by contrast, makes liberal use of loop unrolling and bit-parallel operations. By turing as many of these sequential operations as possible into bit-parallel operations, we can help the JVM perform more operations per clock cycle and allow it to inline critical methods.

Starting with the existing Java port as a base, I ported over the bit-parallel optimizations from Qualcomm’s implementation, in the process eliminating most of the loops requiring serial execution. The results are stark: with the original Java port, VisualVM showed that decrypting a 14 GB file resulted in 38.5 seconds spent in the Turing code. By contrast, the optimized variant spent only 5.8 seconds in the same code.

Qualcomm’s license allows both source and binary modification and redistribution, provided they are credited for Turing’s development. If that license suits your needs and you need a fast Java implementation of the Turing cipher, my port is available on GitHub as QuickTuring. Comments and bug reports are welcome.

Cygwin and the Case of the Unreadable Blues

Cygwin includes some nice built-in aliases for colorizing terminal output, but the default blue hue is so dark as to be unreadable on the black background. It’s easy to fix, though; just add the following Blue and BoldBlue lines to your ~/.minttyrc file:

Blue=64,64,255
BoldBlue=127,127,255

These lines will use a lighter shade of blue than the default, so you’ll be able to read it in a less-than-pitch-dark room. As an example, here’s my complete ~/.minttyrc file:

BoldAsFont=no
Transparency=high
Font=Consolas
FontHeight=10
ClicksTargetApp=no
Columns=120
Rows=52
ScrollbackLines=100000
Term=xterm-256color
Blue=64,64,255
BoldBlue=127,127,255

Hash strings to integers in PHP with the DJB hashing algorithm

I recently found myself needing a PHP implementation of the DJB hashing algorithm, but ran into a problem—in 64-bit PHP5, integers don’t overflow. Instead, they magically turn into floating point variables large enough to hold the new value. For short strings this isn’t really a problem (the hashing algorithm won’t cause an integer overflow in the first place), but for anything over five or six characters, you end up with numbers that aren’t comparable with other implementations of the algorithm (nor, for that matter, will they fit into any of MySQL’s numeric data types).

So, here’s a short function that uses PHP’s GNU Multiple Precision (GMP) module to perform the arithmetic at the necessary level of precision, then convert the result back to a standard PHP int:

define('PHP_INT_MIN', ~PHP_INT_MAX);

function hash_djb2($str){
	$hash = 5381;
	$length = strlen($str);

	for($i = 0; $i < $length; $i++) {
		$hash = gmp_add(gmp_mul($hash, 33), ord($str[$i]));
	        
		while (gmp_cmp($hash, PHP_INT_MAX) > 0) {
			$diff =	gmp_sub($hash, PHP_INT_MAX);
			$hash =	gmp_add($diff, PHP_INT_MIN);
			$hash =	gmp_sub($hash, 1); // off by 1
		}
	}
	return gmp_intval($hash);
}

I’ve only tested this on 64-bit PHP5 on Linux. PHP seems to handle integers a little differently across different platforms, so your mileage may vary. Enjoy!

PHP ZIP Extension for MAMP

Do you use MAMP as a web development testing environment on your Mac?

Do you need the PHP ZIP extension for dealing with archive files?

Are you running Mac OS X 10.6 Snow Leopard?

I do, and found making these tools play nice together to be far harder than it should have.  If you need to fix a similar setup, here are the steps that finally worked for me (on Mac OS X 10.6.4 with MAMP 1.9):

  1. Install XCode if you don’t already have it (we’re going to be doing a bit of compiling).
  2. Download the MAMP source code components (available towards the bottom of the page).
  3. When the MAMP source code package opens, go into the MAMP_src folder and double-click the php-5.3.2.tar.gz file (if you are using a different version of PHP, replace 5.3.2 with your actual version number).
  4. You should now have a php-5.3.2 folder in your Downloads folder.  Open up Terminal and cd to ~/Downloads/php-5.3.2.:
    cd ~/Downloads/php-5.3.2/
  5. Install the pcre.h header file (we need it to compile the extension):
    sudo cp ext/pcre/pcrelib/pcre.h /usr/include/php/ext/pcre/
  6. Configure PHP for i386 architecture:
    CFLAGS="-arch i386" ./configure
  7. Configure the ZIP extension for i386 architecture:
    cd ext/zip; CFLAGS="-arch i386" ./configure
  8. Build the extension:
    make
  9. Install the extension:
    cp modules/zip.so /Applications/MAMP/bin/php5.3/lib/php/extensions/no-debug-non-zts-20090626/
  10. Enable the extension by opening /Applications/MAMP/conf/php5.3/php.ini and appending the following line:
    extension=zip.so
  11. Remove the pcre.h header file we installed earlier, we don’t need it anymore.
  12. Restart MAMP.  If everything worked properly, you’ll be able to goto http://localhost/MAMP, click on phpInfo, and see zip in the list of enabled extensions.

StrayLight Photography

I’ve been meaning to revamp the photography section of this site for a while now; this weekend, I finally found the time to do it. I registered a new domain, straylightphotography.com, and put together a portfolio consisting of my 20 favorite shots (<shamelessPlug>many of which are currently on display at Interzone through February 28th!</shamelessPlug>). I’m hoping to quickly expand the site with themed portfolios (portraits, urban decay, etc.), but… first things first.

Also, the new portfolio has been an excuse to play with CSS3 and jQuery 1.4.  Visitors using Firefox, Safari, Chrome, or Opera should see a site that behaves like it was created with Adobe Flash, but is fully accessible and doesn’t require the proprietary Flash plug-in.  Visitors using Internet Explorer… well… it at least degrades cleanly.  Mostly.