Graham Edgecombe

Linux on a Netgate SG-8860-1U

2019-01-28T21:13:02+00:00

I recently obtained a second-hand Netgate SG-8860-1U firewall, which has now replaced my PC Engines apu2c4 board as the router in my home network. The apu2c4 is a perfectly good piece of hardware, but it only has three Ethernet ports compared to six on the Netgate machine. I was using all three ports (one connected to my modem and two bonded together and connected to my managed switch), so having some spare ports for expansion is useful – I’m already thinking about how I could rewire things to try to achieve 2 gigabits/sec with bonding between my desktop and server.

The Netgate machine comes bundled with pfSense, which is based on FreeBSD, but I’m not a particular fan of either – all my servers run Debian. I know a lot of people rave about how good the BSDs are at networking, but I’m not really convinced if you compare them to a modern Linux system. nftables is a significant improvement over iptables. Furthermore, people struggle to get the apu2c4 boards to do gigabit routing on BSDs, but Linux copes fine while using an insignificant amount of CPU power in comparison.

The specifications of the Netgate machine are:

Intel Atom C2758 (8 cores, 2.4 GHz)
8 GB DDR3L RAM (soldered onto the board)
64 GB eMMC flash
1x mSATA port
2x mPCIe ports
Mini USB serial port
2x USB 2.0 ports
6x Intel Gigabit NICs
- 2x Intel I211
- 4x Intel I354

The firmware is based on coreboot. It’s very bare bones – you can change the boot order and access the PXE console, but that’s about it. PXE only seems to work on the OPT1-4 ports, but I didn’t test this extensively. Booting from a USB stick also works.

It’s all fairly standard, so I thought it’d be quite easy to get Linux booting on it, but it did take me an entire evening trying to get the serial output to work.

On your desktop/laptop you’ll need to use a command like picocom -b 115200 /dev/ttyUSB0 to open the serial port. I didn’t have any problems getting output from the BIOS or GRUB, but I didn’t see any output after that point – making it very difficult to debug.

I discovered, after much trial and error, that the internal USB to serial adapter is actually connected to the COM2 port, so you need to pass console=ttyS1,115200n8 to the Linux command line. I’d been trying with ttyS0 instead, assuming it used COM1!

After that point everything went fairly smoothly, ignoring the slight annoyances that come with doing everything over a 115200 baud serial port instead of a high-speed SSH connection with a full-screen terminal.

Once you’ve installed Linux you need to make sure that your bootloader (e.g. GRUB) will continue using the serial port and pass the correct console= option to Linux. This varies between distributions. In Debian you need to set the following variables in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=1 --speed=115200 --parity=no --stop=1 --word=8"

and then run update-grub to update the /boot/grub/grub.cfg file. systemd automatically spawns a getty on ttyS1, so you don’t neeed to worry about fiddling around with /etc/inittab any more.

Modern Linux distributions assign network interface names predictably, and I spent a little bit of time mapping the predictable names to the labels on the front panel (hopefully this will save some people’s time!):

Network interface name	Front panel label
enp4s0	WAN
enp3s0	LAN
enp0s20f0	OPT1
enp0s20f1	OPT2
enp0s20f2	OPT3
enp0s20f3	OPT4

With a fairly complicated nftables ruleset, including connection tracking, it’s capable of routing traffic between two subnets at around 930 megabits/sec, which isn’t too far from the theoretical maximum after accounting for Ethernet and TCP/IP overhead:

~$ iperf -c rictusempra
------------------------------------------------------------
Client connecting to 81.187.119.162, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.1.2 port 43218 connected with 81.187.119.162 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   934 Mbits/sec
~$

systemctl daemon-reload and Puppet

2018-03-09T20:37:12+00:00

Update: Puppet 6.1 adds support for automatically calling systemctl daemon-reload when required, making the technique described in this post redundant.

I manage my machines at home (and at work) with Puppet, a configuration management tool. Sometimes one of my manifests needs to change a systemd service’s configuration – for example, if the upstream package didn’t ship with a unit file, or to override some settings in a .service.d directory. systemctl daemon-reload needs to be executed after changing configuration in the /etc/systemd directory, but Puppet doesn’t have built-in support for this.

Our initial solution at work was to create a systemd::daemon_reload class. It contains a single exec resource that runs systemctl daemon-reload when the class receives a refresh notification:

class systemd::daemon_reload {

  exec { '/usr/bin/systemctl daemon-reload':
    refreshonly => true,
  }

}

We then set up notification relationships from:

Any unit files in /etc/systemd/system to the systemd::daemon_reload class.
The systemd::daemon_reload class to the corresponding service.

For example, we’d do something like the following to create a custom unit file for fcgiwrap:

class fcgiwrap {

  include ::systemd::daemon_reload

  file { '/etc/systemd/system/fcgiwrap.service':
    # ...
    notify => Class['systemd::daemon_reload'],
    # ...
  }

  service { 'fcgiwrap':
    # ...
    subscribe => Class['systemd::daemon_reload'],
    # ...
  }

}

This creates the following dependency graph, with dashed lines representing a notification dependency:

This solution works nicely if you only have a single service. If fcgiwrap.service is updated, a notification is sent to the systemd::daemon-reload class, which invokes systemctl daemon-reload. In turn, a notification is sent to the fcgiwrap service, which is restarted.

However, it doesn’t work so well if you introduce more than one service. Here’s what the dependency graph looks like if we add a custom unit file for stunnel in the same way:

If any unit file is edited the notification dependencies cause every service to be restarted. This isn’t ideal if you have lots of services in a production environment where you want to minimize downtime caused by configuration changes.

I recently came up with a better solution to this problem by using a mixture of normal order-only dependencies and notification dependencies:

The unit files notify systemd::daemon_reload and the service resource.
The service resource has an order-only (but not notification) dependency on systemd::daemon_reload.

The changes to the code are minimal:

class fcgiwrap {

  include ::systemd::daemon_reload

  file { '/etc/systemd/system/fcgiwrap.service':
    # ...
    notify => [
      Class['systemd::daemon_reload'],
      Service['fcgiwrap'],
    ],
    # ...
  }

  service { 'fcgiwrap':
    # ...
    require => Class['systemd::daemon_reload'],
    # ...
  }

}

This produces the following dependency graph, with dashed lines representing notification dependencies and solid lines representing order-only dependencies:

If fcgiwrap.service is updated, notifications are sent to both systemd::daemon_reload and the fcgiwrap service resource, so systemctl daemon-reload will be invoked and Puppet will restart the fcgiwrap service.

The order-only dependency additionally ensures that systemctl daemon-reload is invoked before Puppet restarts fcgiwrap.

In the case of a single service there’s no real difference from the original solution. However, here’s what the dependency graph looks like when we add an additional custom unit file for stunnel:

Because the notification dependencies on systemd::daemon_reload have been replaced with order-only dependencies, only the corresponding service is restarted when a single unit file is changed.

Another neat side effect of this particular solution is that systemctl daemon-reload is only executed at most once per each Puppet run – which wouldn’t be the case with the more obvious solution of adding one systemctl daemon-reload exec resource per unit file/service combination.

Compressing X.509 certificates

2016-12-22T21:34:21+00:00

Introduction

I run a Certificate Transparency monitor which retains a copy of all the certificates it downloads. As CT logs are append-only, the monitor’s disk usage keeps creeping upwards. I’m always on the look out for ways to optimize disk usage to delay the need to buy bigger disks!

De-duplication

A few months ago I saved lots of space by de-duplicating the certificates. The original version of the monitor stored the raw leaf_input and extra_data fields of each entry. However, leaf certificates often end up in at least three logs and there are a relatively small number of intermediate and root certificates repeated over and over again. The leaf_input can’t be de-duplicated directly as it varies between logs even for the same certificate (e.g. the entry timestamps are almost always different), therefore, the new version of the monitor decodes the leaf_input and extra_data fields and stores the decoded data, allowing just the certificates to be de-duplicated. This reduced the disk usage from around 500 GiB to 150 GiB.

Munin graph showing the size of the database before and after de-duplication, which took place at the beginning of October.

Since then the disk usage increased to 195 GiB, so I decided to look at whether compression would make a significant saving.

Packed Encoding Rules

The certificates are currently stored in DER format. ASN.1 also defines a more compact mechanism for storing data than DER: Packed Encoding Rules (PER). However, PER isn’t suitable for this use case:

PER requires that the schema is known in advance. However, X.509 certificates support arbitrary extensions so it’s impossible to come up with a schema that will accept all possible extensions.
It’s likely that some certificate authorities have issued certificates that are not encoded correctly in DER, so it might not be possible to convert them to PER or get back to the original DER. Reproducing the exact encoding used by the log is required to calculate the log’s current Merkle tree hash.

Postgres TOAST compression

I decided that compressing the DER itself would be the best approach. The certificates are currently stored in a BYTEA field in a Postgres database, and it turns out that Postgres already compresses large rows itself. It has a maximum row size of 8,192 bytes. When a row grows larger than 2,048 bytes, Postgres uses a technique known as TOAST to try to fit the row into 8,192 bytes. First, it compresses the row with a custom variant of LZ77. If it’s still too large, it stores the data out-of-line, with the row containing pointers to the out-of-line data.

However, in a sample of 100,000 certificates selected randomly from the CT logs, only 3,179 (≈ 3%) were larger than 2,048 bytes, so Postgres’s built-in compression was only applied to a minority of certificates. Although the 2,048 byte threshold can be tweaked, I decided to do the compression at the application level.

Compressed X.509 Format

While deciding which compression algorithm to use, I found an interesting draft RFC from 2010, draft-pritikin-comp-x509-00. It defines the Compressed X.509 Format (CXF), which, as the name suggests, is designed specifically for compressing certificates. It’s essentially DEFLATE with a custom preset dictionary.

Preset dictionaries save space if you’re independently compressing lots of small items (such as certificates) as you don’t need to include the dictionary with the compressed data over and over again. The dictionary be distributed out-of-band a single time instead – for example, by embedding the it in the compression program itself.

Here’s a simple CXF implementation I wrote in Java:

private static final byte[] CXF_DICTIONARY = Base64.getDecoder().decode(
   "MIIBOTCCASOgAwIBAgIBATANBgkqhkiG9w0BAQUFADAZMRcwFQYDVQQDEw5odHRw" +
   "Oi8vd3d3LmNvbTAeFw0xMDA1MTExOTEzMDNaFw0xMTA1MTExOTEzMDNaMF8xEDAO" +
   "BgkqhkiG9w0BCQEWAUAxCjAIBgNVBAMTASAxCzAJBgNVBAYTAlVTMQswCQYDVQQI" +
   "EwJXSTELMAkGA1UEChMCb24xDDAKBgNVBAsTA291bjEKMAgGA1UEBRMBIDAfMA0G" +
   "CSqGSIb3DQEBAQUAAw4AMAsCBG6G5ZUCAwEAAaNNMEswCQYDVR0TBAIwADAdBgNV" +
   "HQ4EFgQUHSkK6busCxxK6PKpBlL9q8K1mcQwHwYDVR0jBBgwFoAUn7r/DVMuEpK9" +
   "Rxq3nyiLml10+nQwDQYJKoZIhvcNAQEFBQADAQA="
);

public byte[] compress(byte[] in) throws IOException {
    Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION, true);
    try {
        deflater.setDictionary(CXF_DICTIONARY);
        deflater.setInput(in);
        deflater.finish();

        try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
            byte[] buf = new byte[4096];

            while (!deflater.finished()) {
                int len = deflater.deflate(buf, 0, buf.length);
                out.write(buf, 0, len);
            }

            return out.toByteArray();
        }
    } finally {
        deflater.end();
    }
}

Decompression is very similar and left as an exercise for the reader for brevity.

CXF’s preset dictionary is a valid X.509 certificate containing several common subject/issuer fields and extensions. Some of the fields are left blank (e.g. subject CN), some contain a dummy value (e.g. subject/authority key identifier and the RSA modulus) and some contain the most likely value (e.g. subject country code of “US”, X.509 version number of 3).

Here’s the openssl x509 -text output:

$ <<EOF base64 -di | openssl x509 -inform der -text -noout
> MIIBOTCCASOgAwIBAgIBATANBgkqhkiG9w0BAQUFADAZMRcwFQYDVQQDEw5odHRw
> Oi8vd3d3LmNvbTAeFw0xMDA1MTExOTEzMDNaFw0xMTA1MTExOTEzMDNaMF8xEDAO
> BgkqhkiG9w0BCQEWAUAxCjAIBgNVBAMTASAxCzAJBgNVBAYTAlVTMQswCQYDVQQI
> EwJXSTELMAkGA1UEChMCb24xDDAKBgNVBAsTA291bjEKMAgGA1UEBRMBIDAfMA0G
> CSqGSIb3DQEBAQUAAw4AMAsCBG6G5ZUCAwEAAaNNMEswCQYDVR0TBAIwADAdBgNV
> HQ4EFgQUHSkK6busCxxK6PKpBlL9q8K1mcQwHwYDVR0jBBgwFoAUn7r/DVMuEpK9
> Rxq3nyiLml10+nQwDQYJKoZIhvcNAQEFBQADAQA=
> EOF
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=http://www.com
        Validity
            Not Before: May 11 19:13:03 2010 GMT
            Not After : May 11 19:13:03 2011 GMT
        Subject: emailAddress=@, CN= , C=US, ST=WI, O=on, OU=oun/serialNumber=
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (31 bit)
                Modulus: 1854334357 (0x6e86e595)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Subject Key Identifier:
                1D:29:0A:E9:BB:AC:0B:1C:4A:E8:F2:A9:06:52:FD:AB:C2:B5:99:C4
            X509v3 Authority Key Identifier:
                keyid:9F:BA:FF:0D:53:2E:12:92:BD:47:1A:B7:9F:28:8B:9A:5D:74:FA:74

    Signature Algorithm: sha1WithRSAEncryption
$

Constructing a DEFLATE dictionary

I wondered if I could do a better job of constructing a dictionary. There’s no reason that the dictionary has to be a valid X.509 certificate, other than ease of constructing/distributing it. Constructing a dictionary that isn’t constrained by the X.509 certificate structure could lead to several improvements. Some of my initial ideas included:

Adding OIDs for more signature algorithms, such as SHA-256 with RSA and ECDSA.
Adding more common values for the various fields – e.g. more country codes than “US”, more TLDs than “.com”, etc.
Omitting the random values for the RSA modulus and subject/authority key identifiers, as they’re unlikely to be useful.
Including a larger number of common X.509 extensions, such as:
- Key Usage
- Extended Key Usage
- Certificate Policies
- Subject Alternative Name

Last year CloudFlare wrote a blog post on using preset DEFLATE dictionaries, though from the perspective of compressing HTML pages rather than certificates. Together with the post they released an open-source tool for automatically constructing DEFLATE dictionaries from a sample of training data: dictator.

dictator runs “pseudo” LZ77 against the sample data to find strings that DEFLATE wouldn’t be able to compress. It calculates a score for each string based on popularity and length. The highest scoring strings are used to create the dictionary.

This saved me the need to construct a dictionary by hand, which might not have ended up being very effective. I just took a sample of 100,000 certificates from the CT logs and used dictator to generate one automatically.

I’ve uploaded the dictionary it generated (encoded in Base64) in case anyone else finds it useful.

Running strings against it is quite interesting – it includes common terms like “Domain Validated” and “DV”, names of some popular CAs, common OCSP/CRL URLs and even the disclaimer included in the certificatePolicies extension in every Let’s Encrypt certificate.

I expect examining it with xxd will also show it contains some common OIDs and other bits of DER encoding, but I didn’t look myself to avoid spending effort decoding DER by hand!

Results

As I was slightly concerned about overfitting, I decided to test how well the dictator-generated dictionary worked in comparison to plain DEFLATE and CXF with a completely different sample of 100,000 certificates. I used a compression level of 9 in all cases.

Here are the results:

Algorithm	Mean certificate size (bytes)
Uncompressed	1433.67
DEFLATE	1144.19
DEFLATE with CXF dictionary	1080.59
DEFLATE with `dictator`-generated dictionary	930.33

Shaving off over a third of the bytes on average is pretty impressive, especially considering that some data in a certificate (e.g. the public key and signature) is random and therefore impossible to compress.

Postgres bloat

I decided to go ahead and compress all the certificates in my CT monitor’s database with the dictator-generated dictionary. Although saving 500 bytes per certificate may not sound like much, at the time of writing there are around 69 million certificates in the database (and it keeps growing!)

Just before I started the compression process, the database’s disk usage was approximately 195 GiB. Around 70 GiB was used by the certificates themselves. Indexes and other tables made up the remaining disk usage.

As compressing all the certificates was a one-off job I didn’t spend lots of time optimizing the code, so it took around 4 days to run.

Postgres uses multiversion concurrency control (MVCC) to implement transaction isolation, which means UPDATEs are internally implemented as a DELETE followed by an INSERT. Deleted rows stick around in case another transaction is still reading the row.

The VACUUM command marks deleted rows that are no longer visible to any transactions as free, allowing the space occupied by the row to be re-used. If the deleted row is at the end of the data file the free space can be returned to the operating system.

As it took a while for the automatic vacuuming process to kick in, the size of the table initially grew during the compression process:

Munin graph showing the size of the database before and after compressing all certificates.

The growth slowed down after the autovacuum kicked in as compressed certificates could be inserted in gaps left by the vacuumed rows. However, the new rows didn’t fit perfectly in all the gaps so some free space, known as bloat, was still left over.

VACUUM FULL can be used to eliminate bloat. It makes a copy of the entire table, packing the rows together tightly with no free space between them. The old table is then deleted. This is visible in the graph as a brief spike (as both the old and new table exist simultaneously) before the steep drop to approximately 173 GiB (from 195 GiB at the start of the process).

As the monitor was still running over the 4 day compression period, the before and after sizes can’t be compared directly. Excluding the additional certificates downloaded during those 4 days, the size of the certificates table dropped from around 70 GiB to 45 GiB – a saving of 25 GiB.

Conclusion

The 25 GiB saving wasn’t quite as good as I expected – I over-estimated how much disk space was used by the certificates table in comparison to the rest of the database. However, it’s still a reasonable amount – it saved enough room for another month or so of growth, and the rate of growth is slower now too.

There are opportunities for further work in this area – for example, comparing DEFLATE against other compression algorithms.

It doesn’t just apply to CT monitors either. QUIC, a mix of HTTP/2, TLS and TCP implemented directly on top of UDP, already implements certificate chain compression to reduce round trip times. It uses gzip and a preset dictionary, which is constructed dynamically by concatenating:

~1500 bytes of common substrings from certificates taken from the Alexa top 5000 list.
Certificates the client already knows about (e.g. intermediate certificates or certificates from a previous interaction with the same server).

Purely by coincidence, there was a discussion on the TLS working group mailing list recently about adding a QUIC-style certificate compression extension to TLS.

Linux tproxy server in Java

2013-12-29T00:00:00+00:00

tproxy is a feature in Linux which allows an intermediate router to run a proxy server which can intercept and modify network traffic transparently (i.e. the end systems cannot tell that this has been done, as the source/destination IP addresses in the packets are not modified.) tproxy also works with IPv6 whereas non-transparent mechanisms such as the iptables REDIRECT target do not because of the lack of NAT support in the Linux IPv6 stack in older kernels.

Java’s standard library does not provide built-in support for using tproxy, and it has a few annoyances which make it tricky to implement.

The main operation required to enable tproxy on a socket is to set the IP_TRANSPARENT option with the setsockopt() system call. For example, the code to enable tproxy would look something like this in C:

int yes = 1;
if (setsockopt(fd, SOL_IP, IP_TRANSPARENT, &yes, sizeof(int)) < 0)
{
  perror("setsockopt");
  exit(1);
}

where fd is the file descriptor of the socket. The option must be set before the socket is bound with the bind() system call.

JNA is a library which makes it easy to call C functions from Java. Unlike JNI, you do not need to write any ‘glue’ code - you can convert the C signature to a Java signature, and JNA does all the hard work for you.

Here’s the JNA code which allows setsockopt() to be called from Java:

public interface CLibrary extends Library {
    public final CLibrary INSTANCE = (CLibrary) Native.loadLibrary("c", CLibrary.class);

    /* from /usr/include/bits/in.h */
    public final int SOL_IP = 0;
    public final int IP_TRANSPARENT = 19;

    /* from /usr/include/sys/socket.h */
    public int setsockopt(int socket, int level, int option_name, Pointer option_value, int option_len) throws LastErrorException;

    /* from /usr/include/string.h */
    public String strerror(int errnum);
}

Then setsockopt() can be called from Java like so:

IntByReference yes = new IntByReference(1);
try {
    /* option_len = sizeof(int) = 4 */
    CLibrary.INSTANCE.setsockopt(fd, CLibrary.SOL_IP, CLibrary.IP_TRANSPARENT, yes.getPointer(), 4);
} catch (LastErrorException ex) {
    throw new IOException("setsockopt: " + CLibrary.INSTANCE.strerror(ex.getErrorCode()));
}

So we just need to create a new java.net.Socket object, call setsockopt() and finally call bind() on the socket - easy, right? Unfortunately, it’s not quite so simple - creating a new Socket object in Java (in OpenJDK and the Oracle JVM) does not actually allocate a file descriptor. Instead, the file descriptor is allocated within Java’s bind() function itself - making it rather difficult to call setsockopt() at the appropriate point.

If you dig into the OpenJDK code, you can find the following native function which actually allocates the file descriptor (which I have simplified for readability):

/*
 * Class:     java_net_PlainSocketImpl
 * Method:    socketCreate
 * Signature: (Z)V */
JNIEXPORT void JNICALL
Java_java_net_PlainSocketImpl_socketCreate(JNIEnv *env, jobject this,
                                           jboolean stream) {
    jobject fdObj;
    int fd;

    fdObj = (*env)->GetObjectField(env, this, psi_fdID);
    if ((fd = JVM_Socket(domain, type, 0)) == JVM_IO_ERR) { /* calls the system's socket() function to allocate an fd */
        /* raise exception */
    }

    (*env)->SetIntField(env, fdObj, IO_fd_fdID, fd);
}

This is called from AbstractPlainSocketImpl’s create() method:

protected synchronized void create(boolean stream) throws IOException {
    fd = new FileDescriptor();
    socketCreate(stream); /* calls the native code from above */
}

In turn, the ServerSocket class itself calls the SocketImpl’s create() method within its own createImpl() method:

void createImpl() throws SocketException {
    try {
        impl.create(true); /* calls create() from above */
        created = true;
    } catch (IOException e) {
        throw new SocketException(e.getMessage());
    }
}

Until the socket has been bound or connected, the only way to trigger a call to createImpl() is via the bind() or connect() methods.

An astute reader might recognize the possible solution of calling createImpl() via reflection at this point to force the file descriptor to be allocated, which would then allow setsockopt() to be called before bind(). However, I have not managed to get this approach to work.

Instead, it turns out that the equivalent class to ServerSocket in NIO - ServerSocketChannel - allocates the file descriptor upon calling ServerSocketChannel.open(). Binding is done afterwards with a separate bind() method - which allows the setsockopt() call to be inserted in the correct place.

You can see this in the sun.nio.ch.ServerSocketChannelImpl class - the constructor calls Net.serverSocket():

ServerSocketChannelImpl(SelectorProvider sp) throws IOException {
    super(sp);
    this.fd =  Net.serverSocket(true);
    this.fdVal = IOUtil.fdVal(fd);
    this.state = ST_INUSE;
}

It’s fairly obvious to see that Net.serverSocket() allocates a file descriptor without digging into the native code which implements socket0() and setfdVal():

static FileDescriptor serverSocket(boolean stream) {
    return IOUtil.newFD(socket0(isIPv6Available(), stream, true));
}

public static FileDescriptor newFD(int i) {
    FileDescriptor fd = new FileDescriptor();
    setfdVal(fd, i);
    return fd;
}

The next problem is how to extract the integer file descriptor of a ServerSocketChannel. This can be done with reflection by reading the fd field within the sun.nio.ch.ServerSocketChannelImpl object. The fd field is a java.io.FileDescriptor. The FileDescriptor object itself contains the underlying file descriptor from the operating system as an integer in a field also called fd.

The following code implements the reflection technique as described above:

private static final Class<?> SERVER_SOCKET_CHANNEL_IMPL;
private static final Field SERVER_SOCKET_CHANNEL_FD;
private static final Field FD;

static {
    try {
        SERVER_SOCKET_CHANNEL_IMPL = Class.forName("sun.nio.ch.ServerSocketChannelImpl");

        SERVER_SOCKET_CHANNEL_FD = SERVER_SOCKET_CHANNEL_IMPL.getDeclaredField("fd");
        SERVER_SOCKET_CHANNEL_FD.setAccessible(true);

        FD = FileDescriptor.class.getDeclaredField("fd");
        FD.setAccessible(true);
    } catch (NoSuchFieldException | ClassNotFoundException ex) {
        throw new ExceptionInInitializerError(ex);
    }
}

public static int getFileDescriptor(ServerSocketChannel channel) throws IOException {
    try {
        FileDescriptor fd = (FileDescriptor) SERVER_SOCKET_CHANNEL_FD.get(channel);
        return FD.getInt(fd);
    } catch (IllegalAccessException ex) {
        throw new IOException(ex);
    }
}

This can be combined with the setsockopt() JNA code to create an openTproxyServerSocket() function:

public static ServerSocketChannel openTproxyServerSocket() throws IOException {
    ServerSocketChannel ch = ServerSocketChannel.open();
    int fd = getFileDescriptor(ch);

    IntByReference yes = new IntByReference(1);
    try {
        /* option_len = sizeof(int) = 4 */
        CLibrary.INSTANCE.setsockopt(fd, CLibrary.SOL_IP, CLibrary.IP_TRANSPARENT, yes.getPointer(), 4);
    } catch (LastErrorException ex) {
        throw new IOException("setsockopt: " + CLibrary.INSTANCE.strerror(ex.getErrorCode()));
    }

    return ch;
}

openTproxyServerSocket() could be used like so:

ServerSocketChannel ch = openTproxyServerSocket();
ch.bind(new InetSocketAddress(8443));
for (;;) {
    SocketChannel ch0 = ch.accept();
    /* do something with ch0 */
}

IP_TRANSPARENT can be set on a SocketChannel (as opposed to a ServerSocketChannel) in a completely analogous way - ServerSocketChannel and ServerSocketChannelImpl need to be replaced with SocketChannel and SocketChannelImpl respectively. The usage of the socket is the only part which is significantly different:

SocketChannel ch = openTproxySocket();
ch.bind(new InetSocketAddress("royal.gov.uk", 49152));   /* spoofed source IP address */
ch.connect(new InetSocketAddress("whitehouse.gov", 80)); /* destination IP address */

If you’re happy with using NIO, then that’s all the code you need. However, I was adding tproxy support to an existing program which used Java’s old-style I/O rather extensively - in particular, the SSLSocket class. Converting all the code to use NIO would be quite a lot of work - particularly as SSLEngine (the recommended way to use SSL with NIO in Java) is not a particularly friendly class.

It turns out that SocketChannel and ServerSocketChannel both have a socket() method which return an old-style Socket and ServerSocket respectively, which (supposedly) work as if you’d called new Socket() or new ServerSocket(), but in reality forward all the method calls to the corresponding NIO channel.

There is actually a long-standing bug I came across while doing this (it has been in the Java bug tracker since 2001). A Socket which is backed by a SocketChannel cannot read() and write() simultaneously - one call will block until the other completes. This is quite problematic in an application which is proxying data back and forth - consider the following example:

Proxy server opens connection to example.com:80.
Thread 1 in the proxy server calls read(). read() will not return until at least one byte of data is read.
Thread 2 in the proxy server calls write() with the HTTP request. write() dutifully waits for read() to finish. However, read() will never finish (ignoring a timeout, by which time it is too late anyway) because the HTTP server at example.com has not received a request to reply to.

To see why it happens, we need to dig into the SocketAdaptor source code - this is the class which extends Socket but forwards all of its calls to its corresponding SocketChannel. Its getInputStream() method returns a ChannelInputStream whose read() method does the following (if a timeout is not set, which is the default):

protected int read(ByteBuffer bb)
    throws IOException
{
    synchronized (sc.blockingLock()) {
        if (!sc.isBlocking())
            throw new IllegalBlockingModeException();
        if (timeout == 0)
            return sc.read(bb);

        /* code that won't be reached */
    }
}

SocketAdaptor’s getOutputStream() method returns an OutputStream whose write() method ultimately calls Channels.writeFully():

private static void writeFully(WritableByteChannel ch, ByteBuffer bb)
    throws IOException
{
    if (ch instanceof SelectableChannel) {
        SelectableChannel sc = (SelectableChannel)ch;
        synchronized (sc.blockingLock()) {
            if (!sc.isBlocking())
                throw new IllegalBlockingModeException();
            writeFullyImpl(ch, bb);
        }
    } else {
        writeFullyImpl(ch, bb);
    }
}

Both of these methods acquire the SocketChannel’s blockingLock() and then block. The lock is not released it until the operation has completed, which is the root cause of the bug.

To work around this bug, I created a class extending Socket (imaginatively called WorkaroundNioSocket). It overrides and passes through almost every method call to another Socket object given in the constructor - the socket obtained from SocketChannel.socket(). The only method which is not passed through directly is getOutputStream(), which wraps the underlying SocketChannel in a WritableByteChannel and returns Channels.newOutputStream() on the wrapped channel. WritableByteChannel does not extend SelectableChannel, so this prevents Channels.writeFully() from trying to obtain the blockingLock(), which allows simultaneous reads/writes to work again.

public final class WorkaroundNioSocket extends Socket {
    private final Socket socket;

    public WorkaroundNioSocket(SocketChannel channel) {
        this.socket = channel.socket();
    }

    @Override
    public OutputStream getOutputStream() throws IOException {
        final SocketChannel ch = socket.getChannel();
        return Channels.newOutputStream(new WritableByteChannel() {
            @Override
            public int write(ByteBuffer src) throws IOException {
                return ch.write(src);
            }

            @Override
            public boolean isOpen() {
                return ch.isOpen();
            }

            @Override
            public void close() throws IOException {
                ch.close();
            }
        });
    }

    /* pass through every other method */
    @Override
    public void connect(SocketAddress endpoint) throws IOException {
        socket.connect(endpoint);
    }

    /* etc */
}

Something similar also needs to be done for the ServerSocket - this time to ensure Sockets it accept()s are wrapped with WorkaroundNioSocket objects:

public final class WorkaroundNioServerSocket extends ServerSocket {
    private final ServerSocket socket;

    public WorkaroundNioServerSocket(ServerSocketChannel channel) throws IOException {
        this.socket = channel.socket();
    }

    @Override
    public Socket accept() throws IOException {
        Socket client = socket.accept();
        return new WorkaroundNioSocket(client.getChannel());
    }

    /* pass through every other method */
    @Override
    public void bind(SocketAddress endpoint) throws IOException {
        socket.bind(endpoint);
    }

    /* etc */
}

These wrapper classes can be used by replacing the ch.socket() call with either new WorkaroundNioSocket(ch) or new WorkaroundNioServerSocket(ch).

Bringing everything in this post together allows you to transparently proxy SSL traffic in Java on Linux, the goal I wanted to achieve in the project I’m working on:

ServerSocket server = new WorkaroundNioServerSocket(openTproxyServerSocket());
for (;;) {
    Socket socket1 = server.accept();
    SocketAddress src = socket1.getRemoteSocketAddress();
    SocketAddress dst = socket1.getLocalSocketAddress();

    Socket socket2 = new WorkaroundNioSocket(openTproxySocket());
    socket2.bind(src);
    socket2.connect(dst);

    SSLContext ctx = ...;
    SSLSocket sslSocket1 = (SSLSocket) ctx.getSocketFactory().createSocket(socket1, ...);
    sslSocket1.setUseClientMode(false);
    SSLSocket sslSocket2 = (SSLSocket) ctx.getSocketFactory().createSocket(socket2, ...);

    /* code to copy data from sslSocket1 -> sslSocket2 and vice-versa */
}

Using IntelliJ IDEA's javac2 in Gradle

2013-04-03T00:00:00+00:00

JetBrains’s IntelliJ IDEA uses a wrapper around the Java compiler, named javac2, to provide additional support for compiling .form files produced by the IDE, and for processing @Nullable and @NotNull annotations. It is naturally supported inside IDEA itself, and also by Apache Ant. However, other build systems like Gradle do not support it out of the box. Supporting it is useful - e.g. if you wanted to run a continuous integration server, which means you cannot use IDEA for building, and want it to compile your forms. Also, it could be useful if other developers on the same project used a different IDE.

Using Gradle’s integration with Apache Ant, it is possible to add javac2 support using the javac2 Ant task provided by JetBrains. This solution is by no means perfect - it doesn’t completely integrate with Gradle - but it did enough to meet my requirements.

I added a new configuration, antTask, so that javac2 does not pollute the classpath of the project itself:

configurations {
    antTask
}

Unfortunately the latest version of javac2 is not available in the Maven central repository, so as a workaround I added a flatDir repository to load the jars from a local directory (in this example, the directory lib/ below the root of the project:)

repositories {
    flatDir dirs: "${rootDir}/lib"
    mavenCentral()
    /* any other repositories... */
}

javac2 also depends on a customized version of ObjectWeb ASM, the IDEA forms runtime and JDOM 1.x, which can be expressed like so:

dependencies {
    antTask name: 'javac2', version: '12.1.0'
    antTask name: 'forms_rt', version: '12.1.0'
    antTask name: 'asm4-all', version: '12.1.0-idea'
    antTask group: 'org.jdom', name: 'jdom', version: '1.1'
}

There’s some flexibility in what you call the first three dependencies, as they are loaded locally. I chose to give them the same version numbers as the IDE, with a suffix of -idea on the ASM dependency to indicate this is a modified version of ASM.

These files must be copied over from IDEA’s directory like so (where $IDEA_ROOT is the path to your IDEA installation:)

mkdir lib
cp $IDEA_ROOT/redist/javac2.jar lib/javac2-12.1.0.jar
cp $IDEA_ROOT/redist/forms_rt.jar lib/forms_rt-12.1.0.jar
cp $IDEA_ROOT/lib/asm4-all.jar lib/asm4-all-12.1.0-idea.jar

The fourth and final dependency, JDOM, can be found in the Maven central repository and has not been modified by JetBrains, so it doesn’t need to be copied over from the IDEA directory.

Finally, the compileJava task needs to be overwritten to use javac2 via the Ant task:

task compileJava(overwrite: true, dependsOn: configurations.compile.getTaskDependencyFromProjectDependency(true, 'jar')) {
    doLast {
        project.sourceSets.main.output.classesDir.mkdirs()
        ant.taskdef name: 'javac2', classname: 'com.intellij.ant.Javac2', classpath: configurations.antTask.asPath
        ant.javac2 srcdir: project.sourceSets.main.java.srcDirs.join(':'),
            classpath: project.sourceSets.main.compileClasspath.asPath,
            destdir: project.sourceSets.main.output.classesDir,
            source: sourceCompatibility,
            target: targetCompatibility,
            includeAntRuntime: false
    }
}

The overwrite flag ensures the default task is replaced with the new one.

In a multi-module project, to ensure that a module is compiled after any modules it depends on, the getTaskDependencyFromProjectDependency method is used to populate the dependsOn property. It converts the list of module dependencies to a list of jar tasks for those modules. For example, if a module dependend on moduleA and moduleB, then the function would return moduleA:jar and moduleB:jar. This behaviour is required because the dependencies need to be added to the classpath of javac, and for that they need to have been compiled.

Ant will not create the output directory itself, so there’s a call to mkdirs() to do this before running the Ant task.

The ant.taskdef and ant.javac2 lines essentially map directly to <taskdef> and <javac2> in Ant’s XML syntax. The taskdef is used to load the javac2 task from the antTask configuration. The call to javac2 immediately following this is what actually does the compiling work. The arguments passed to it are exactly the same as the options the standard <javac> task accepts, so it shouldn’t need much explaining. Their values are taken from Gradle’s project object, so it’ll work even if you’ve changed your paths around.

There is some scope for future work, for example extending it to also compile test sources with javac2, or converting it to a plugin. However, I didn’t need these features, so I chose to go for a simple implementation instead (and I don’t know enough about the Gradle internals yet either!)

Credit should go to Douglas Bullard, who posted the original code for integrating javac2 with Gradle on the JetBrains forum, which this blog post improves upon (by removing the hard-coded paths, adding multi-module support and simplifying it.)

Aggregated Javadoc with APIviz in Gradle

2013-03-09T00:00:00+00:00

Gradle is a relatively new Java build system, which mixes some of the ideas from Maven and Ant: the well-defined structure, multi-module support and dependency management from Maven, but also the customizability from Ant.

I recently switched to using it for one of my projects, which was using Maven, when I needed to implement some rather project-specific behaviour in the build and was fairly impressed with it. Unfortunately, it still lacks some things which are easier to do in other build systems, and this post describes how I added code to my build script to generate aggregated Javadoc with APIviz UML diagrams.

There is an open ticket in the Gradle bug tracker about built-in aggregated Javdoc support, so it might be possible that this will be built into Gradle in the future. However, as I lack a time machine, I decided to base my implementation on the ‘workaround’ code in that ticket, with changes to support APIviz.

To avoid cluttering the classpath of my actual application with APIviz, I created a new configuration for doclet JARs, and then added APIviz as a dependency in this configuration to the root project (you’ll need to have added the Maven central repository to your repositories section for this to work:)

configurations {
    doclet
}

dependencies {
    doclet group: 'org.jboss.apiviz', name: 'apiviz', version: '1.3.2.GA'
}

The aggregateJavadoc task itself inherits from the standard Javadoc task. Mostly it consists of configuration, however, to workaround some APIviz projects I also had to introduce some code which runs before the task - APIviz expects all the -sourceclasspath directories passed to it to actually exist, whereas Gradle does not create these directories unless a file is placed in them.

You can see the workaround in the start of task, which simply ensures these directories actually exist to stop APIviz from complaining. This is implemented by adding the code in the doFirst section, which is executed before Javadoc is.

task aggregateJavadoc(type: Javadoc) {
    group 'Documentation'
    description 'Generates aggregated Javadoc API documentation for the main source code of all projects.'

    doFirst {
        subprojects.each { project ->
            project.sourceSets.main.output.each { dir ->
                dir.mkdirs()
            }
        }
    }

The next piece of code is what I borrowed from the ticket I mentioned earlier. It merges the source and classpaths of all submodules into a single path containing them, and sets these as options in the Javadoc task:

    source subprojects.collect { project ->
        project.sourceSets.main.allJava
    }
    classpath = files(subprojects.collect { project ->
        project.sourceSets.main.compileClasspath
    })

APIviz also expects another option to be passed to it called -sourceclasspath, which is the location where the compiled .class files are placed for your project’s .java files. This is the code I used to set that option:

    options.addStringOption('sourceclasspath', files(subprojects.collect { project ->
        project.sourceSets.main.output
    }).getAsPath())

I chose to make the destination directory match the directory non-aggregated Javadocs are placed in:

    destinationDir file("$project.buildDir/docs/javadoc/")

The remainder of the options are mostly down to your preference. The -nopackagediagram is an APIviz-specific option, which I decided to use as the package diagram tends to look messy when your project has more than a handful of packages. The other options are all fairly standard Javadoc options, which are explained well in the official Javadoc documentation.

    options.showAll()
    options.addBooleanOption('nopackagediagram', true)
    configure(options) {
        windowTitle '<super secret project name here>'
        docTitle '<super secret project name here>'
        links 'http://docs.oracle.com/javase/7/docs/api/'

The final part to mention is the options to set the class name of the APIviz doclet, and the path to it, which can be done like so (notice the use of the configuration created earlier:)

        doclet 'org.jboss.apiviz.APIviz'
        docletpath file(configurations.doclet.asPath)
    }
}

and that’s it! Running gradle aggregateJavadoc should create Javadocs for all of your sub-modules in the build/docs/javadoc directory, including the UML diagrams APIviz draws.

Hosting Trac with Passenger and nginx

2011-09-28T00:00:00+00:00

As well as being able to host Ruby applications with Rack, Phusion Passenger can also host Python applications with WSGI. This article shows how I set up Trac on Debian with nginx and Passenger.

I decided to use the Trac package available in the Debian repository, installing it is done with a simple apt-get command:

sudo apt-get install trac

You then need to create a Trac database. I decided to store it in /srv/trac but you can choose any location in which Passenger has read/write permissions. The command to run is:

trac-admin /srv/trac initenv

You will be asked a series of questions including the name of the project, the database connection string (I left this as the default SQLite option), the type of repository and the path to the repository. These values will vary depending on your current setup.

The next step is to deploy the static files to the document root we will use in nginx. I store all my websites in /srv/www so used the following command:

trac-admin /srv/trac deploy /srv/www/trac.grahamedgecombe.com

This will create two sub-directories: /srv/www/trac.grahamedgecombe.com/cgi-bin and /srv/www/trac.grahamedgecombe.com/htdocs. However, Passenger expects a folder called public and a script called passenger_wsgi.py. The best way to fix this is to create some symbolic links like so:

ln -s htdocs public
ln -s cgi-bin/trac.wsgi passenger_wsgi.py

Configuring Passenger generally requires lines like the following in the http block in your nginx.conf file:

passenger_root /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9;
passenger_ruby /usr/local/bin/ruby;

The appropriate lines will have been given to you as part of the Passenger installation procedure.

I also include the following configuration in the http block:

passenger_use_global_queue on;
passenger_user www-data;
passenger_group www-data;
passenger_friendly_error_pages off;
passenger_pool_idle_time 0;

You should also consider configuring the pool size with passenger_max_pool_size, passenger_min_instances and passenger_max_instances_per_app, the exact values you should use depend on how many applications you are running with Passenger and how much memory your server has.

Configuring a virtual server is relatively simple now and can be done by placing something like the following in your http block:

server {
  listen 80;
  server_name trac.grahamedgecombe.com;
  root /srv/www/trac.grahamedgecombe.com/public;
  passenger_enabled on;
}

Once that is done, reboot nginx and everything should work.

Custom meshes in Ogre3D

2011-08-05T00:00:00+00:00

Introduction

Recently I have been working on a small project using the Ogre3D rendering engine and needed to dynamically create a mesh at runtime. There is an example on the Ogre3D wiki but it assumes you have some underlying knowledge on the way meshes are represented by lower-level APIs such as OpenGL and Direct3D so I decided to write this blog post to give some more detail.

Vertices and triangles

Most of the time in a 3d game, you are looking at a bunch of triangles which make up more complicated shapes. Each of these triangles is defined by a set of three points - its vertices.

In order for the graphics card to render these shapes, all you need to do is send it the coordinates of the vertices and then tell it how to join them up. Really old graphics cards used to be sent the vertices they needed to render each frame, however, this isn’t the best way of doing it (nevertheless, you still see it in many OpenGL tutorials around the web). Your application becomes limited by the capacity of the bus for little reason, especially considering that most of the models in the scene are static.

This has been improved by the creation of “vertex buffers”. Vertex buffers allow you to send the coordinates of the vertices to the card once. They are then kept in video memory, so that you don’t need to clog up the bus each frame. Because you are no longer limited by the bus, you can also render many more models.

Vertex buffers in OGRE

In OGRE, the HardwareBuffer class represents a vertex buffer. The HardwareBufferManager singleton allows you to create new hardware buffers.

Creating a vertex buffer is quite simple, you call the createVertexBuffer method and tell it the following pieces of information:

How you plan to use the buffer
The number of bytes that represent each vertex
The number of vertices in the buffer

I’ll focus on the first point to begin with. You can either mark the buffer as being HBU_STATIC or HBU_DYNAMIC. Static buffers are ones you don’t plan to modify very often (you can still modify them, though). Dynamic buffers are ones you plan to modify frequently.

You can also mark a buffer as being HBU_WRITE_ONLY which means you don’t plan to read data back from the buffer.

These allow the graphics card/driver to pick the best way to store your buffer.

The number of vertices in the buffer is fairly self explanatory. However, the number of bytes that represent each vertex requires more explaining. You need to pass this value because there is no fixed way to represent a vertex. As well as the position, your application might need normals, texture coordinates, colours, etc.

This leads us into the next section - vertex declarations.

Vertex declarations in OGRE

The VertexDeclaration class holds a list of VertexElements. These are used to describe the structure of each vertex.

For example, if you wanted each vertex to hold its position, a 2d texture coordinate and a normal, you would need to create three elements for each of these.

Each element has a type and semantic. The type is the data type - e.g. a 2d texture coordinate is usually represented by two floating point numbers, so its type would be VET_FLOAT2. The semantic is VES_TEXCOORD, as it is a texture coordinate.

So for the example earlier, we’d need to have:

VET_FLOAT3 VES_POSITION for the position
VET_FLOAT2 VES_TEXCOORD for the texture coordinate
VET_FLOAT3 VES_NORMAL for the normal

Index buffers

The final thing we need is to do is to tell the graphics card how to connect the buffers into triangles. There are several ways to do this, but the easiest and most versatile is to use OT_TRIANGLE_LIST. With this, you supply a list of indices which point to the vertices from earlier. Each three indices describe a triangle. You must specify the indices in anti-clockwise order in order for the triangle to face you. The other side of the triangle will be ‘culled’ and invisible, so for two-sided triangles you must specify the indices twice.

The code

The following code is a commented example on how to create a simple mesh, which should be fairly easy to understand if you read the theory above.

/* create the mesh and a single sub mesh */
Ogre::MeshPtr mesh = Ogre::MeshManager::getSingleton().createManual("CustomMesh", "General")
Ogre::SubMesh *subMesh = mesh->createSubMesh();

/* create the vertex data structure */
mesh->sharedVertexData = new Ogre::VertexData;
mesh->sharedVertexData->vertexCount = 3;

/* declare how the vertices will be represented */
Ogre::VertexDeclaration *decl = mesh->sharedVertexData->vertexDeclaration;
size_t offset = 0;

/* the first three floats of each vertex represent the position */
decl->addElement(0, offset, Ogre::VET_FLOAT3, Ogre::VES_POSITION);
offset += Ogre::VertexElement::getTypeSize(Ogre::VET_FLOAT3);

/* the second three floats of each vertex represent the colour */
decl->addElement(0, offset, Ogre::VET_FLOAT3, Ogre::VES_COLOUR);
offset += Ogre::VertexElement::getTypeSize(Ogre::VET_FLOAT3);

/* create the vertex buffer */
Ogre::HardwareBuffer *vertexBuffer = Ogre::HardwareBufferManager::getSingleton().
    createVertexBuffer(offset, mesh->sharedVertexData->vertexCount, Ogre::HardwareBuffer::HBU_STATIC);

/* lock the buffer so we can get exclusive access to its data */
float *vertices = static_cast<float *>(vertexBuffer->lock(Ogre::HardwareBuffer::HBL_NORMAL));

/* populate the buffer with some data */
vertices[0] = 0; vertices[1] = 1; vertices[2] = 0; /* position */
vertices[3] = 1; vertices[4] = 0; vertices[5] = 0; /* colour */

vertices[6] = -1; vertices[7] = -1; vertices[8] = 0; /* position */
vertices[9] = 0; vertices[10] = 1; vertices[11] = 0; /* colour */

vertices[12] = 1; vertices[13] = -1; vertices[14] = 0; /* position */
vertices[15] = 0; vertices[16] = 0; vertices[17] = 1; /* colour */

/* unlock the buffer */
vertexBuffer->unlock();

/* create the index buffer */
Ogre::HardwareBuffer *indexBuffer = Ogre::HardwareBufferManager::getSingleton().
    createIndexBuffer(Ogre::HardwareIndexBuffer::IT_16BIT, mesh->sharedVertexData->vertexCount, Ogre::HardwareBuffer::HBU_STATIC);

/* lock the buffer so we can get exclusive access to its data */
uint16_t *indices = static_cast<uint16_t *>(indexBuffer->lock(Ogre::HardwareBuffer::HBL_NORMAL));

/* define our triangle */
indices[0] = 0;
indices[1] = 1;
indices[2] = 2;

/* unlock the buffer */
indexBuffer->unlock();

/* attach the buffers to the mesh */
mesh->sharedVertexData->vertexBufferBinding->setBinding(0, vertexBuffer);
subMesh->useSharedVertices = true;
subMesh->indexData->indexBuffer = indexBuffer;
subMesh->indexData->indexCount = mesh->sharedVertexData->vertexCount;
subMesh->indexData->indexStart = 0;

/* set the bounds of the mesh */
mesh->_setBounds(Ogre::AxisAlignedBox(-1, -1, -1, 1, 1, 1));

/* notify the mesh that we're all ready */
mesh->load();

/* you can now create an entity/scene node based on your mesh, e.g. */
Ogre::Entity *entity = sceneManager->createEntity("CustomEntity", "CustomMesh", "General");
entity->setMaterialName("YourMaterial", "General");
Ogre::SceneNode *node = rootNode->createChildSceneNode();
node->attachObject(entity);

Elementary cellular automata in Ruby

2011-07-02T00:00:00+00:00

Introduction

Stephen Wolfram’s elementary cellular automata are something I read about a while ago and thought looked quite cool. Cellular automata, in general, are grids of cells which evolve over time without any external input. I decided to write a small Ruby program that would be capable of calculating a new generation for any elementary cellular automaton given an initial state.

This is the code I ended up with:

def evolve(rule, state)
  def state.[](x)
    (x < 0 or x >= length) ? 0 : fetch(x)
  end
  next_state = []
  for i in (-1 ... state.length + 1)
    bit = state[i + 1] | (state[i] << 1) | (state[i - 1] << 2)
    mask = 1 << bit
    next_state[i + 1] = ((mask & rule) != 0) ? 1 : 0
  end
  return next_state
end

I think it illustrates some of the nice features from Ruby, such as metaprogramming and the clean syntax so I decided to put it up here with a bit of an explanation.

def evolve(rule, state)
  def state.[](x)
    (x < 0 or x >= length) ? 0 : fetch(x)
  end

The first line defines the method, evolve(), which is passed the rule number and an array containing the current state. The array should simply contain a sequence of 0s and 1s.

The next three lines add a method only to the state object. This method overrides the array access operator, so that indices outside of the array always return 0, as opposed to the normal Ruby behaviour. This simplifies access to the array later on.

  next_state = []
  for i in (-1 ... state.length + 1)

The next line simply creates a new array for the new state. Then, all of the indices inside the current state array as well as one before it and after it are looped through by the for loop. This is done because for each generation the array of cells becomes larger by one cell at each end.

    bit = state[i + 1] | (state[i] << 1) | (state[i - 1] << 2)

This snippet takes the values of the cells to the top left, top middle and top right above the current cell and turns it into a single number using bit shifts. This number can be used to find the next state of the cells.

For example, in Rule 30, if the current state is 100 in binary, this is the 4th bit. 30 in binary is 00011110, so the next state would be 1.

    mask = 1 << bit
    next_state[i + 1] = ((mask & rule) != 0) ? 1 : 0

This code actually implements the above check using the AND operator. If we want to test the 5th bit, we perform 2^4 (equivalent to 1 « 4) which gives 16. By performing 30 AND 16 we get a non-zero value, which indicates the next state is indeed 1.

  end
  return next_state
end

The last few lines simply terminate the loop, return the state and end the method.

ASCII art

I then wrote a quick program to test this by drawing ASCII art:

rule = ARGV[0].to_i
state = [1]
max = 64
max.times do |gen|
  (max - gen).times { print ' ' }
  state.each { |x| print x == 1 ? '*' : ' ' }
  puts

  state = evolve(rule, state)
end

This is the first few lines of output it produces for rule 90, one of my favourites:

               *
              * *
             *   *
            * * * *
           *       *
          * *     * *
         *   *   *   *
        * * * * * * * *
       *               *
      * *             * *
     *   *           *   *
    * * * *         * * * *
   *       *       *       *
  * *     * *     * *     * *
 *   *   *   *   *   *   *   *
* * * * * * * * * * * * * * * *

I really find it amazing that from a simple rule you can get such an elegant pattern (or with a different rule, you can get something that is chaotic or even turing-complete).

Nature

What I find most startling about elementary cellular automata is that patterns built up from these simple rules can also be found in nature. For example, here is a picture of rule 30 and a species of sea snail that has a similar pattern on its shell:

(photograph from Richard Ling, licensed under CC BY-SA 3.0)

Conclusion

This post has probably seemed a bit rambling and pointless, but it was on my list of interesting things to talk about and if you haven’t seen these before then hopefully you’ll get the ‘oh, that’s quite cool’ moment too!

IP over DNS with Iodine

2011-06-01T00:00:00+00:00

Introduction

If there are any two services your school or workplace allows through their firewall, you can be almost sure they are HTTP and DNS. Unfortunately, HTTP access is usually filtered, and if your network admins are anything like mine, lots of useful, legitimate websites get blocked for no reason. I used to abuse the CONNECT method to establish a TCP connection to my SSH server and from there would tunnel traffic using SOCKS over the SSH connection to be able to browse properly again.

Unfortunately, recently they changed the firewall software and it now checks to see if traffic over CONNECT is using SSL, and if it is, it checks to see if there is a non self-signed certificate in use. If those conditions fail, the connection is blocked. This means that I could no longer connect to my SSH server. Running SSH over SSL using something like stunnel was also out, since I don’t own a signed certificate.

DNS tunneling

I wasn’t going to be beaten that easily. After remembering a few old Slashdot posts about DNS tunneling, I decided to look into it and compared a few pieces of software. The one I found to be most suitable was Iodine.

There’s a few drawbacks: it’s quite slow and it doesn’t seem to be able to co-exist with a real DNS server. But it works!

The first thing you need to do is to add an NS record to a subdomain of your main domain. Apparently it’s a good idea to make this as short as possible to give extra space for the data itself, so I chose t.grahamedgecombe.com. The NS record should point to the address of the server you’re running Iodine on. For me that was simply grahamedgecombe.com.

On the server, I ran these commands to install and start Iodine:

sudo apt-get install iodine
sudo iodined 10.0.0.1 t.grahamedgecombe.com

The apt-get may need to be substituted with another package manager if you’re not on something Debian-based.

The second command starts Iodine in the background, using the subdomain t.grahamedgecombe.com. Iodine creates a virtual network, so you also need to select an IP address for the server - 10.0.0.1 in this case. The client’s IP address will be the next following the server’s - 10.0.0.2 in this case. You should select an IP address within the private IPv4 network ranges that doesn’t conflict with any private networks your server and client is connected to.

On the client, I used these commands to install and start Iodine:

sudo apt-get install iodine
sudo iodine t.grahamedgecombe.com

Once this is complete, you’ll be able to connect to the server using the IP address you specified on the server. So to set up my SOCKS SSH tunnel, I ran the following command:

ssh -D1080 10.0.0.1

With luck, it should all be working!

If not I’d recommend checking the following things:

The DNS server Iodine is using is operating properly - when testing it locally I encountered issues with the local dnsmasq server that caches queries.
The firewall - I forgot to add the dns0 interface Iodine creates to the firewall configuration, this meant that Iodine said it was connected but all the packets were dropped.

Generating sitemaps with Rails

2011-05-11T00:00:00+00:00

Introduction

Sitemaps are XML files sitting on your web server that are used to give hints to search engines about the existence of new pages, how often pages change and the priority a search engine should give to each page.

The format is very simple to understand and complete documentation is available on sitemaps.org. There is a root urlset element which contains multiple url elements. Each of these url elements must contain the loc element which specifies the location of that URL. It can optionally contain lastmod, the date and time at which the page was last modified, changefreq, how often you expect the page to change and priority, the priority of that page compared to others on your website.

Implementing sitemaps in Rails

Rails makes it very easy to generate your own sitemap.xml file dynamically. The sitemap can be implemented using a simple controller to generate a list of pages and a view that use Builder to generate the XML output.

The first step is to create the route in the config/routes.rb file, which will look like the this:

get 'sitemap', :to => 'sitemap#show'

This route simply routes GET requests for sitemap.xml, to the SitemapController’s show method. You do not need to worry about specifying .xml in the route, Rails will automatically figure it out and include the correct view.

The next step is to create the controller in the app/controllers/sitemap_controller.rb file. This needs to fetch information about all of your pages from the database or disk or wherever you are storing them, so the code will vary. In this example I’ll pretend that I’ve got a Photo model and a page for each photo, along with a Link model used by a single page of links.

class SitemapController < ApplicationController
  def show
    # grab info about all the photos since they each have their own page
    @photos = Photo.all

    # grab info about the most recently-updated link as they share a page
    @link = Link.first :order => 'update_at desc'
  end
end

This is fairly simple code that should be easily understood by anyone with a basic understanding of ActiveRecord.

The final part is to create a view called app/views/sitemap/show.xml.builder. The name signifies that this is a view for the XML format (as I said earlier, Rails uses this to automatically detect the view to use) using Builder as a template engine.

Builder uses a domain-specific language implemented on top of Ruby to allow you to easily create XML documents.

In general, your calls to Builder look like this:

xml.tag_name :attribute => 'value of attribute' do
  # nest tags here, can include ruby code to do loops, etc.
end

Here is the code in the view that we can use to produce the sitemap:

# this produces the <?xml ... ?> tag at the start of the document
#   note: this is different to calling builder normally as the <?xml?> tag
#         is very different to how you'd write a normal tag!
xml.instruct! :xml, :version => '1.0', :encoding => 'UTF-8'

# create the urlset
xml.urlset :xmlns => 'http://www.sitemaps.org/schemas/sitemap/0.9' do
  # photo pages
  @photos.each do |photo|
    xml.url do # create the url entry, with the specified location and date
      xml.loc photo_url(photo)
      xml.lastmod photo.updated_at.strftime('%Y-%m-%d')
    end
  end

  # links page
  xml.url do
    xml.loc links_url
    xml.lastmod @link.updated_at.strftime('%Y-%m-%d')
  end
end

We’re almost done now. First I’d fire up WEBrick and test to see if it works fine and fix any issues. The last step is to create or modify your robots.txt file to specify the location of the sitemap.xml file.

To do this, simply add a line like the following to the bottom of your public/robots.txt file:

Sitemap: http://example.com/sitemap.xml

where example.com is your domain name.

Caching your sitemap

If you’re running a small site then you probably don’t need to worry about this. However, on a large site each request to your sitemap could end up pulling a large amount of data from your database so you may wish to cache it to speed things up.

On my sitemap I use page caching, to enable this you simply need to add the following line to the top of your controller:

caches_page :show

This will make Rails save the page to the disk when it is generated, so Rails isn’t even involved when a client requests the page for the second time.

However, the final thing you need to do is to expire the sitemap when something changes. To do this you need to simply add the following snippet of code:

expire_page :controller => :sitemap, :action => :show

You might also want to look into sweepers to avoid copying and pasting that snippet of code around everywhere in a complex application, but if you’re just running a simple blog or personal site the code above will probably be sufficient.

Submitting your sitemap to search engines

Once you’ve done, you’ll probably want to submit your sitemap to search engines. Generally this happens automatically, but some of them provide tools to see how often they look at your sitemap or if there are any problems with it.

For Google, you can do this with the webmaster tools. The official sitemaps website also has more information about this.

Running nginx in front of Apache

2009-08-17T00:00:00+00:00

Introduction

I wrote this article a while ago when my website was using much more resource-hungry software (Drupal). Now I just run Apache allowing it to serve static and dynamic content and everything is fine. I’ve left it here in the hope that some people might find it useful.

The aim of this guide is to set up nginx to serve static content, something which it is good at, and then forward dynamic requests on to Apache to deal with. Maybe you have some legacy Apache config your application needs (e.g. lots of .htaccess files) or perhaps you just can’t get it to work with nginx. It also seemed to provide an improvement in performance once I had tuned the Apache settings so it expected less simultaneous requests (the vast majority of the requests on my site were actually for static content).

This tutorial was written for Ubuntu but I see no reason why it shouldn’t work for Debian. It could probably be adapted to other distributions as well.

Installation

I already had Apache installed, however, if you are doing this from scratch you can easily install it like so:

sudo apt-get install apache2

Installing nginx can be done in a similar way:

sudo apt-get install nginx

You will also need mod_rpaf later on in the tutorial and it can be installed like so:

sudo apt-get install libapache2-mod-rpaf

Configuration

The first thing I did was to change the port nginx listens on. This can be done in the /etc/nginx/nginx.conf file. I simply change the listen 80 line to listen 81. Later on we’ll swap over so nginx is on port 80 and Apache is on port 81. This is just so I could keep things running in the production environment.

I added an upstream server for Apache. In the http block I added this upstream block:

upstream apache {
    server 127.0.0.1:80;
}

In the server block I added some settings to expire static content in the future to stop browsers re-requesting it. You could also add additional file types which you want to be static:

location ~ (js|css|gif|jpg|jpeg|png|htm|html|txt|rar|zip|exe|tgz|tar.gz|tar|gz)$ {
    expires 30d;
}

Following the first location block, I added another to proxy anything else to Apache:

location / {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://apache;
    proxy_redirect default;
}

The final step is to configure mod_rpaf so that it knows the real client’s IP address. If it is not already enabled, the following command can enable it for you:

sudo a2enmod rpaf

Then to configure it I put this in my /etc/apache2/httpd.conf file:

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
RPAFheader X-Forwarded-For

Testing

Once everything is configured, I suggest rebooting Apache and nginx and testing your new configuration extensively. Remember to use port 81 right now - we will swap 81 and 80 around later.

Finishing up

To finish off I changed Apache to listen on 81 and nginx to listen on 80. Apache’s port can be changed in /etc/apache2/ports.conf. You might also need to change any virtual hosts which you made. nginx’s port can be changed as shown earlier in the tutorial. Once you are sure everything is ready, just reboot them both and your new combination of Apache and nginx will be used for all of your users!

What now?

Further testing is a very good idea. There are likely to be several things requiring changes.
Tune the Apache and nginx settings to improve performance.
If you use virtual hosts, you will need to set up configuration which mirrors Apache’s configuration on nginx so the static content is served correctly.