Fix for the ClearType pixel junk in GVIM

Update: This doesn't work for fonts that use the right-most column. In fact the issue seems to be cleartype writing outside the leftmost pixel in the bounding box. :-(

Here's a fix for this super-annoying bug (Lucida Console 11, insert spaces before the C to leave a trail of cleartype pixels).

===================================================================
--- gui_w32.c   (revision 1289)
+++ gui_w32.c   (working copy)
@@ -2234,6 +2234,8 @@
         * Note: FillRect() excludes right and bottom of rectangle.
         */
        rc.left = FILL_X(col);
+       if (rc.left > 0)
+           --rc.left;
        rc.top = FILL_Y(row);
 #ifdef FEAT_MBYTE
        if (has_mbyte)

Querying Wow64 registry keys from a 64-bit Python

KEY_WOW64_32KEY gets the job done

import _winreg

key = None
KEY_WOW64_32KEY = 0x0200

try:
  key = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, \
    r"SOFTWARE\Microsoft\VisualStudio\9.0", 0, \
    _winreg.KEY_READ | KEY_WOW64_32KEY)
  val, type = _winreg.QueryValueEx(key, "InstallDir")
  print val
except EnvironmentError, ex:
  print ex
finally:
  if key:
    _winreg.CloseKey(key)

Hardware Inventory

Turning the basement upside down yielded

• 2 Commodore 64s (old style breadbox, one white and one brown)
• 2 1541 old-school drives (brown enclosure)
• A Commodore 128
• A 1541-II drive
• An Amiga 500 w/ old-school slow 512 kb expansion memory
• TrumpCard SCSI sidecar enclosure for A500
• External memory expansion (2 MB) for A500
• Power adapters for most of above
• Extra floppy drives for Amigas
• An Amiga 1200 vanilla
• A HP-UX workstation box of unknown origin
• A 680x0-based SUN box
• Two SGI Octanes
• External CD-ROM SCSI enclosures for the SGIs
• And a shitload of cables and PSUs

Continuing upstairs:

• One dual Opteron server (disconnected)
• One Athlon-based generic Linux box (serving)
• One desktop box
• Another desktop box from el cheapo gear (dell)
• A laptop
• A fanless mini-server

Update: New addition: Amiga 1200 w/ 060 and an 14" CRT.

hash_map band aid

Until TR1 is widely available, I'm using this wrapper to construct hash maps using the hash_maps distributed with Visual C++ and GCC. I thought it might be useful for someone else, so here goes.

#if defined(_MSC_VER)
#include <hash_map>
#elif defined(__GNUC__)
#include <ext/hash_map>
#endif

template <typename K> struct default_hash {};

template <> struct default_hash<int> {
 size_t operator()(int v) const
    { return size_t(v); }
};

template <> struct default_hash<unsigned int> {
 size_t operator()(unsigned int v) const
    { return size_t(v); }
};

#if defined(_MSC_VER)

template <
 typename KeyType,
 typename ValueType,
 class HashFun = default_hash<KeyType>
 >
struct HashMapOf
{
 struct Traits
 {
  static const size_t bucket_size = 4;
  static const size_t min_buckets = 8;

  HashFun hasher;

  inline size_t operator()
    (const KeyType& key) const
  {
   return hasher(key);
  }

  inline bool operator()
     (const KeyType& lhs,
      const KeyType& rhs) const
  {
   return lhs < rhs;
  }
 };


 typedef stdext::hash_map<
    KeyType,
    ValueType,
    Traits> Type;
};

#elif defined(__GNUC__)

template <
 typename KeyType,
 typename ValueType,
 class HashFun = default_hash<KeyType>
 >
struct HashMapOf
{
 typedef __gnu_cxx::hash_map<
    KeyType,
    ValueType,
    HashFun> Type;
};

#else
# error Unsupported toolchain.
#endif

Vista and Visual Studio, Oil and Water

So I've installed Vista (Ultimate, in fact), but Visual Studio 2005 seems like a terrible Vista citizen. In order to run it you need to apply the service pack, but also a beta version of a hotfix package. Even then, Microsoft recommends you run Visual Studio as Administrator. Now, I ask myself: Does compiling source code demand complete access to the entire box? On Vista, that's apparently the case. It seemed to work fine on XP. Now to the most juicy bug of the day. Once you've started a debugging session from within Visual Studio, the target EXE file becomes locked. Closing Visual Studio removes the lock, but the devenv process doesn't hold any handles to the file. What gives? I'm starting to suspect that the SUA services might have something to do with this, so they're uninstalling right now. It's pretty annoying to restart the IDE after each debugging session to be able to recompile. Update I've identified SUA to be the problem. After uninstalling the SDK and removing the windows services it installs, I can recompile and debug again.. For now.

The pleasures of a stable ABI

As broken as the Win32 API might appear, the thing I really like about it is its stable ABI. That, and the fact that Microsoft spend thousands of hours making sure that old programs work when they release a new version of Windows is worth a lot to me. Solaris has a decent ABI, and I assume most other unices have too. What would it take for Linux to achieve the same level of stability? The Linux kernel does have a fairly stable ABI as long as you say away from the proc file system and device nodes (devfs and whatnot come and go). The C library is also well designed. Most issues with Linux binary (in)compatibility have been in the C++ libraries. The C++ libraries have changed ABIs with each and every version of the compiler, sometimes even breaking with point releases. Breaking the C++ ABI isn't a problem in itself, it happens with the Visual Studio compilers all the time. The real problem is that when deployed, Linux binaries depend on the distribution to include a suitable C++ runtime library to which they can dynamically link, whereas on Windows, programs either carry their C++ runtime with them (pre-VC8) or rely on a separate distributable that can be installed side-by-side in the DLL cache so that a compatible version is always available. For the Battlefield Linux server builds I had this exact problem. I had to use at least GCC 3.x as GCC 2.x was (and still is) a severely outdated compiler without proper namespace support. However, using GCC 3.x meant that people with rented servers that ran older versions of Red Hat were left out in the cold, unless they could find a standard C++ library that would somehow let them run the binary! This was seen as a Bad Thing. The only solution I could think of at the time was to provide a statically linked executable in addition to the regular executable, so that older systems had a hope in hell to run it. That required me to tweak and override certain symbols of the standard library to avoid them from referencing versioned symbols from glibc that weren't present on the target systems, and the whole things was just a terrible mess that took too much time. In the end, linking statically turned out to be somewhat illegal (as we couldn't satisfy the LGPL on that binary) so that option had to be removed. I considered it fair use because there was indeed a dynamically linked executable available as well that did satisfy the LGPL, but it was thought safer to drop it. It would have been so much nicer to ship a C++ library redistributable for x86 and amd64 and let users install it if needed..

ODE, .NET, handles and references from hell

I've recently toyed around with .NET in C# and I was excited to see that there are .NET bindings for a lot of free game development libraries in the Tao framework. I was especially interested in playing with the ODE bindings. ODE is a free physics and collision library that is starting to become usable for a lot of things. As with many .NET wrappers, the ODE wrapper is just a set of static functions that take System.IntPtr objects. You are supposed to juggle these handles carefully, because they correspond to some real unmanaged object. This is all good, but these APIs break down when the unmanaged objects form graphs on their own, which is the case with the ODE API. Assume we have a wrapper for the following API: public sealed class SomeApi {  static IntPtr MakeContainer();  static void DestroyContainer(IntPtr i);  static IntPtr MakeItem();  static void DestroyItem(IntPtr i);  static void AddItem(IntPtr container, IntPtr item); // Many other functions. } Further, assume that the semantics of a "Container" in the API is that it destroys its contents when it is destroyed. Also assume that we've written our wrapper classes for Container and Item so that they correctly call the corresponding cleanup function in the API when objects are Disposed and finalized. With the prerequisites out of the way, consider this code fragment: Item i = new Item(); Container c = new Container(); c.AddItem(i); What is happening here is that the unmanaged code is forming a reference graph behind the covers, but the available tools for dealing with unmanaged resources can't see such graphs. The .NET process will crash because regardless of whether the item or the container is destroyed/finalized first, the other one still refers to a (now deleted) unmanaged object. Tao's ODE wrapper exhibits these exact problems, making it very hard to reason about code correctness. Essentially I'm finding it much harder to use in C# than in C because not only do I have to deal with my own code, I also have to consider how ODE maintains memory internally at every API invocation and what that does to the validness of my wrapper objects. The only solution I can see is to build wrapper APIs that explicitly maintain the life-time of handles as well. In the above example, such a wrapper would detect that the "AddItem" operation changes the cleanup origin of the item and that it shouldn't be destroyed when the Item wrapper is destroyed, because the container is now doing that. Sigh.

Non-ignorable return codes

I'd like to have a nonignorable-type that could be used to wrap return values that must be handled, but using an exception would be to heavy-weight. Consider: bool IntersectLineAndPlane(...); I'd like to say nonignorable<bool> IntersectLineAndPlane(...); And then have code that doesn't handle the return value give a compile-time error (or warning) so that mistakes don't slip through into production code. You would think that perhaps something like this would work:

template <typename T>
class nonignorable {
  public:
      nonignorable(const T& v) : value_(v) {}

      // check() does the destruction and extracts the value
      friend T check(const nonignorable<T>&);

 private:
     ~nonignorable();
};

This could be used as:

nonignorable<bool> Function() {
 return nonignorable(false);
}

And in client code: if (!check(Function)) { /* take action */ } Alas, this will not work, because there are more destructions going on when the nonignorable instance is returned on the stack. I think the only way to solve this is to rely on the new C++0x proposition to add move semantics into the language so that there is only one destruction (in the check() template function).

Mixing Scheme and C

In my quest to find the perfect Scheme/C mix I've found that Bigloo Scheme really goes a long way to overcome the language barriers. The cool part is that Bigloo compiles to efficient C code, and you can influence the generated code in a number of ways. For example, it's possible to include headers and declare functions with type annotations straight in the Scheme source code. Since the Scheme files are translated to C when compiled, there's no marshalling to consider. I'm assuming that annotated procedures yield a very low overhead compared to Python which requires a object allocations for everything (even integers). Using Bigloo with DLLs symbols on windows has one quirk though, the symbols must be imported as macros, and the header file with _declspec() crud must be included explicitly. This does the trick:

(module my-module
        (extern
          (include "include/mydll.h")
          (macro simple::int (::int) "simple")))

Given these declarations, you can call the function simple as if it were a Scheme procedure, which is awesome. When compiled, the resulting executable links properly to the DLL. I haven't found a tool that can produce these annotations automatically, but writing one should be trivial. I'm slightly confused about how to make a stringent build system for a combined Scheme/C project, but I'm sure it's possible. Armed with such a build tool it might actually become fun to program again..

Reducing boilerplate by embedding XSLT

I've recently been working on an custom IDL compiler that can target Python and C++. Think COM, only portable between platforms and targeted towards a select few languages. A lot of patterns immediately appear in both the frontend and backends of the compiler that's just boilerplate work. Consider the classic implementation of an IDL parser (or any programming language parser, for that matter):

1. Parse the input, building an abstract syntax tree of the declarations.
2. Build a type graph and interconnect the type nodes so that types can be resolved.
3. Resolve all type references (using fixups or in a separate pass if forward references allowed.)
4. Invoke one or more backends if the resulting tree is valid.

The backends in turn will walk the tree (via some variation of the Visitor pattern) and produce output. This quickly becomes repetitive and mundane work. Visitor is well known for it's strong impact on code coupling and strong binding between entities. In another sandbox project of mine I tried out a completely different approach. Instead of writing yet another AST hierarchy to interface the frontend I experimented with XML as an intermediate format in the compiler itself. Given an input file, my LALR parser will essentially emit a well-formed XML document using the DOM APIs. This in itself doesn't buy much, but it provides tremendous flexibility down the road. Think of it as capturing the entire AST in a portable format. Indeed, the GCC project has recently added a AST-to-XML converter as part of their (excellent) compiler frontend so more people are thinking along these lines. With the AST swimming around in an XML DOM tree, it becomes possible to rewrite repetitive and boring tasks in XML query tools that are designed for those jobs rather than a general-purpose programming language such as C++. Whipping up flexible type resolvers, type and syntax checkers using a decent XSLT processor is a lot of fun, and it can be done with a surprisingly small amount of XSLT code. XQuery processors can easily build statistics and metrics with a minimal amount of code. Another interesting thing about keeping the AST in XML is that it becomes trivial to add whatever meta-data you like in a matter of seconds. That data can travel through the filters (for example in an XML namespace of its own) all the way to the backend, where it can be used to tweak the output. Also, changing the XML AST doesn't require recompiling hundreds of AST classes! The key concept to making this work is pipelining, which is a common design pattern anyway in compilers. Connecting the output of one XSLT processing filter to the input of the next nicely captures the pipelining pattern while at the same time enforcing the idea that one piece of code should have a single, well-defined purpose. XSLT also works wonders for code generation backends, because it designed for transforming data, something C++, Java and Python simply aren't. So what are the drawbacks? Simply put: speed and memory consumption. Clearly, a hand-written implementation of say a type resolver will have a lower memory consumption and better processing speed than a XSLT-based implementation. Also, while XSLT is a very useful language (especially with version 2) it's not very strong with error reporting or string handling, to name a few areas. If you're willing to strike a deal with the portability devil and settle on a single XSLT library it's possible to work around most of these problems. Given the prototyping advantages I still think this is a viable implementation strategy for compiler projects.