SWIG-1.3.12, SWIG-2.0 and Beyond

With the release of SWIG-1.3.12, I thought I'd take a few moments of everyone's time to talk about the past, the present, and the future of SWIG development. I'm really quite excited about the current release because I think it represents a huge turning point in SWIG's development. Furthermore, it is only the beginning of bigger and better things to come. However, we definitely need your help.

To put a little perspective on the discussion, I'd start with a few development statistics. In the last 12 months, there have been over 300 entries added to the CHANGES log and over 4000 CVS commits. Although that may not sound like a lot compared to a huge software project, it is significant in the context of SWIG. As a point of comparison, there has been more SWIG development this year than in any other year of the project and more than in the previous three years combined. This even includes the first few years of development in which there was also a lot of activity. Furthermore, many of the recent changes have been extremely non-trivial (e.g., templates, namespaces, type system, operators, etc.). As a result, SWIG is more capable than I ever imagined possible.

Regrettably, I must admit that I've been a little negligent in discussing the roadmap for where I thought this flurry of SWIG development was actually headed. In part, this is because I've been buried in work. However, the real reason is that I didn't really know where we were going---except that in a time far far far away in the future, we might arrive at some kind of "stable" release with a version number other than "1.3.x". Needless to say, that's not a very compelling story.

That said, I've spent a lot of time thinking about SWIG and trying to wrap my brain around it. Specifically, just what is (or should be) the primary focus of this project and what are we really trying to do? That's what the rest of this message is about.

SWIG Prehistory

The first version of SWIG was written in 1995. The original system was developed to help with some software wrapping problems I encountered while writing molecular dynamics software at Los Alamos. Later that year, I became interested in extending the wrapper generator to support other scripting languages so it was rewritten in C++ and modified with multiple backends (Tcl, Perl, and Guile). This led to a first public release in February, 1996. Feedback from this release led to a series of enhancements and the release of SWIG 1.0 in September 1996. Throughout this process, my intent was to create a tool that I would want to use---I never viewed the project as an CS experiment in programming languages or software engineering.

SWIG 1.1

SWIG-1.1 (June, 1997) represented a series of enhancements that were added in response to feedback at conferences and from users. Shadow classes, exception handling, typemaps, and a number of more useful features were added. However, the overall structure of the system was relatively unchanged from the initial version. Following the release of 1.1, a series of minor patch releases were made. This resulted in the release of SWIG-1.1p5 in February, 1998. Unfortunately, this release would remain the last "stable" release for quite a long time---in fact, it is still listed as the last "stable" release on the SWIG web page!

SWIG Hell

Even during the development of SWIG-1.1, it was clear that the whole design of the system was deeply flawed. The implementation was a mess and the C/C++ support was full of holes and nasty corner cases. Furthermore, there was no unifying principle that tied all of the different SWIG directives together. Not only that, fixing these problems appeared to be nothing short of impossible---requiring a total ground-up rewrite at best. The only redeeming quality was that the system basically worked "well enough," it was extensively documented, and its flaws mostly known. People could use it and there were work-arounds for most common problems.

To deal with the design problem, there were at least four attempts to completely rewrite SWIG, some of which were attempted in parallel with the work on SWIG-1.1. Unfortunately, none of these were ever completed. The primary problem was a strong "second system" effect and a desire to make SWIG do everything that one might conceivably want to do with a wrapper generator (somehow). Clearly, this was a recipe for disaster. In fact, all such attempts to rewrite SWIG were eventually abandoned several years ago. In hindsight, I think the real problem was that these rewrite efforts focused far too much attention on implementation technique rather than principles. In short, the failure of these efforts was due to a lack of clarity in understanding how SWIG ought to work (regardless of how it was actually implemented).

SWIG Restart (1.3a1-1.3a5)

Having languished for several years, the SWIG1.1p5 release had a growing pile of maintenance issues. It didn't work for newer versions of certain language modules and a lot of minor bug reports and feature requests had been building up. With a lot of help from Loic Dachary and Thien-Thi Nguyen, we put together the 1.3a1 release (February, 2000). This was mostly a bug fix release to 1.1p5 except that the preprocessor module from SWIG1.2 was added and a lot of minor enhancements were added.

For the next six months, a massive effort was made to rewrite all of SWIG's internal data structures (strings, lists, hashes, etc.). This work was all going on underneath the covers while we tried to keep SWIG in an operational state. The primary focus of this work was really one of cleanup. Having given up on a total rewrite, perhaps we could settle with making the implementation incrementally better than before. In addition this, Matthias Koppe jumped on board to reimplement the Guile module and to make other improvements to the system.

An important aspect of these releases was that many parts of the system not directly related to wrapper code generation were removed. This included the documentation system and Objective-C support. These were not removed because they weren't useful. Instead, the documentation system was removed because it accounted for nearly half of the special SWIG directives, yet had no direct bearing on what SWIG actually did. Objective-C support was removed because it was tangled with C++ support in a way that was very difficult to understand and maintain. The removal of these features was seen as a way to vastly simplify cleanup--and to buy some time where we could rethink their role in a future release.

SWIG Redux (1.3.6-1.3.11)

This work, started in February 2001, is the beginning of the current SWIG implementation. With the help of William Fulton, Matthias Koppe, Lyle Johnson, Luigi Ballabio, Jason Stewart, Richard Palmer, Sam Liddicott, and others, this work can best be described as the wholesale destruction of everything remaining from SWIG-1.1. The language module API, type system, the parser, numerous SWIG directives, and SWIG library files---all destroyed or rewritten. Not only that, we started introducing significant incompatibilities with SWIG-1.1---mostly in an effort to correct past wrongs and get ourselves out of the tangled mess of earlier versions. A huge number of long-standing bugs and difficult feature requests have also been resolved.

The general goal of this development could best be described as an attempt to reduce SWIG to an easily described set of general "ideas" about how it should operate. Special SWIG directives have been eliminated or combined with others. Different parts of the system have been better integrated. Features not directly related to wrapper code generation have been removed and the system has become more focused. Changes in internal data structures and APIs have allowed SWIG to capture more information from interface files and to resolve hard corner cases. More often than not, these are things that you never notice unless you are an old user and you suddenly realize that a problem you had several years back has disappeared.

Along with the destruction of old code, this work has quietly introduced a new core--the most significant features of which are a new C++ type system and multi-pass compilation. More importantly, this work has really tried to provide a more principled foundation for future SWIG development. However, just what is this "more principled foundation?"

Convergence (1.3.12)

With the upcoming 1.3.12 release, SWIG is about to take a big leap forward. Almost all of this is due to one realization---that almost every hard problem in past SWIG releases has been directly related to errors and limitations in its type system. Types are the key to understanding the structure of C and C++ code. They are at the heart of understanding advanced language features like namespaces, nested classes, and templates. They are directly related to the data marshalling that occurs in wrappers. Not only that, they interact with nearly every SWIG directive. A proper type system *is* the necessary foundation for moving SWIG forward.

To be honest, I don't think that the emphasis on types is entirely obvious. In fact, a great deal of work in past SWIG rewrites has focused on the problem of C++ parsing. For instance, modifying the parser to handle more advanced C++ code or representing parse trees as XML. Furthermore, if one looks at the SWIG mailing list, you can find a *lot* of messages related to issues of C++ parsing whereas almost no one ever talks about types (well, other than typemaps). Even other wrapper generation tools seems to spend a lot of time dealing with the parsing issue. Although parsing is clearly important, I don't think it has ever been the real problem in SWIG. This is because even though a parser can tell you what's in a header file, it doesn't tell you anything about how the different pieces of the system behave or how they might interact. To do that, you need to do a lot more than just parsing--and that's really the whole point.

Although earlier 1.3 releases have made big improvements to the type system, SWIG-1.3.12 is the first release that really tries to tackle the type-system issue in a major way. We have patched nearly all remaining holes in the type system and we have added full support for C++ namespaces. Not only that, we have completely reimplemented C++ template support in a way that supports templates, member templates, and template specialization. Luigi and I are currently using this to work on proper SWIG library support for parts of the C++ standard library and the Standard Template Library (STL). Although some crusty C programmers (present company excepted), might balk at such apparent insanity, this work has impacted all parts of SWIG at all levels. Even a variety of subtle errors in C support have been fixed by this work.

In addition to the type system work, SWIG-1.3.12 contains continued reduction in the implementation. Directives have been removed, refined, renamed, or consolidated. We're still narrowing the focus of the system and working towards some kind of convergence. "Convergence to what?!?", you ask.

So, what is SWIG?

In a nutshell, SWIG is a C/C++ declaration compiler that generates wrapper code (okay, so you knew that much). However, to really understand what SWIG is doing and where SWIG-1.3.x is headed, it is useful to know that the whole system is essentially being built around three extensions to the C++ type system:

Typemaps. Typemaps are rules that define the process by which data is converted between languages. They are fully integrated with the C++ type system and they are applied using a type-based pattern matching mechanism. All data conversion SWIG is defined by typemaps and is fully reconfigurable.
Declaration annotation. There are special directives that modify the wrapping behavior of individual declarations. Declarations can be selectively identified and decorated with arbitrary attributes that affect wrapper generation. Like typemaps, declaration matching is fully integrated with the C++ type system. Almost all SWIG customization directives are a form of declaration annotation.
Class extension. The ability to extend classes and structures with new methods/attributes when building wrappers. Classes are part of the type system so class extension is naturally integrated with the C++ type system as well (big surprise).

And that's it--this is the end-game of SWIG-1.3.x development. When stabilized and documented it will become SWIG-2.0.

The Bigger Picture

I really want to emphasize that all of this work is part of a much bigger picture. SWIG is used by a surprising number of people in industry, government, and academia. It's used to build systems used by millions of people every day. It has even shown up in video games and other unlikely settings. Although SWIG doesn't have the same visibility as many large software projects, over 12000 people have downloaded SWIG-1.3.11 in the last 4 months. Clearly someone is using it for something! Because of this, I think it is important for us to work on moving SWIG towards a more solid foundation. Doing so will give the system more credibility and long term viability---and it will be a lot more fun to use!

It's also worth noting that there are some rather interesting CS connections at work here. Extensions to the type system and typemaps have some interesting relations to work in programming languages. The SWIG declaration annotation system and class extension feature seem oddly similar to work in the emerging area of Aspect Oriented Programming (AOP). There are undoubtedly connections to other areas of software engineering and architecture.

The key point is that SWIG isn't going to connect to anything if no-one can quite describe what it is or how it works.

SWIG-2.0 and the Future

SWIG-1.3.12 still represents work in progress. There are bugs, the documentation is still incomplete, and there are parts of the implementation that are rather ugly. We are also still working out a few very hard problems like nested classes, callback functions, and overloading. A few old features are still missing (Objective-C, documentation). However, I believe the end of the 1.3.x series is near and achievable.

Over the summer, a few more 1.3.x releases may appear but the current plan is to aim for a SWIG-2.0 release in September. This release is really moving towards the design principles described above and will be a very major enhancement over SWIG-1.1.

As for the future, a number of interesting ideas are on the table. I want to add support for contracts/assertions in order to solve some reliability issues that arise when retrofitting legacy codes with a scripting interface. Support for an extension language has been promoted by David Fletcher and was suggested by someone else on the mailing list rather recently. I have a graduate student working on SWIG support for the Microsoft CLR and .NET languages. Other work might include support for alternative parsers, dynamically loadable language modules, and so forth. However, none of this is really going to materialize if we can't get the 2.0 release stablized. In fact, I see SWIG-2.0 as a necessary step for moving forward with these ideas.

We need your help! Yes, you.

Nobody gets paid to work on SWIG. The developers are volunteers who work in their spare time. Furthermore, SWIG is not supported by investors, a large corporation, or research grants. I work on it because it's fun, challenging, and useful. I presume that other developers do the same. However, we only have limited resources and we need your help.

If you like SWIG and find it useful, we need you to try new versions. We want you to torture test the releases and to break them. We need bug reports. No bug is too obscure or unimportant---we *will* fix it if we can. We also need feedback about things that are annoying or compatibility features that might help in going from 1.1 to 2.0.
We need help with documentation, examples, testing, libraries, and all sorts of other aspects of the release. Even if you have never written a SWIG language module or dived into its implementation, there are many ways that you can help. Consider writing a case study about how you wrapped a big library. Contribute tests that break the implementation in horrible ways. Help with the web page or FAQ.
Most of the SWIG-1.3.x work has focused on the SWIG core. However, as the 2.0 release nears, we will be working on a variety of enhancements to the language modules. If there are things you would like to see in any of the language modules, you need to let us know.
There are SWIG language modules that have not made it into the distribution. Examples that I know about include ITCL, Swig-Eiffel, and Swig-Lua. We would gladly make these part of the standard SWIG distribution. However, we also need help to do it. Porting from SWIG-1.1 is no easy task, but we're more than willing to help. It's not as bad as one might imagine.
We are always looking for developers. Subscribe to to the swig-devel mailing list or send me email to get involved.

Acknowledgements

I'd just like to thank everyone who has submitted feedback, bugs, made contributions, and put up with my occasional thrashing over the years. I welcome any comments about this document and how we can make SWIG even better.

Dave Beazley June 2, 2002

Feedback and questions concerning this site should be posted to the swig-devel mailing list.

Last modified : Sat May 13 14:25:49 2017