Ffidl allows you to define Tcl/Tk extensions using pure Tcl wrappers and any shared library installed on your system, including the Tcl and Tk core libraries.

Ffidl uses libffi-1.20, with patches, to dynamically construct calls to C functions from Tcl, and the dlopen() and dlsym() interface to load dynamic libraries and discover the locations of functions. Ffidl was developed under linux-2.0.32 using Tcl8.2.1 and should be able to run on any system with a stubs enabled Tcl, libffi support, and a libdl implementation. Libffi has currently been ported to:

• SunOS 4.1.3 & Solaris 2.x (Sparc v8)
• Irix 5.3 & 6.2 (System V/o32 & n32)
• Intel x86 - Linux (System V ABI)
• Alpha - Linux and OSF/1
• m68k - Linux (System V ABI)
• PowerPC - Linux (System V ABI)
• ARM - Linux (System V ABI)

Ffidl 0.1 is an alpha release. There are a few configuration details which you will need to attend to by hand in the current release. The initial development turned up two bugs in libffi-1.20 under linux-x86, so users on other architectures should be alert for similar problems. There are several open design issues still to be resolved.

I'm releasing at this early point because it's really too much fun to keep to myself, I need assistance verifying that the implementation works on all the architectures supported by libffi, and the open design issues could use some discussion.

Commands, Functions, and Procs

Ffidl defines four commands in the Ffidl package: ffidl-proc, ffidl-symbol, ffidl-typedef, and ffidl-info; exports one function from the Ffidl shared library: ffidl_pointer_pun; and defines two essential helper procs in the Ffidlrt package: ffidl-find-lib, and ffidl-find-type, which are currently just stubs of their true form.

These interfaces should be considered subject to revision.

ffidl-proc name {?arg_type1 ...?} return_type address

ffidl-proc defines a Tcl command with the specified name which, when invoked, converts its arguments according to the arg_types specified, calls the function at the specified address, and converts the specified return_type into a Tcl result. The allowed types are described below.

ffidl-symbol library symbol

ffidl-symbol loads, if necessary, a dynamically linked library of name library and fetches the loaded address of symbol from the library.

ffidl-typedef name type1 ?...?

ffidl-typedef defines a new ffidl type name. This may be either a simple alias for an existing type, or a list of types which form a structured aggregate. To pass a structure by value or return a structure by value, you must make a ffidl-typedef for it. But even if you only pass or receive structures by reference, you might want to define a structure in order to use the format, sizeof, and alignof options of ffidl-info on it.

ffidl-info option ?...?

ffidl-info implements a variety of information functions.

ffidl-info typedefs

returns a list of ffidl-typedef defined names.

ffidl-info procs

returns a list of ffidl-proc defined names.

ffidl-info signatures

returns the list of function call signatures used by ffidl-proc.

ffidl-info libraries

returns the list of libraries opened by ffidl-symbol.

ffidl-info alignof type

returns the alignment modulus for type.

ffidl-info sizeof type

returns the size of type.

ffidl-info format type

returns a format string for type, in the style of the Tcl binary format and binary scan commands, using the correct endian format for integers and, for structures, including any pad bytes required for alignment of fields.

ffidl-info interp

returns the current Tcl_Interp as an integer value.

EXTERN void *ffidl_pointer_pun(void *pointer);

ffidl_pointer_pun is exported from the ffidl shared lib to allow conversions between pointer representations to be coded as ffidl bindings. There are some examples in ffidlrt.tcl.

ffidl-find-lib library

ffidl-find-lib converts a conventional name for a library into the path name for the library name appropriate to the host system. It is currently implemented in ffidlrt.tcl as a table lookup which returns the libraries appropriate to my Linux system.

ffidl-find-type type

ffidl-find-type converts a standard types such as size_t and time_t into real types appropriate to the host system. It is currently implemented in ffidlrt.tcl as a table lookup which returns the types appropriate to my Linux system.

Types

The ffidl builtin types include the scalar C types in both their unsized forms and as explicitly bit sized types, and a variety of pointer treatments. Note that some types are only valid in certain contexts: arguments (arg), return (ret), or struct elements (elt).

In addition to the builtin types, the ffidl-typedef command may be used to define new types. Aliases for existing types may be used where ever the existing type may be used. Structured aggregates may be used as arguments, returns, or elements of other structures.

arg	ret	elt	type	definition
-	+	-	void	void
+	+	+	int	int
+	+	+	unsigned	unsigned int
+	+	+	short	signed short int
+	+	+	unsigned short	unsigned short int
+	+	+	long	signed long int
+	+	+	unsigned long	unsigned long int
+	+	+	float	float
+	+	+	double	double
+	+	+	long double	long double
+	+	+	sint8	signed 8 bit int
+	+	+	uint8	unsigned 8 bit int
+	+	+	sint16	signed 16 bit int
+	+	+	uint16	unsigned 16 bit int
+	+	+	sint32	signed 32 bit int
+	+	+	uint32	unsigned 32 bit int
+	+	+	sint64	signed 64 bit int
+	+	+	uint64	unsigned 64 bit int
+	+	+	pointer	pointer as an integer value
+	-	-	pointer-byte	pointer from ByteArray
+	+	-	pointer-obj	pointer from Tcl_Obj
+	+	-	pointer-utf8	pointer from String
+	+	-	pointer-utf16	pointer from Unicode
+	-	-	pointer-var	pointer from ByteArray stored in variable. If the ByteArray is shared, then an unshared copy is made and stored back into the variable.

Installation

Installation consists of:

tar xzvf ffidl-0.1.tar.gz
cd ffidl-0.1
(cd libffi-1.20 && configure && make)
configure && make

(cd libffi-1.20 && make test)

At this point you should edit demos/ffidlrt.tcl and look at the table of libraries in ::ffidlrt::libs and the table of types in ::ffidlrt::types. Either or both of these tables will probably need attention if you go further. You may need, for instance, to build and install libraries for gmp and gdbm, or to adjust the pathnames for libc, libm, tcl, and tk.

Running

make test

The configure for Ffidl started as a TEA configuration, but it needs more work. It doesn't configure and build libffi automatically. It doesn't test that libdl is actually available. It builds the ffidl_test library and runs the test scripts in demos/, but it wasn't clear how Makefile.in was supposed to support this or cleaning up test binaries.

The included copy of libffi-1.20 has been patched. The distributed libffi-1.20 erroneously makes "long int" a 64 bit integer on x86, and doesn't return unsigned long long. If you have installed a copy of libffi on your machine, beware of finding the unpatched headers in /usr/local/include before the patched ones. The patch, which has already been applied to the included sources, is included as libffi-1.20.patch.

Demos

The demos directory contains several small and medium size examples of ffidl bindings to shared libraries, and some code for making comparisons to other ways of doing the same thing.

Makefile	for building mathswig.so
atol.tcl	a ffidl binding to atol()
ffidlrt.tcl	run time support for ffidl bindings
gdbm.tcl	a ffidl binding to gdbm-1.8
getrusage.tcl	a ffidl binding to getrusage()
gmp.tcl	a ffidl binding to gmp-2.0.2
gmpz.tcl	arbitrary precision integers via gmp.tcl
gmpq.tcl	arbitrary precision rationals via gmp.tcl
gmpf.tcl	arbitrary precision floats via gmp.tcl
libm.tcl	a ffidl binding to libm
mathswig.i	a SWIG binding to libm
tkphoto.tcl	a ffidl binding to the Tk photo image.
pkgIndex.tcl	hand built package index
test-ffidl.tcl	a test of ffidl using ffidl_test.c
test-gdbm-1.tcl	a test of the gdbm binding
test-gdbm-2.tcl	a test of the gdbm binding
test-gmpz.tcl	a test of the gmpz routines
test-libm.tcl	a test of the libm binding
time-libm.tcl	a timing comparison of ffidl and expr
test-tkphoto.tcl	a short example of tkphoto.tcl usage

ffidlrt.tcl will need attention unless you're running on my machine. There are two functions, ffidl-find-lib and ffidl-find-type, which abstract library names and system typedefs out of the rest of the code. However, the abstraction is currently limited to the correct results for my Linux box. You'll need to rewrite the mapping for your own machine.

ffidlrt.tcl contains some examples of binding into the Tcl core itself.

tkphoto.tcl allows extraction and insertion of photo image pixels as binary data. See test-tkphoto.tcl for an example.

The gdbm.tcl extension should be plug compatible with Tclgdbm0.6, a C coded Tcl extension for manipulating gdbm files. Since gdbm passes and returns structures, it also tests the ffidl struct code.

The gmp*.tcl extensions make a nice example. The main Gmp package wraps all the exported mpz_*, mpq_*, and mpf_* entries from the Gnu multiple precision library. The subsidiary Gmp[zqf] packages use the Gmp package to define arbitrary precision integers, rationals, and floats which are represented as strings. This isn't the most efficient way to do arbitrary precision arithmetic, but it is convenient, it does avoid needing to know what type mp_limb_t and mp_size_t actually are, and it does show how to use the underlying library if you want to build something more efficient.

Performance

Performance appears to be excellent, but I can't take any credit because libffi is doing most of the work. The demos/time-libm.tcl script compares ffidl-proc wrapped libm functions to the Tcl expr versions of the same functions. If you have SWIG you can build mathswig.so, and if you're running on Linux-x86 you can install ::dll, and time-libm.tcl will time them on the same functions.

The ffidl bindings to libm run a little faster than ::dll bindings and a little slower than SWIG wrappers, all of them coming in slower than expr itself. The bottom line is that all three extensions run better than 1.5 times the speed of expr over the same functions.

Open Issues

There are many open issues.

Port to windows - simulating libdl appears to be trivial, and according to the source for ::dll the windows ABI for x86 is actually the same as the sysvr3 ABI for x86 already implemented by libffi.

Port to macintosh - no idea what's involved here.

Importing libdl compatability code from Tcl. It would be real if the Tcl core exported some neatly packaged libdl abstraction.

Finding the right library is a pain. dlopen("libm.so") finds libm on my machine, but dlopen("libc.so") returns an error string decorated with binary characters while dlopen("libc.so.6") works. If you work with shared libraries you build yourself, it's not an issue, but for all the standard stuff there is no standard. In demos/ffidlrt.tcl the ffidl-find-lib function provides an abstraction for at least removing these issues one layer away from your ffidl bindings to the library, but the implementation of the abstraction hasn't gone farther than listing where I find my standard libraries.

Discovering what type a type is is a pain. Include headers are typically so heavily conditionalized, that one needs to search and search to find which typedef is actually implemented. In demos/ffidlrt.tcl the ffidl-find-type function abstracts these issues out of the ffidl bindings, but again the implementation of the abstraction will need some work.

A backend for SWIG which generates ffidl bindings might be nice.

There are some more pointer types which ought to be defined: a variant of pointer-var which requires an unshared value; a pointer to a native character string -- but couldn't that be pushed back to the Tcl layer?

Writing Tcl extensions with ffidl is very much like writing C code in Tcl. I'm not sure what the actual required skill set is. But if you're not sure what you're doing, you might be in over your head. In any case, try not to take the core dumps personally.

Loading snippets of code into a Tcl interpreter with ffidl loaded could be very hazardous, as in downloading "Try ME!" scripts from the web. There is no Ffidl_SafeInit(), we'd probably need signed scripts to even begin to consider such a thing.

I've looked at SWIG and at dll and seen that they very carefully duplicate any shared Tcl_Obj before attempting a conversion to Int or Double. I've also looked at the source for Tcl's expr command, and it converts objects to Double or Int and only duplicates shared objects when it finds a valid Int or a Double with an existing string representation. Ffidl only duplicates shared objects when processing pointer-var, though I'm open to explanations why it should do otherwise. It seems that if you pass a parameter to a typed function that you shouldn't be upset if the parameter is converted to that type.

Hmm, this is a really pared to the bone. It would be nice for newbies and experimenters and the careless if ffidl implemented a debugging mode which verified that constraints were observed: 1) that Tcl_Obj string reps were not modified, 2) that Tcl_Obj bytearray reps were not modified outside their allocated sizes, and so on. This could be done by switching in an alternate implementation of tcl_ffidl_call() which made copies and verified the constraints after the call.

Some naming consistency in the demos. I seem to be reinventing my Ffidl extension style each time I start a new example.

Some style consistency in the tests. The tests just run, some generate descriptions, some report what they've done, some say nothing, some give summaries.

Credits

Robin Becker's ::dll package, which does much the same thing as Ffidl, provided the immediate inspiration for this work and pointed to the solution of some of the design issues for me.

Anthony Green's libffi package provided most of the implementation of ffidl.

Copyright, License, & No Warranty

Ffidl Version 0.1, Copyright © 1999 by Roger E Critchlow Jr, Santa Fe, NM, USA

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ``Software''), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL ROGER E CRITCHLOW JR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.