SQL is a great example for this. Relational databases are more useful with strong typing, so EMPLOYEE_ID is incompatible with PRODUCT_ID even if they are both implemented as INT. It'd be a great idea to see those constraints implemented at the perl level, presumably by giving perl more knowledge of the database schema than even the database engine has.

Imagine that you have, or can write, a little language parser for a SQL-like language. My simple example was:

SQL {{
    UPDATE users SET address = { Address $address } WHERE user = { User $user }
}}

This can decompose into several operations:

• Get the value of the $address variable.
• Get the primary key of the $user variable.
• Prepare a database query with a rewritten query string which uses placeholders for the $address and $user variables to avoid SQL injection and other interpolation errors.
• Execute the query.

That's a nice interface, but you can do better. As I suggested, you can add error checking if you know the structure of the database:

• Get the metadata which describes the users table.
• Verify that the required fields (address and user exist).
• Get the value of the $address variable.
• Get the primary key of the $user variable.
• Prepare a database query with a rewritten query string which uses placeholders for the $address and $user variables to avoid SQL injection and other interpolation errors.
• Execute the query.

You can take advantage of type checking too:

• Get the metadata which describes the users table.
• Verify that the required fields (address and user exist).
• Verify that the type of $address is compatible with the type of the address field. Repeat for $user and user.
• Get the value of the $address variable.
• Get the primary key of the $user variable.
• Prepare a database query with a rewritten query string which uses placeholders for the $address and $user variables to avoid SQL injection and other interpolation errors.
• Execute the query.

If you know the structure of the database when the program starts, you can start to push some of this type checking to the point of compilation. (You may not be able to perform all of the type checking at compilation time, but you can do as much as possible as early as possible to prevent as many errors as possible.)

That's simple and easy. Now imagine something more interesting:

SQL {{
    SELECT name, address FROM users, addresses GIVEN { User $user }
}}

It's obvious from the syntax of the query language that the database needs to perform a join operation, and it's obvious that the primary key of the $user object is the important key of the operation. If the program knows the relationship of the users and addresses tables, it can join them effectively as well.

Don't get caught up in the syntax or the semantics of the remainder of examples here; they exist to demonstrate possibilities, not the final form of battle-tested code. Even so, imagine a dynamic query:

SQL {{
    SELECT @fields FROM { Table $table_one }, {Table $table_two } }
}}

Again the structure and intent of the code is obvious. The operations are now:

• Find the primary keys for $table_one and $table_two.
• Verify that they're joinable.
• Verify that all members of @fields are present in either $table_one or $table_two.
• Construct the query.

If I were to implement this, I'd make a join_tables multimethod. It takes two arguments (generalizable to more, but follow along with two for now). Imagine that it looks something like this:

multi join_tables( Table $t1, Table $t2 ) { ... }

multi join_tables( Any, Any ) { fail() }

Given two Table arguments, the first multi candidate matches and gets called. Given any other combination of arguments, the second candidate matches and produces an error.

Knowing that you have two Table objects isn't enough, however. The tables might have no relationship to each other. Imagine if you somehow could verify that the tables have an appropriate relationship. If I were to implement this, I might check that the keys of the tables matched types, perhaps with a syntax something like:

multi join_tables ( Table $t1, Table $t2 where { $t1.primary_key eqv $t2.foreign_key( $t1 ) } ) { ... }

That is, the keys must be of equivalent types. If one key is a user_id and the other is an Integer, the where clause won't match for this candidate, so a different multi will get called.

Now imagine that for those embedded SQL minilanguage statements where table name is available at compilation time and sufficient type information exists to verify the statements themselves at compilation time:

SQL {{
    SELECT name, address FROM { User users }, { Address addresses }
}}

... then everyone who uses this minilanguage (and has set up the table information appropriately) gets safety and correctness by default. Some of that can even occur before the program runs. The rest of it can occur as the program runs.

(A really, really good type checker and optimization system could infer that some errors are impossible even if it can't prove the use of a single type in every case.)

Now imagine that you have a language which allows you to build minilanguages like this, to build APIs which specify correct operations and fall back to good error reporting on incorrect operations, and which do so without interfering with other code and other extensions.

Welcome to Perl 6.

my $thingie =<<'END';
Thaddeus Droit
4616 NW Washington Place
Beaverton, OR 97006
END

It's obviously an address, but what does Perl know about it? Perl knows it's a string. Perl knows it's some 60 characters long. Perl may even know that it's a valid string of Latin-1 characters.

Perl doesn't know where the string came from, nor that it contains a street address or a legal name nor a zip code (and not a zip + 4). Any meaning to the program beyond "It's a string of some 60 characters and is valid in the Latin-1 encoding" is far beyond what Perl knows about it. That's why the name of the variable is $thingie; even though Perl doesn't care about variable names, calling it $address instead could have led you to believe there's more structural meaning to this chunk of memory than actually exists.

Names are important, at least to people maintaining source code. This code is obviously wrong:

$user->set_address( $birthday );

... but to Perl it might as well be:

$foo->bar( $baz );

... for all of the semantic meaning it understands. There's no obvious intent.

I know you're smart and you're way ahead of me and you think "If I wanted a good static type system, I know where to find Haskell or OCaml and I'd never let code that bad get out of code review and why aren't you writing tests." but that's not the point. You can be super careful or make APIs which restrict the most natural way to write code in the host language in favor of extra security. That may be the right approach. (You have to be careful, though: the ease of interpolating untrusted user input into a raw string or the use of register globals in PHP seems analogous to the attractive nuisance doctrine, where people who don't know any better can't analyze the risk appropriately.

There may be another way.

Suppose I annotated the address:

my Address $thingie =<<'END';
Thaddeus Droit
4616 NW Washington Place
Beaverton, OR 97006
END

It's still a chunk of memory with certain characteristics, but now it has an extra piece of metadata related to the program itself (and not merely Perl itself). A clever compiler could detect certain places where the semantics of an operation don't match:

method set_address(Address $addy) { ... }

... though you do have to be able to resolve this kind of dispatch at compilation time to prove the type safety of the entire program at compilation time. (I've seen suggestions that even Smalltalk programs can resolve some 85-90% of dispatch targets in a static fashion.)

You don't have to go that far; runtime verification with a good test suite is effectve, can be fairly cheap, and is available right now in Perl 5 with Moose.

There's still another way. Consider again the untrusted input example. If you enable tainting, you might read user input into an address:

my Address $untrusted_addy = $req->get( 'address' );

You don't see it in the declaration, but the "This is tainted!" metadata is present in $untrusted_addy. How do you deal with that?

You could be picky about always untainting untrusted data, but can you do that accurately and effectively? Can you rely on everyone always getting it right?

What if you could write:

SQL {{
    UPDATE users SET address = { Address $address } WHERE user = { User $user }
}}

... and Perl could verify that $address is an appropriate Address (and $user is an appropriate User), could quote and escape and validate both of them effectively, could extract the primary key from $user, and could untaint any tainted $address or $user?

If your language supports multiple dispatch, lets you define your own types, lets you override stringification, and can override interpolation for cases like these, you can do such things.

In other words, you could turn what would otherwise be a raw string into an embedded little language with its own syntax and semantics, interoperate with native data structures in the host language, and provide composable safety—and users don't have to know much of anything about how this works, as it pretty much does what they expect.

I can imagine a language like that.

Bad code is possible in any language and wrong code is possible with any API. Even so, it's possible to create languages and APIs which make the right thing so much easier than the wrong thing that only the most incompetent (or dangerously malicious) write bad code.

Imagine, for example, a database access layer which forbids the use of raw strings to create SQL queries. You might have to write:

my $sth = $dbh->select( @tables )->join( %relations )->where( %conditions );

That's not necessarily a beautiful interface dashed off after a moment of thinking, but it has an important security property: it avoids the interpolation of untrusted user input. All data sent to the database may go through a quoting or untainting process without the user having to remember to do so.

A similar library could help avoid malicious user input from interfering with the display or operation of a web site, for example. These are both specific cases of a general principle: replace unstructured string data with structured data. In both cases, the structure of the data makes the intent of the data clear, which allows the library to ensure as much safety as possible.

This principle has other implications as well; more on that next time.

Yuval Kogman asked Are we ready to ditch string errors? I am; there's a general principle of API design beyond his question.

One problem with die "Some error!" is how to identify what error that represents—not to a programmer or user, who ostensibly speaks enough English and problem domain jargon to have some idea of what the error means—but the rest of the program. How does your code catch this error and distinguish it from some other type of error? Can you determine which of the two you can handle and which you must delegate?

Break out split or the regular expression engine and prepare to write heuristics which guess, and woe to you if someone someday internationalizes your error messages or runs all of your exceptions through a logging mechanism which changes their formatting slightly or....

The problem is that you can't take advantage of the structure of the exception data because it's not present in the string. The same goes for DBI's connection strings:

my $dbh = DBI->connect( 'dbi:DriverName:database=database_name;host=hostname;port=port' );

As the documentation suggests in the very next sentence:

There is no standard for the text following the driver name. Each driver is free to use whatever syntax it wants.

Compare this to a keyword argument form:

>my $dbh = DBI->connect(
    driver   => 'DriverName',
    database => 'database_name',
    host     => 'hostname',
    port     => 'port',
    extra    => 'arguments',
);

This has several advantages. The method doesn't have to guess (or parse) the string. The layout and vertical alignment makes the keyword form easier to read and to modify. DBDs can decorate and augment this argument list without parsing and recreating a string. Verification and default arguments are much easier.

The same argument goes for using a module such as File::stat instead of parsing the output of `ls -l filename`.

The same argument goes for... you get the point. It's far too easy to unfold the regex widget from the swiss-army chainsaw when a little bit of caution decomposing data into structured data makes your programs safer, easier to use, more flexible, and more robust.

(I consider sometimes how a language would look if it had only keyword arguments and how you could optimize them with immutable, internable strings and cached call sites and a zero-copy register allocation mechanism, but I made it as far as writing a self-hosting garbage collector before I had real work to do.)

]]> http://perlblogs.com/2010/07/07/dont-parse-that-string/feed/ 0 http://www.modernperlbooks.com/mt/2010/07/luck-and-the-class-struct-api.html http://www.modernperlbooks.com/mt/2010/07/luck-and-the-class-struct-api.html#comments Fri, 02 Jul 2010 20:01:01 +0000 chromatic Class::Struct has been a core module for ages. (Previously it was Class::Template, but a great renaming occurred 13 years ago.) If you've never seen it before, it might remind you a little bit of Moose:

package Cat;

use Class::Struct;

struct( name => '$', age => '$', diet => '$' );

You don't get all of the benefits of Moose, but you do get attributes and accessors. You also get a default constructor.

Of course, the default constructor reads something like:

{
    package Cat;
    use Carp;

    sub new
    {
        my ($class, %init) = @_;
        $class = __PACKAGE__ unless @_;
        ...
    }

    ...
}

If that emboldened line is curious to you, it's curious to me too. I saw a note in one of the test files somewhere suggesting that the purpose of this was to allow you to write:

package Cat;

my $cat = new();

I don't know why you'd do that, however. In what kind of object design does it make sense to create objects of a class from within that class? (That seems like a violation of responsibilities to me.) You can also write:

package NotCat;

my $cat = Cat::new();

... though that's exceedingly fragile. For one thing, it implies that you could also write RobotCat::new()—assuming that RobotCat extends Cat, but avoiding method dispatch for calling a constructor means that RobotCat had better provide its own new() which behaves as a function as well as a method. (Even if you somehow convinced the subclass to inherit the superclass's function through some sort of exporting scheme, the hardcoded __PACKAGE__ would hurt.)

Hardcoding a method dispatch as a function dispatch means that the maintainers of Cat are not free to change how Cat provides its constructor, much for the same reason.

Woe unto you if there's an inherited AUTOLOAD somewhere.

I realize that in 1994 or 1995, people who wrote OO code in Perl 5 might have had familiarity with OO in C++ (where this syntax makes a little more sense) or, perhaps, Java where the indirect constructor call (the my Cat $cat = new Cat; is prevalent), but the benefit of hindsight is that experienced Perl 5 programmers can look back on this API a decade later and cringe at its potential for misuse.

If you're lucky, everything will go right—but what kind of a defensive programmer relies on luck when designing an API?