You have to develop and deploy a model of those URLs somehow.

The well-worn CGI approach is to write separate semi-interdependent programs for each step of the application. Each new URL element is a separate program. (This model matches the statelessness of HTTP nicely, but requires careful thought on the server side to manage the necessary statefulness securely.

Much of the web has settled on an MVC approach where all requests come in through one or more controllers which dispatch to functions or methods based on request contents: the URI, any form parameters, and the query string. Often a framework handles this dispatch by unpacking all of this information and matching regular expressions or fraguments of the URI, then traversing all the way to an endpoint.

I like Catalyst's approach to dispatch (see especially Catalyst's chained dispatch, a feature I'm pleased to see Dancer adopting as well): dispatch points are methods on a controller object with metadata attached in the form of subroutine attributes. This metadata registers methods as dispatch points and identifies the pieces of the URI they consume and any necessary preconditions.

It's flexible and powerful. Once you understand the terms used in the Catalyst world, you have many options to do the right thing for your application. You also get some form of validation because subroutine attributes are merely Perl 5 syntax, not entirely raw string text (though admittedly, the parameters to these attributes are raw strings which require validation—fortunately, as Perl 5 processes attributes during compilation, the attribute handlers can sanitize and validate these parameters to some degree during compilation).

Another approach, used in many places including the Java framework which shall remain nameless, is to use an external file to manage routing information. In this case, it's an XML file which maps URI components to controller classes and methods on those controllers. It also maps the results of those methods to template bundles (declared in another XML file).

Thanks to the Java web world's tragicomic affair with Beans and the prevalence of CamelCase, I constantly get capitalization wrong in this file. You see, while I may have a controller class named AdminAction, the right way to instantiate classes in Java is to let Spring manage their lookup and instantiation and lifecycle.

In other words, even though I know that this application needs a class named AdminAction and that that class will always be a controller and that the routing information declared in this XML file relies on AdminAction behaving as coded, the right way to write this application means declaring a bean named adminAction (notice the capitalization) in yet another XML file and using that in the routing XML file instead of the class name.

(Apparently this makes replacing the real controller with a mock controller easier for testing, as if testing a mock controller were worth my time—what could it tell me about the validity of my Spring configuration or my routing configuration that basic end-to-end testing couldn't tell me better? And people who like to write tests wonder why some people complain that a focus on unit testing is, in many cases, useless busy work.)

The funny thing about the bean syntax is that it means, at its core, that if you have an object with instance data such as companyName or userAddress, those fields are private which means that you have to have public getters and setters which match the form getCompanyName and setUserAddress, but when you refer to those bean properties, you leave off the get and set and follow the capitalization of the field names (companyName)—the private names, even though you're using the public accessors with different capitalization.

Compound this with the fact that the right way is to use Spring, which instantiates these objects lazily and introspects their properties and... well, you get to find your capitalization typos at runtime.

Whereas in a language or framework or toolkit which allowed you to define syntax for all of these things would let you validate even capitalization conventions (hey, if such a parser existed, it could have opinions about things like this!) and let you know far, far before some hapless tester clicking his way through a 38 page test script hit a runtime error that you made a silly typo...

... because hoping that unstructured data is correct without ensuring that it is correct is a recipe for errors.

That's the sign of a very good library or framework: the common things are easy and they resulting code is easy to understand, maintain, and extend.

Moose has a very nice feature of reducing the surface area of the APIs you create with it. You can declare a class with several attributes but only expose a few public getters or setters. The ability to use lazy initialization of attributes—as well as default values—allows the calculation of dependent attributes as needed.

Best yet, if you can arrange your classes in such a way as to provide their necessary attribute values when you call their constructors, you can treat your objects from the outside as immutable, and you're halfway toward inversion of control, if that's useful to you.

(Read more about Moose and immutability and the value of these features and designs in the Modern Perl book, available in several freely redistributable electronic forms.)

Think of all of this as not coloring outside the lines. While good art is aware of the structure and limitations of its chosen form and breaks those conventions when it makes for better art, good software is primarily a conversation mechanism where perhaps you don't want to try to outdo Pynchon or DeLillo or Wallace. Perhaps.

Contrast this mechanism of working with objects with the bean style as expressed in parts of the Java world: a class has several private attributes. Each of those attributes has public getters and setters which follow a standard naming convention. Objects interact with other objects by their conformance to a protocol and discovery and use of those getters and setters enabled by the magic genericity of reflection.

(Did anyone else hear a duck quack?)

For the convenience of allowing a dependency injection such as Spring or a web framework inspired by J2EE to manage some of the boilerplate of managing object instantiation or object lifespan or even mapping object data to and from strings for web or web request display, you not only give up the creation of your objects, but you expose their internal attributes behind a thin (and, be honest, autogenerated by your IDE when it pops up a warning!) layer of accessor method.

There is value in the loose and ad hoc adherence to a protocol of practice...

... but for a language which claims great maintainability benefits from forcing the use of static, manifest typing, you may encounter a fair few perplexing errors if you don't get the CamelCase type of your bean property just so. (Remember, reflection!)

While sometimes I do wish Perl 5 were stricter about types and signatures—certainly I look forward to using Perl 6's gradual typing for more projects—I have more sense of safety with the Perl approach, where we don't assume our language is completely safe by itself. We're a little more paranoid, and—at least in this case—I believe it leads to better designs.

The good news is that the Perl world (at least for those enlightened souls fortunate enough to be able to use modern Perl techniques) has little to worry about from the Enterprise Class world.

The bad news is that some of the problems afflicting Perl—especially for people who don't write it all day, every day—are common to software development in general.

For example: this CRUD project uses an Apache framework which has been around for most of a decade and is on its second iteration. Apache projects get a lot of credit for rigorous documentation...

... except that their documentation in this case is, as with most open source projects, a mish-mash of conceptual overviews, API references, contributed guides, tutorials, and links to mailing list archives and blog postings, with little narrative flow between them and, worse, often precious little internal information on which version of the framework applies.

Search for information with your favorite search engine and be ready to wade through inapplicable and inappropriate data, whether wrong from the start or rendered irrelevant several years later. (This is as much an indictment of search engines as anything else.)

Perhaps the most concerning trend in this project is that the Enterprise Class nature of the project betrays a desire not to express any design opinion. In fact, the dominant design opinion seems to be "Anything is up for grabs"—you can perform validation in several different ways, you have your choice of several competing templating engines, you can map URIs to controller actions in at least three ways, and there are at least two (possibly more) mechanisms of accessing model data in templates.

The cost of all of this complexity exposes itself in two places. First, you have to decide (through coding standards or stumbling into a design you like) how to structure your application before you can get anything done. Second, you swim through a Styxian deluge of XML to configure (and reconfigure and modify and adapt and configure and reconfigure...) anything and everything.

(A plugin exists which nominally supports convention over configuration, but the lack of documentation and, apparently, interest renders it far too risky.)

I hesitate to blame too much the authors of the framework for this mess (and, as such, refuse to name the framework). Certainly some of their design choices reflect fundamental flaws of the Java language itself (in particular, Java's relative inflexibility especially with regard to its silly static manifest type system combined with the IDE-based tooling necessary to manage its verbosity and lack of abstraction make even programming in XML seem more flexible). Even so, the ceremony necessary to produce even the single CRUD steps of "Create an item and view it" involves editing multiple XML files, creating several Java classes, and—if you do things the Enterprisey way—creating and configuring subtemplates for the entire layout of the site as well as embracing the inversion of control object instantiation and initialization pattern (and given the unpopularity of the immutable object pattern in Java as compared to bean-style mutators, it's not easy to reason about the state of your objects) is damning.

The lessons for Perl are simple:

• Avoid the overcomplexity of projects by emphasizing the flexibility and simplicity of the language. Yes, you read that right. Meditate on it.
• Write better documentation.
• When the best way to do something changes, deprecate old documentation and point to the new approach.
• Emphasize simplicity. Make the default a good way to do most things. Provide comprehensive tutorials which explain the entire system using that effective default.
• Provide smaller upgrade paths and avoid big-bang incompatible releases which fork documentation and knowledge.
• Unify where possible.

The experience has been enlightening.

One bright spot is Java's notion of web deployment. It has its obvious flaws, but it has obvious benefits. You gather all of the artifacts necessary for your project: XML files, JAR files (bundles of Java libraries), web templates, the classes you've written, and other resources, all in the appropriate subdirectories governed partly by convention and partly by a manifest file. Your deployment process bundles everything up in to a WAR file (a web archive). Then you dump this WAR file into an application server.

Compare that to another project I delivered recently where I sent a CPAN-style distribution tarball to a (much more technically savvy) client, who set about installing dependencies from the CPAN (where Module::Install couldn't get the job done) and from yum, where you don't want CPAN installing, for example, PostgreSQL for you.

Granted, I could have used a tool such as Shipwright to bundle my code, but I didn't. For better or worse, CPAN-style distribution is awfully common.

Imagine instead if I could have bundled everything up into a single WAR-style file which a hypothetical Perl web application server could deploy.

Java achieves this in at least two ways. First, Java wants to own the world—Java does have bindings to native code, but the Java desire to have pure-Java everywhere tends to encourage people to write pure-Java code where Perl hackers might write XS. (XS means you have to compile native code on the target deployment platform.) Someday perhaps Perl 5 can reduce its reliance on XS.

Second, Java application servers also have the advantage of a stronger memory separation between different applications running on a single JVM. Some of this is due to the static nature of the Java language. When you can resolve all references within an application at load time, you don't have to look things up by strings in a global directory. (Of course, you do have to be clever in your VM design to allow runtime reflection, so the implementation of the VM is also important.)

Plack offers a way around this. Imagine a Perl app server based on Plack to which you can deploy multiple applications. The app server could start multiple independent backends—even so far as having multiple installations or at least library directories for each application, as necessary—to use Unix process separation to achieve this memory separation.

Such an approach would still likely use less memory than Java.

Make the Plack app server a reverse proxy capable of serving static files and SSL and redirecting invalid requests and you might have something really exciting. (Make it cloud aware and give me a 10% equity stake.)

These are idle musings, and the Java ghetto doesn't tempt me to ditch the world of Perl for a work life full of beans and mappers and inversion of control governed by there's-no-angle-bracket-shortage-in-this-sea-of-XML—but even in that mess, there are still some good ideas worth borrowing, buffing, and bettering.

Firstly I’d like to point out that, funnily enough, I’m not very familiar with Java or Perl6. It’s entirely possible that I’ll make all sorts of errors in the following details. If you spot any do please let me know.

Background

The Java language ecosystem is big and mature after years of heavy investment of time and money.

It doesn’t have a central repository of Open Source modules like CPAN (though Maven repositories like these are similar I guess). It does, however, have a number of mature high quality class libraries, and a very large number of developers familiar with those libraries (more on that below).

Goals

The primary goal of the java2perl6api project is to make it easy to create Perl 6 class libraries that mirror Java equivalents. By mirror I mean share the same method names and semantics at a high level (though not at a low-level, more on that below).

Secondary goals are to do that well enough that:

• the documentation for Java classes can serve as primary the documentation for the corresponding Perl 6 classes. The Perl 6 classes need only document the differences in behavior, which these should be minimal and ‘natural’. The same applies to books describing the Java classes.
• Java developers familiar with the Java classes should feel comfortable working with the corresponding Perl 6 classes.
• and, hopefully, some way can be found to convert test suites for the Java classes into Perl 6 code that’ll test the corresponding Perl 6 classes. (I appreciate that this is a non-trivial proposition, but there are viable approaches available, like xmlvm.) Even if that can’t be done, extracting and translating tests manually is less work, and more effective, than creating them from scratch for a new API.

Why?

Firstly, creating good APIs is hard. Java APIs like JDBC 3.0 and NIO.2 are the result of years of professional effort and demanding commercial experience. Why not build on that experience?

I appreciate that Java APIs are often limited by the constraints of the language, such as the lack of closures, and that Perl 6 can probably express any given set of semantics more effectively than Java. My point here is that some Java APIs embody, however inelegantly, years of hard won experience that we can benefit from. I’d rather make new mistakes than repeat old ones.

Secondly, there are many more Java developers than Perl developers. Many many more if job vacancies are any indication:

I think we’d be foolish not to try to smooth the path for any Java developers who might be interested in Perl 6. The java2perl6api project is just one small aspect of that.

I really hope someone starts writing a “Perl 6 for Java Developers” tutorial. Perl 6 has the potential to become a very popular language¹. Getting just a tiny percentage of Java developers (and Computer Science majors and their teachers) interested in it could be a big help.

Thirdly, any future DBI for Perl 6 and Parrot needs a much better foundation than the very limited and poorly defined one that underlies the Perl 5 DBI. I plan to adopt the JDBC 3.0 API and test suite for that internal role. (You could call this a “Test Suite Driven Strategy”.) I’ll talk more about that in a future blog post.

The History java2perl6api

I’ve been kicking around various ideas for integrating Java and Perl6/Parrot for years. I think I first decided to use JDBC as the inspiration for the DBI-to-driver API in 2006.

You may remember back in 2004, around the 10th anniversary of the DBI, the Perl Foundation setup a “DBI Development Fund” that people could donate to. I’ve never drawn any money from that fund. I want to use it to oil other peoples wheels.

In 2007 Best Practical sponsored Perl 6 Microgrants through the Perl Foundation. I asked if I could piggyback my idea for a Java to Perl 6 API translator onto their microgrant management process but using money from the DBI Development Fund. TPF and Best Practical kindly agreed. I posted a description of the task and Phil Crow volunteered and was awarded the microgrant in April 2007.

At OSCON in July 2007 I gave lightning talk called “Database interfaces for open source languages suck” which explained the rationale for using JDBC as a foundation for the DBI-to-driver API and mentioned Phil’s java2perl6 project.

Development ground to a halt around the end of 2007 for various reasons. It picked up again for a few months after OSCON 2009 (where I gave a short lightning talk asking for help) then stalled again in October. Partly because we seemed to have hit a limitation with Rakudo and partly because I was focussed on Devel::NYTProf version 3 and then version 4, which took way more time than I expected.

There’s life in the project again now. We’ve dodged the earlier problem, put the code on github, brought it into sync with current Rakudo Perl 6 syntax, and generally instilled some momentum.

The Current java2perl6api

Let’s take a look at a simple example.

To generate a perl6 file that mirrors the API of the java.sql.Savepoint class you’d just execute java2perl6api like this:

$ java2perl6api java.sql.Savepoint
loading java.sql.Savepoint
wrote java/sql/Savepoint.pm6 - interface java.sql.Savepoint
checking java/sql/Savepoint.pm6 - interface java.sql.Savepoint

That’s loaded and parsed the description of the java.sql.Savepoint class (from the javap command), generated a corresponding perl6 module, and run perl6 to validate it.

The generated module (with some whitespace and cruft removed) looks like this:

use v6;
role java::sql::Savepoint {
    method getSavepointId (
    --> Int   #  int
    ) { ... }
    method getSavepointName (
    --> Str   #  java.lang.String
    ) { ... }
};
=begin pod
=head1 Java
  Compiled from "Savepoint.java"
  public interface java.sql.Savepoint{
      public abstract int getSavepointId() throws java.sql.SQLException;
      public abstract java.lang.String getSavepointName() throws java.sql.SQLException;
  }
=end pod

The pod section shows the description of the class that javap returned. The java2perl6api utility parsed that Java interface and generated the corresponding Perl6 role. The ‘java.sql.Savepoint’ has been mapped to ‘java::sql::Savepoint’. The generated methods are stubs using ... (the “yada, yada, yada” operator). The types int and java.lang.String have been mapped to Int and Str. Because the only types used were built-ins, no type declarations were added.

Currently java2perl6api handles the above plus overloaded methods (which generate multi methods), multiple implements clauses (which generate multiple does clauses). There’s also partial support for class/interface constants (which currently generate exported methods).

The default behavior is to recursively process any Java types referenced by the class which aren’t mapped to Perl 6 types. So executing java2perl6api java.sql.Connection, for example, will generate 48 Perl 6 modules! (Because java.sql.Connection refers to many types, including java.sql.Array which refers to many types including java.sql.ResultSet which refers to java.net.URL which refers to java.net.Proxy etc. etc.) The --norecurse options disables this behavior.

Normally you’ll want to use the recursion but instead of letting it drill all the way into the Java types, you would supply your own ‘typemap’ specification via an option. That tells java2perl6api which Java types you want to map to which Perl 6 types. So instead of recursing into the java.net.URL type to generate a java/net/URL.pm6 file, for example, you can tell java2perl6api to use a specific Perl 6 type. Perhaps just Str for now.

How this relates to JDBC / DBDI / DBI v2

I want to start applying java2perl6api to the JDBC classes now to create a “Database Driver Interface” or “DBDI” for Perl 6.

Starting with the DriverManager class and the Connection interface I’ll use java2perl6api to generate corresponding Perl 6 roles with heavy stubbing out of types. Basically anything I don’t need to think about right now will be mapped to the Any type.

I’ll start fleshing out some basic implementation logic for each in a Perl 6 class that does the corresponding role. I’ll probably use PostgreSQL as the first driver and the guts of MiniDBD::Pg as inspiration.

The first minor milestones will be creating connections, then execute non-selects, then selects then prepared statements. Somewhere along the way I expect they’ll be a Perl 6 DBDI driver implemented for the Perl 6 MiniDBI project. The next key step would be to start refactoring the code heavily so anyone wanting to implement a new driver should only have to implement the driver specific parts. (There are some JDBC driver toolkits that can provide useful ideas for that.)

What needs doing

There’s a TODO file in the repository that lists the current items that need working on.

One fairly simple item is to add a --prefix option to specify an extra leading name for the generated role. So java.sql.Savepoint with a prefix of DBDI would generate a DBDI::java::sql::Savepoint role.

Another item, less simple but more important, is to automatically discover the values of constants and embed them into the generated file. Probably the best way to do that is to extend the parser (which uses Parse::RecDescent) to parse the verbose-mode output of javap, which includes those details.

There are plenty of others.

How you can get involved

Firstly, come and say “Hi!” in the #dbdi IRC channel on irc.freenode.net.

The code is on github. You can get commit access by asking on the #perl6 channel.

There’s also a mailing list at dbdi-dev@perl.org which you can subscribe to.

I look forward to hearing from you!

1. 
   When I say “Perl 6 has the potential to become a very popular language” I do so with typical British Understatement.