(I've been refactoring one of our business projects in preparation for another round of deployment in the next couple of weeks. We could launch without these improvements, but administrative work took almost two weeks longer than the afternoon I'd planned for it, so I decided it was worth my time to reduce technical friction so that further improvements are easier. More users means more work, so why not accelerate that work while I have the chance? I have another longer technical post to write to praise the use of Moose roles for a plugin system and to show off the stupidly-great task launcher, but that's for later.)

I found myself writing two code couplets that were similar enough they triggered my "Hey, refactor away this duplication!" alert. It's extra sensitive, because I know I'll have a few more couplets like this in the very near future:

while (my $stock = $stock_rs->next)
{
    my $pe_update = $self->analyze_pe( $stock );
    $stock_txn->add( $pe_update ) if $pe_update;

    my $cash_yield_update = $self->analyze_cash_yield( $stock );
    $analysis_txn->add( $cash_yield_update ) if $cash_yield_update;
}

The *_txn variables contain objects representing deferred and scoped SQL updates. I'll talk about that at YAPC::NA 2012 in When Wrong is Better.

The general pattern is this: for every stock in the appropriate resultset, call a method in this plugin. The method will return nothing if it fails (or has nothing to do) or it will return data to be added to the appropriate transaction. I have at least two types of transactions available here at the moment, and may have more later: one transaction updates stock data and the other updates analysis data.

I have several options. I could rework the data model so that this stage always only updates one transaction, in which the loop body could instead look like:

{
    for my $method (qw( analyze_pe analyze_cash_yield ))
    {
        next unless my $result = $self->$method( $stock );
        $txn->add( $result );
    }
}

This technique of hoisting the variants into an ad hoc data structure and using existing looping techniques works well sometimes. (I use it in other parts of the system.) It's relatively easy to expand, even though it moves interesting information ("I'm calling the analyze_pe method!") to a place where tools have more trouble finding it. (I search for >analyze_pe when I want to find method calls.) You may have used something similar to define several parametric methods at BEGIN time. It's the same type of pattern, and while Perl 5 provides most of the tools necessary to allow this, it doesn't natively express this pattern well.

I could also change the transaction object's add() method to do nothing when it receives an empty list of arguments. I like that in some ways, but I don't like it in others. I've come down on the side of keeping its invariant (it always takes only one scalar as an object) pure for now. If I change it to take a list of updates, that might be the right time to reconsider this.

What I notice in the code as it stands right now is that the individual variables $pe_update and $cash_yield_update are synthetic variables. They only exist to support the code as written; they're not necessary for the algorithm. If I were to modify this code but only this code, I'd really rather write:

{
    ADD_TXN_WITH( $self, analyze_pe,         $stock, $stock_txn    );
    ADD_TXN_WITH( $self, analyze_cash_yield, $stock, $analysis_txn );
}

... though that syntax doesn't thrill me either. The clearest possibility I see right now is:

{
    $stock_txn->add(    SKIP unless $self->analyze_pe( $stock )         );
    $analysis_txn->add( SKIP unless $self->analyze_cash_yield( $stock ) );
}

... where SKIP does some magic to move to the next statement, not the next loop iteration. (I have some ideas how to write XS to make this work, but that creepy yak needs a shave and some mouthwash.)

The second best option right now is adding a function or method as indirection to encapsulate the synthetic code. I'd rather avoid synthetic code, but at least it reduces the possibility of copy and paste bugs.

For now, with only two steps in this analysis, I'm leaving it as it is. Two repetitions of something this similar set off my refactoring alarm, but I resist the urge for refactorings this small until I see three instances of near-duplicate code.

You may have read my Controlling Test Parallelism with prove and Parallelism and Test Suites. I still have Test::Harness parallelism enabled by default on most of the machines where I install my own Perls. While I haven't yet filed tickets and tried to write patches for modules which need a little help to run tests in parallel, I've only found a few lately that need work. That's nice—having a module install through cpanm in five seconds is a lot better than ten seconds or more. (I like cpanm because it's fast and quiet, and part of its speed comes from not printing to the console.)

I like instant feedback.

Like Steven, I noticed quite a while that installing a custom Perl through perlbrew takes a while, but then I remembered that a lot of work went into the Perl 5 test suite to make tests run in parallel. (We did something similar with Parrot several years ago, and it changed the way I work forever.)

To run core tests in parallel, set the environment variable TEST_JOBS=n, where n depends on your computer. I use a value of 9 on a quad-core machine; in practice, that tends to keep the CPU busy while not blocking anything too long on IO. You can set it globally in your shell's configuration file or create an alias or wrapper for perlbrew.

As most of the time spent compiling and installing Perl 5 through perlbrew goes to running the test suite, this has saved me a measurable amount of time.

It looks like Red Hat is distributing Perl without the core library ExtUtils::MakeMaker. If you're not familiar with the details of the Perl 5 build chain, all you need to know is this: without MakeMaker, you're not installing anything from the CPAN.

Ostensibly Red Hat and other OS distribution vendors split up Perl 5 into separate packages to save room on installation media. Core Perl 5 is large and includes many, many things that not everyone uses all the time... but the obvious reaction to defining a core subset of Perl 5 that a vendor can call "perl" is another of those recurring discussions which never quite goes anywhere.

For example, who needs the documentation just to run code? (Except that the diagnostics pragma relies on the existence of perldiag.pod to run.) Who needs the huge Unicode encoding tables for ideographic languages such as you might find in Japan, China, Korea, and other Asian locals? (Answer: Asia.) Who needs the ability to install code from the CPAN? (Answer: users.)

While there's a lot of stuff in the core that probably doesn't need to be in the core, or at least installed by default (a LaTex formatter for POD, the deprecated Switch module, Perl 5.005 Thread emulation), one thing is both clear and almost never said.

I'll give you a moment to think about it.

Here's a hint: you're usually better off compiling and installing your own Perl 5 under your complete control such that you can compile in options you want (64-bit integers, for example) and out options you don't (threading imposes a 15% performance penalty even in the single-threaded case) and so that you can manage your own library paths without changing the behavior of the system). perlbrew changes the game. Learn it, like it, love it.

The perpetual discussion misses one important point:

The vendor perl—especially on installation media—is not for general purpose Perl programming. It's there only to support basic administrative programs provided with the system as a whole. That's why you don't replace the system Perl. That's why you don't mess with the system CPAN modules. That's why you fence off whatever's in /usr/bin/perl like it's Yucca Mountain and you're stuck with a '50s reactor design instead of something safe and clean.

Vendors can tune and tweak that Perl to their satisfaction to provide just what they need to install and configure a working system. They can keep it as crufty and out of date as they like. When it breaks, they get to keep all of the pieces and sew them back together like some sort of Fedorastein's monster. They just can't let it out of the lab.

This of course means that they need to provide packages of Perl 5 Actual for users and developers such that it's the full core of Perl 5. (It'd be nice if they called not-a-perl as such, but one thing at a time.)

You can't predict what users will and won't do. That's why you code defensively. The moment distributions started carving up Perl to install just the little bits they needed in the hopes that their guesses as to what users wanted were right, they put everyone in a bind.

Certainly Perl 5 could benefit from a thorough review of what's in core and why, but I suspect that even if p5p came up with packaging guidelines for all of the imaginable use cases and combinations of distributor needs and user wants, it still wouldn't solve the real problem.

(Credit Allison Randal for pointing out the real problem years ago. We've discussed several times the idea of a stripped-down VM for a real language—something with better abstraction and reuse than Bash—with easy access to libraries and a very small footprint, but it's a bigger job than either of us could accomplish. It's still a righter approach than bowdlerizing an upstream distribution.)

I want to inspect the call stack within a helper module, but I don't care about the call stack within the module. I want to use lots of little helper functions, because that's good design, but caller works against that. Looking up the stack means keeping track of the magic number of call frames within my module I currently use, and that's one more thing to update when I change things.

That's structural code highly coupled to the arrangement of other code, and if that doesn't wrinkle your nose with the subtle aroma of fragility and peril, I don't know what will.

Imagine if instead:

my $caller_object = capture_caller_state();

while (my $call_frame = $caller_object->next)
{
    next if $call_frame->is_eval;
    say $call_frame->location_as_string;
    my $next_frame = $call_frame->previous;
    ...
}

Imagine if you could pass this object around.

I know things get complicated if you pass this object up the call chain, but stack unwinding is a solved problem in that anyone capable of recognizing the problem should be able to figure out cheap and easy ways to fix it.

Alas, Perl 5.14 doesn't have this feature, and it's probably too late for 5.16 to get it, so for today I'm stuck imagining what might be.

(If you've never thought about this sort of thing before, you owe it to yourself to learn more about Continuation Passing Style, which is at least an order of magnitude more mind-bending at the start and at least two orders of magnitude more useful.)

I've long used test-driven design for code I care about. Knowing that it works, works reliably, and will continue to work helps me write the right thing. As I've gained experience with testing, I've come to appreciate the nuances of how and what to test and why. For example, I often don't bother to test exploratory code—it's more important for me to learn how code might work to help me figure out how to design the real thing than it is to hew to a rigid mindset of "You must always test everything."

Similarly, I see little value in rigorously testing basic accessors, and thank goodness for Moose for turning class declarations into declarative code.

In more serious discussions of testability and testing, people often argue that testing UIs is impossible, or merely difficult, or rarely worth the trouble to do so completely. I agree to some respect; the most effective way I've seen to test UIs is to test the backend data model rigorously, write a very thin controller to marshall and unmarshall requests and responses, then to write a handful of tests which explore the edge cases of UI interactions. That requires a deep understanding of what happens for each interaction, and you can only test those interactions which produce measurable side effects.

Let that latter point sink in for a moment.

My most recent work on multiple projects includes a lot of embarrassingly parallel batch processing work, where the user interface is largely static, produced daily from a series of templates. Generally the templates are correct if they pass the eyeball test with live data.

This works except when it doesn't.

Consider a snippet of Template Toolkit code to display a list of stories:

[% FOR e IN entries %]
    <div class="story story[% loop.count %]">
        <h3><a href="[% e.url %]">[% e.title %]</a></h3>
        <p>[% IF e.author %]
            [% e.author | html %]<br />
        [% END %]
        [% f.format( e.posted_time ) %]</p>
        [% IF e.has_image %]
            <img src="[% e.image.full_file %]" alt="[% e.title | html %] image" class="image" />
        [% END %]
        <p>[% e.short_content | html %] <a href="[% e.url %]">Read More</a></p>
    </div>
[% END %]

That code had a bug. Can you find it by eyeballing it? Neither could I, for a long time. I'll give you a hint: it never displayed images associated with stories. Give up?

I wrote backend test after backend test, fixing a couple of other bugs along the way, then traced through the code in a debugging batch until I had convinced myself that the data model was completely correct. The bug had to be in the template because it could be nowhere else.

Rigorous template tests didn't seem to help either, and I'd almost convinced myself that I had a nasty spooky action at a distance bug somewhere in the data model before I saw the problem:

[% IF e.has_image %]
    <img src="[% e.image.full_file %]" alt="[% e.title | html %] image" class="image" />
[% END %]

When I added a test for the has_image() method on story objects to the data model tests, it failed immediately because that method did not exist. It used to exist, but a refactoring somewhere went too far and removed that method and its tests from everything except the template. Because Template Toolkit is not so strict about the distinction between method and property access, the lack of a method is no big deal. It fell back to property access on a hash and, finding no such value in the hash, returned a false value.

There are ways to fix this. I'm sure there's a plugin for Template Toolkit, and probably a debugging/diagnostic mechanism somewhere I haven't yet discovered. Yet the problem remains.

I've heard it said (Rich Hickey? Martin Odersky? I think it was Rich.) that every bug in your program has passed the type checker. That statement contains wisdom. Yet it's also true that every bug in your program has passed the test suite. (We assume you run with type checks and keep your test suite fully passing.)

Neither is a substitute for careful thought and understanding. Both protect you only as far as you can take advantage of them.

My next post will explore some options for making these kinds of errors easier to catch way before the point of weird debugging sessions.

You can already preorder Modern Perl: 2011-2012 from Amazon and it'll be available from other bookstores in the next couple of days. I expect copies to ship by early next week.

What's new?

I closed every bug filed against the previous edition—a handful of typos, a few confusing sections, and some suggestions on improvements. I took most of these suggestions.

I also revised almost every paragraph of the book, even at the sentence level. While I'm proud of the first edition, this new edition is clearer, shorter, more concise, and more accurate.

I removed some sections that didn't work. This didn't amount to much, but some of the things I recommended made little sense anymore and some of the experimental modules I mentioned turned out to be less useful than hoped. You won't miss them. This includes a big revision of the smart matching section in anticipation of changes in 5.16 and 5.18 (simplicity is the rule of the day; the simpler your use of the operator, the less risk you face for confusing behavior and edge cases).

I also added explanations for the new features in Perl 5.14, especially the /r modifier on regular expressions, and prepared for further features added in 5.16.

You might notice subtle improvements in formatting; the font size used for code examples is larger and callouts are much more attractive. We're still experimenting with the best way to format books to make readable text while avoiding brick-sized tomes, so feedback is more than welcome.

As with the first edition, we'll make electronic versions available for free from the Onyx Neon site. We hope to have ePub versions ready in the next two weeks. We'll also put raw HTML online on modernperlbooks.com at the same time (this also allows us to link to translations with greater ease). In addition, we're making most of our publishing tools available as CPAN distributions, including the formatting modules (Pod::PseudoPod::DOM) and the rendering pipeline (Pod::PseudoPod::Book). This will allow you to make your own version of the book in other languages if you desire, or to create new and attractive books in many formats. We'll make our own Kindle version, rather than using Amazon's conversion tools (they didn't work as well as we'd hoped), and we hope to get the book in the Nook store as well.

We plan to release a new version of the book late this year or early next year to cover 5.14 and 5.16. We'll almost certainly drop support and mention of 5.10.

We couldn't have done this book without you. Thanks for reading. Please share with your friends.

Instead, I see Modern::Perl as enabling the core features I wish were on out of the box by default in Perl 5. While it'd be nice to pull in Try::Tiny and sometimes I might wish for fatal warnings, the former is not a core module and the second isn't something I use in every program.

Consequently the module hasn't needed much maintenance. Yet I've added a couple of missing features that should keep it useful and usable into the future. I uploaded a new version yesterday and will upload one more today with a little bit more polish.

First, it now requires autodie as a distribution dependency. It doesn't load autodie, but installing Modern::Perl on 5.10.0 will also install autodie. (Anything 5.10.1 and newer includes autodie in the core.) You don't have to use it, but now you can rely on any Perl considered modern to have autodie available.

Second, it now loads IO::File and IO::Handle so that you can call methods on lexical filehandles without having to load either manually. Perl 5.14 fixed that usability niggle, but Modern::Perl fixes this for people using 5.10 or 5.12. (Why both? I can never remember which one superseded the other in 5.12, but better safe than sorry. I welcome a patch to load one over the other with a version check—and please test carefully.)

Third, I added unimporting support so that you can write no Modern::Perl; within a scope to disable strictures, warnings, and language bundle features. It's an all or nothing switch and will remain that way, but I can see this being useful in specific cases, especially when updating older code in stages.

Finally I added date support to importing. If you write use Modern::Perl; you'll get the features of Perl 5.10 (with the caveat that if you're using Perl 5.11.3 or newer, you also get the unicode_strings feature, of which all you're likely to notice is that Unicode strings work better in more places).

Yet for forward compatibility, you should be using:

use Modern::Perl 2012;

... which enables 5.14 features. You can use 2009 and 2010 to get 5.10 features and 2011 to get 5.12 features. If you use the wrong date on the wrong version of Perl 5, you'll get an error, which is as it should be.

Next year I'm likely to drop support for Perl 5.10, in which case you'll probably get an error message that that year isn't modern enough, but I could be convinced to do a version check instead. I haven't decided. The tradeoff is between providing a minimal module suitable for use in programs which helps people write better code from the start and between telling people what they should and shouldn't use. Besides all that, the relevant code is only a couple of dozen lines of very simple Perl. Anyone reading this can reimplement it almost trivially.

Of course, this all comes about because the Modern Perl book goes to the printer today. We're very proud of the new 2011-2012 edition which concentrates on Perl 5.12 and Perl 5.14. It addresses all of the known typos and confusing parts of the previous edition, covers new features in Perl 5.14, and is, from all reports, even better than the first edition.

The book should be in stores in the next week or so, and we'll have electronic editions up this month for free download and redistribution. (We hope you tell lots of people to buy the print edition because it's great and more people need it on their desks, but sharing is caring and we support that.)

In the first web bubble, the Silly side of Silicon Valley chased vanity metrics such as "the number of eyeballs on the site" and "brand awareness" and "unique visitors". Those numbers are only interesting when you can correlate them to producing value for customers and bringing in real cash in the form of revenue.

I've enjoyed the book The Lean Startup by Eric Ries because he offers a much better mechanism to track the success or failure of any attempt to produce real value to customers. While split testing (or A/B testing) is useful to see how small changes lead to different customer behaviors, Ries recommends cohort analysis, where you can see the behavior of real customers through the sales funnel and correlate the X-axis with individual changes to your business or product.

That means tracking customer behavior. If you're building some sort of software as a service product, and if the mechanism of delivery of that product is primarily a web site, you probably already know the punchline.

Assume I already know how to identify and log events for each salient customer action type. (I've built that kind of system before.) Assume I don't want to collect personally identifiable information (I don't). Assume I'm using Plack and its middleware heavily, and assume I'm happy using Catalyst as a web framework.

How can I identify unique users (with and without accounts) on a daily basis, anonymize them, but group their actions across the site such that my automated daily cohort graphs correspond with reality?

So far I've identified few points of possible contention. I can rely on browser cookies for unique identification of users if I know that user sessions have unique identifiers within a 24 hour period. (I could generate GUIDs for this, but that may be overdoing things.) I think< I also have to track the transition from anonymous visitor to authenticated user, but I might be able to convince myself that either replacing the current session or smple subtraction of successful login events from total number of unique anonymous visitors would give the right numbers.

(I also haven't dived much into how Catalyst 5.9 and Plack interact in terms of session and cookie handling. Everything's just worked, so I've ignored the details until now.)

I don't mind building such a system if necessary, but if all of the pieces are out there and available—or if someone's already built this and can give guidance—so much the better.

Have you solved this problem? If so, how did you do it? If not, how would you do it? Would you handle logging at the Plack level or the application level? Would you worry about tracking session changes? Does Catalyst need to know about this?

Any modern web framework has to deal with the idea of routes and request routing somehow. Given a request path (such as /stocks/AA/view_analysis), how does your application know what to do?

Catalyst solves this elegantly with a feature known as chained actions. Controller methods can consume zero or more parts of the path but, when explicitly chained, can combine. Consider the example request path. The controller is Stocks.pm. The second component of the path (/AA) is the identifier for a stock (Alcoa, to be specific. I'm neither long nor short on Alcoa itself, though I probably own some shares as part of a fund somewhere.) The final component of the path, /view_analysis, is an action—a verb representing an action the controller should take on the object representing Alcoa in the system.

You can probably start to see the idea of the chain right away.

The Stock controller has a controller method called get_stock which grabs the stock symbol from the request path, looks it up in the database, and stores the object representing that stock for further processing. If no such symbol exists, it throws an exception.

The view_analysis method chains off of the get_stock method such that Catalyst will only dispatch to view_analysis when it's already successfully dispatched to get_stock. Unless you write a custom dispatch system which bypasses the dispatch rules, users will never be able to call view_analysis without a valid stock object available.

(Further, these methods are part of a chain which requires that users have successfully logged into the system; they chain off of a user authentication system.)

In code terms, the relevant attributes look something like:

sub authorized :Chained('/login/required') :PathPart('stocks') :CaptureArgs(0);

sub get_stock :Chained('authorized') :PathPart('') :CaptureArgs(1);

sub view_analysis :Chained('get_stock') :PathPart('view_analysis') :Args(0);

The :Chained attribute is most relevant here. :PathPart governs how Catalyst's dispatcher makes each method visible to user requests (get_stock doesn't consume a part of the path on its own, while authorized consumes the name of the controller and view_analysis consumes its own name). :CaptureArgs and :Args control how many other pieces of the path the methods consume; in the case of get_stock, it's the single path element between /stocks and any subsequent chained actions—in this case, /AA. As view_analysis is the end point of a chain, you use :Args instead of :CaptureArgs.

With that all explained, request method chaining is fantastic. I can reuse get_stock() for other request methods and get all of its benefits, including the fact that only authorized users can even reach this point.

Yet I want to prove these characteristics of my application.

I want to prove these features so definitively that I don't want to write tests for them. I want my program to fail to compile if these characteristics are untrue.

I see chaining from get_stock() as supplying an invariant precondition to view_analysis() such that it proves, to my satisfaction, that I can always rely on a valid stock object being available within the analysis method. Always. Similarly, I can always rely on a valid user being available within both methods. Always always.

The problem comes in that it's easy to make a typo in the name of a chain or a method, or to use :CaptureArgs instead of :Args or vice versa.

Here's the thing: all of this metadata is metadata. All of this information is available at compile time, before Perl has to execute anything.

If I had a really good and extensible type system in Perl 5, I could write a couple of pieces of predicate logic to say that every chained method should be a starting point or have a valid predecessor. These are trivial properties of my program (no matter how large it gets) and they're resolvable with the information available at the point of compilation. Even with complex controller construction through the use of roles and parametric roles, this information is available.

I know how to emulate this behavior by injecting some sort of CHECK block into the code and schlepping through the symbol table and inspecting attributes myself, but that's emulating a useful feature we could exploit in a lot of ways.

Forget the talk about making Perl into Java or C++ by adding a silly manifest static type system. We could find and fix real errors in logic—trivial errors, trivially discoverable—if we had an extensible type system which let us define our own simple predicates.

(Implementing such is left as an exercise for a small army of readers cloned from a very small army of brilliant p5p hackers with copious spare time and a habit of reading ACM papers before breakfast.)

Some clever programs provide several symlinks to the main program and examine $0 to enable or disable certain behaviors. This is an easy way to hide the details of execution from users while making those behavior mnemonic.

I usually don't write those kinds of programs, but this past year I've written several batch processing programs which have several interdependent states. For example, one program runs from cron regularly to run through a pipeline of behaviors. Data moves through that pipeline; it's basically one big state machine.

The core of the program is a pipeline manager which runs the appropriate processing stages in order, such that on every invocation, the program moves data through at least one stage and potentially every stage. It doesn't have to move everything through the pipeline all in one invocation, but it does have to make progress on every invocation.

For various uninteresting optimization and locking reasons, I made this program a single execution unit. (I do use asynchronous IO for things like network access, but that's because the program is largely IO bound.) The program also has copious logging of the stage traversal, split between one log which tracks stage transitions and timings and stage-specific log files which have more details on the progress of those stages.

Until a few minutes ago, the easiest way to see the program's current stage was to tail the top-level log file. While running some live tests on a new feature, I found myself with free time and the desire not to switch back and forth to a tail -f screen again, so I checked the documentation for $0 again.

I knew that on certain platforms (GNU/Linux, which makes my life easier) you can actually write to it. If you do this, you can control what appears in the output of ps and top.

Every stage runs from a closure (shades of Plack):

    my $sub    = sub
    {
        my ($self, $config) = @_;
        my $log             = $self->get_fh_for_step( $config, lc $app );

        # show app stage in ps output
        local $0 = $app;
        my $app  = $module->new(
            logger   => $log,
            map { $_ => $config->{General}{$_} } @keys,
        );

        $app->run;
        $log->log( sprintf $message, $app->count ) if $app->count;
    };

A loop in the pipeline manager creates a new closure over the name of the module which implements the stage to create a new object for the stage, set up the logger, provide the appropriate configuration, and run the stage. The emboldened code shows the change I made.

Right now, my top window shows that the image processing stage has just given way to the report writing stage—and now the program has exited. In a couple of minutes, everything will start again.

Writing this entry took longer than implementing this feature. Five minutes of experimenting has improved the visibility and monitoring of this program immensely. Maybe it'll help you.