The beast attacked livestock and people in south-central France in the middle of the 18th century. Some say the beast killed over a hundred people. The attacks ended after the deaths of two large wolves—one in 1765 and the other by Chastel in 1767.

By the 20th century the legend had grown to include the story that Chastel had melted down a medallion of the Virgin Mary to make two bullets of silver.

In 1986, Fred Brooks used this metaphor to propose that no single technology nor management technique could produce an order of magnitude improvement in productivity, reliability, or simplicity within the decade. (Most people get this quote very wrong. Most people haven't read the paper in detail or at all.)

Perl 1 entered the public world in 1987.

Today (while solving hard problems), I realized that Moose and its ecosystem—including projects it's inspired, such as other declarative mechanisms for describing classes and objects—may just represent an order of magnitude improvement in productivity, reliability, and simplicity.

I write a lot less code. I write less repetitive code. My code is easier to read and to maintain. My code is much more correct. My code is much more flexible and easier to test. (Compilation error messages are much worse. That doesn't bother me terribly, but keep it in mind.)

Keep your blessed hash references all you want and lament that kids these days don't have to walk uphill in the snow both ways to stuff things in package global variables like @INC and that we're lazy, fat slobs for using CPAN modules for what you have done by hand since 1994, just as Moses did and King James always intended...

... but I'm loaded for werewolf.

Why not modify add_txn to reject undefs?

The code in review is:

while (my $stock = $stock_rs->next)
{
    my $pe_update = $self->analyze_pe( $stock );
    $stock_txn->add( $pe_update ) if $pe_update;

    my $cash_yield_update = $self->analyze_cash_yield( $stock );
    $analysis_txn->add( $cash_yield_update ) if $cash_yield_update;
}

... and the near duplication obscures (to me) the point of the code. Both Aristotle and Chas. are right—perhaps it's clearer to allow $transaction->add to do nothing when it receives nothing. I write "perhaps" because I see the appeal of that change, but I'm not sure I like it.

As usual with my software, the system has a fundamental design principle: either succeed in full or do nothing. It's fine to skip half of the analysis steps if the data just isn't there. (The project as a whole can succeed if it's only 60% correct; the joy of a margin of error. It's a lot more accurate than that.)

I take this principle to mean that robustness is more important than completeness. Skipping bad data and moving on is perfectly fine. The next run may improve transient errors, and catastrophic errors will require human intervention anyhow.

When these principles translate into design, I prefer to handle errors at the point of detection and not spread error handling throughout the system. All of these analysis methods should return something. When they succeed, they return a hash reference mapping column names to values in a database table. When these methods fail—whether the existing data isn't sufficient to calculate updated values or something else went wrong—they return;. As you well know, that's an empty list in list context and undef in the scalar context of the example code.

Why add nothing to a transaction when I know there's nothing to add? Yes, add() could check that it has nothing to do and do nothing, and that's fine, but it seems like that expands the behavior of add()'s API to include caller errors. Then again, the add() method must check that each hash reference contains a value for the transaction's bound primary key, or it will generate buggy output.

I suspect that both Aristotle and Chas. have in mind Postel's Law:

Be generous in what you accept and picky about what you emit.

The result might look something like:

while (my $stock = $stock_rs->next)
{
    $stock_txn->add(    $self->analyze_pe( $stock )         );
    $analysis_txn->add( $self->analyze_cash_yield( $stock ) );
}

This change has an advantage: it only necessitates a change in the add() method. All of the analyze_*() methods can continue to work as implemented.

Of course, there's a slight performance penalty to doing this. In my case, it's immaterial, but it wouldn't be present with macros. This is an IO-bound application anyhow, and the transaction manager exists to avoid very real, very measured bottlenecks.

Finally, Aristotle's mention of the null object pattern was about real objects, and not methods which return empty lists or hash references. If that's your style, good for you—but it's not mine in this case. While it's not obvious from the small snippets I've posted so far, the responsibility of the analysis methods is smaller in scope than the responsibility of the transaction objects. Coupling transaction management to the analysis methods—in as much that they have to know about transactions to return the right objects—would turn the design of the system inside out. The result would very likely not be an improvement.

At its core, dependency injection is a formalization of the design principle "Don't hard-code your dependencies." Consider the code:

sub fetch
{
    my ($self, $uri) = @_;
    my $ua           = LWP::UserAgent->new;
    my $resp         = $ua->get( $uri );

    ...
}

That's not bad code by any means, but it's a little too specific and a little less generic due to the presence of the literal string LWP::UserAgent. That might be fine for your application, but that hard-coding introduces a coupling that can work against other uses of this code. Imagine testing this code, for example. While you could use Test::MockObject to intercept calls to LWP::UserAgent's constructor, that approach manipulates Perl symbol tables to work around a hard coded dependency.

An alternate approach uses a touch of indirection to allow for greater genericity:

use Moose;
has 'ua', is => 'ro', default => sub { LWP::UserAgent->new };

sub fetch
{
    my ($self, $uri) = @_;
    my $ua           = $self->ua;
    my $resp         = $ua->get( $uri );

    ...
}

Now the choice of user agent is up to the code which constructs this object. If it provides a ua to the constructor, fetch() will use that. Otherwise, the default behavior is to use LWP::UserAgent as before.

Adding one line of code and changing one line of code has provided much more flexibility.

An alternate approach is to allow setting ua through an accessor instead of a constructor, but as far as I can tell the only reason to do this is if you're stuck in The Land of the Java Bean Eaters.

While the existing literature on inversion of control and dependency injection tends to throw around big words which have the effect of obfuscating rather than enlightening, the basic concepts is simple. You've probably already done it. Now you know what it's called and why it's useful, and probably also can find ways to use it where it helps.

(See also: how you select a DBD with DBI.)

It's possible—even advisable—to analyze buggy code to see if the bugs point to misfeatures in the interfaces. Plenty of bugs come from misunderstandings of requirements, but plenty of bugs lurk in interfaces which are unclear or difficult to use correctly.

Consider this snippet of code:

use Moose;

has 'summary', is => 'rw';

around summary => sub
{
    my ($orig, $self) = splice @_, 0, 2;
    return $self->$orig() unless @_;
    my $summary       = shift;

    return $self->$orig() unless length $summary;

    my $content = $self->remove_markup( $summary );
    return '' unless $content;

    $self->$orig( $content );
};

This snippet of Moose-based code declares an attribute named summary with an autogenerated accessor and mutator of $object->summary. It also wraps this autogenerated method with code to sanitize any data set through the mutator by removing markup.

Everything seems straightforward, but consider all of the parts of the interface which might go wrong. (This is not a criticism of Moose, which does the best it can; it's an exploration of language and API design which just happened to come up as I debugged some very buggy code I wrote.)

my ($orig, $self) = splice @_, 0, 2;

Perl 5's minimal argument handling nevertheless has some common idioms. One of those is that $self is the first argument to all instance methods. Is this an instance method? Sort of. It's also a wrapper around an instance method which may or may not need to redelegate to the wrapped method in various ways. Thus this wrapper needs access to the wrapped method somehow.

The Moose developers chose to provide the wrapped method as a code reference passed as the first argument to the wrapper, mostly because this is the least worst approach [ 1 ]. You can't really add syntax to Perl 5 within the scope of this wrapper even with Devel::Declare without rewriting the op tree of the method itself to get at the wrapped method somehow, because Perl 5's internals know nothing about this technique. Besides that, D::D is a heavy hammer to pull out.

(Again, sometimes adding syntax to Perl itself makes otherwise intractable problems simple.)

Passing this parameter at the end of @_ would lead to the even more awkward idiom of:

around 'summary' => sub
{
    my $orig = pop;
    my $self = shift;
    ...
};

The splice approach is the least worst way I've found to deal with the situation where I have variadic argument counts to differentiate between attribute access and attribute mutation.

One problem remains: this wrapper only matters for the mutation case. I'm starting to believe that variadic redispatch within user code is a code smell of the rotting vegetation variety.

If Perl 5 supported multiple dispatch even only on the number of arguments provided to a method, this code would not be a problem. The wrapper could apply only to the case where the method took one or more arguments and would leave the accessor case of zero arguments unmodified.

The most right solution in my case is to change the name of the mutator to set_summary()... except that that's only the obvious change in the local case. For the sake of consistency with the rest of the system, I ought to change every mutator's name to set_foo: a large change. It's worth doing, but I should have done it from the start, if I'd predicted that a strict separation between setting and getting were necessary to reduce the possibility of bugs in wrapped methods.

Consider also what happens if you don't take the splice approach but do follow the common redispatching idiom:

around summary => sub
{
    my ($orig, $self, @new_values) = @_;
    return $self->$orig() if @_ == 2;

    ...

    $self->$orig( @_ );
};

I made that mistake too. It was easy enough to spot after DBIx::Class gave me odd errors about not knowing how to store a coderef in a column that should only contain text.

Again, this isn't Moose's fault. It does the best it can with the features Perl 5 provides. Yet imagine if Perl 5 knew what an invocant was and understood something about wrapping. (You can even unify redispatching to an overridden parent method in this—and while you're at it, fix the SUPER bug.) The syntax might look like:

around 'summary' => method
{
    return $self->*around::original() unless @_;
    my $summary       = shift;

    return unless length $summary;

    my $content = $self->remove_markup( $summary );
    return '' unless $content;

    $self->&around::original( $content );
};

Perhaps the star selector syntax isn't the right syntax to identify a different dispatch mechanism than simple name-based lookup, but you get the idea.

Until then, I'll be looking for an easy way to rename all of my mutators.

[1] Least worst? Consider the alternatives. If Moose provided an internal mechanism to set attributes directly (and not through direct hash access, which is prima facie the worst possible access mechanism), it would likely need access control to restrict direct mutation to declaring classes and their immediate descendents or applied roles. Figure out that system. Now strengthen it against everyone who suddenly discovers that a quick hack here and there will speed up synthetic benchmarks by a couple of percent and break the carefully constructed encapsulation, and ... well, welcome to Perl 5. (back)

One of my few ambivalences toward DBIC is that it commingles database operations (create, retrieve, update, and destroy) with model-specific operations ("level up this character", "sell the magic sword", "deactivate the smoke trap"). This is useful in that my objects have smart persistence and updating and relationship tracking provided by their DBICness, but it's difficult in that sometimes I want a stricter separation of database concerns from data model concerns.

Suppose I want to test that a document processing model can successfully filter out many hundreds of thousands of algorithmically generated permutations of unwanted data. Suppose, as is the case, that these documents are DBIC objects normally retrieved from a database. A test database might seem like just the thing for data-driven testing, but in this case it seems more work to generate a large transient database full of hundreds of thousands of documents to satisfy the constraints of DBIC.

In other words, I want a way to generate a new model object programmatically without having to store it in and retrieve it from a database. (Alex Hartmaier and Matt S. Trout both reminded me of the $rs->new_result() method which creates an object given the appropriate resultset but does not store it in the database.)

I do have tests which verify that the storage concerns behave as appropriate. They represent a small investment in mock data which sufficiently exercises all of the cases my code needs to handle. My other test concerns have little to do with the database itself. They care about what the model objects do with their data, not how they get that data.

Thank goodness DBIC is compatible with Moose.

I extracted all of the non-database model methods into a role. That role requires the accessors the database model provides for persistent data. I created a very simple dummy class which has those necessary attributes and performs that role. Then I wrote a very, very simple loader function which generates the necessary data algorithmically, instantiates instances of the dummy class, and tests the role's methods.

(I plan to write a longer article for Perl.com showing example code.)

The result is a model class which consists solely of the code generated from DBIx::Class::Schema::Loader and a with statement to apply a couple of roles. The tests are in two parts: one tests the model-specific code. The other tests the persistence-specific code, itself a combination of the generated code and another role which collects the remaining persistence behavior.

Even though both roles have an obvious coupling in terms of providing necessary behavior to the model (and to each other), decoupling them in terms of storage provides much improved testability—and, I suspect, more opportunities for reuse and genericity throughout the system.

While explanations of the value of Perl roles often focus on reusability and the separation of similar concerns from otherwise unrelated classes, roles can also provide a separation between dissimilar behaviors and concerns. In this case, it doesn't matter to the role methods where the data comes from (a live database, a testing database, an algorithm, hard-coded in a test case, the command line, wherever). It only matters that that data is there.

This technique doesn't always work. This technique isn't always appropriate. This technique does not replace verification that your behavior roles interoperate with your persistence models appropriately. Even so, it has simplified a lot of my code and improved my tests greatly.

I've been refactoring an existing system from a prototype to a well-designed and maintainable system. Part of that process is finding a better design. Part of that process is improving test coverage to find and fix bugs.

Something fascinating happens during that process if you pay attention.

Because the design of the project is fluid at this point, I have free reign to redesign to make testing easier. Because test coverage is spotty in places, redesigning to make testing easier is almost a necessity. Fortunately, the project is small enough and communication with the stakeholders easy enough that determining what the code should do where it's not clear is easy.

Plenty of people criticize test-driven development as if it were an iron-clad rule of law, where every line of code must have at least one associated test case, and if you do that, you get a badge and some grape flavored juice to hold you over until the spaceship comes. If that were the only benefit, that would indeed be a fair criticism—but how do you test this code?

sub save_to
{
    my ($self, $dirpath) = @_;

    my $filepath = File::Spec->catfile( $dirpath, $self->cached_file );

    my $i = Imager->new( file => $self->full_file );
    return $self->invalidate() unless $i;

    my $width  = $i->getwidth;
    my $height = $i->getheight;

    # verify size
    return $self->invalidate()
           if MyCoolApp::Exclude->image_size( $width, $height );

    my @resize = $self->should_resize( $width, $height );

    try
    {
        $i = $i->scale( @resize ) if @resize;
        $i->write( file => $filepath );
        unlink $self->full_file;
        $self->state( 'FETCHED' );
        $self->full_file( $filepath );
    }
    catch { warn $_; $self->invalidate };

    return 1;
}

This code will not be trivial to test. Perl 5 has a few idioms for testing the existence of files in the filesystem. With some careful use of File::Temp, it's possible to verify that a file which did not exist before the call being tested exists after the call, but the rest of this code has other difficulties.

The hardest part of this code to test comes from the hard-coded call to the Imager constructor. I could hijack the loading of Imager with Test::MockObject::Extends and test all of the paths through this method by changing the mock Imager's behavior, but that would couple of the details of the test too tightly to the details of the code being tested.

This method's biggest problem is that it does too much. It creates an Imager object. It tests to see if the image represented should be excluded from saving. It resizes the image. Then, finally, it saves the image. All of this behavior made sense when the only operation performed against this API was "save this image, if applicable" but these operations are very obviously distinct operations confused in one spot when trying to test this API.

This is where the people who dismiss TDD as a only brain-numbing checklist of rote behavior are wrong. I'd never let myself write a method with this many separate concerns if I had written it with TDD, but I can admit that I'd write code with this many separate concerns if I'd written prototype, just get it working, see of it's possible code. (I did write the exclusion code.)

Years ago I would have forged ahead to dummy up an entire Imager, just to get full test coverage. Fortunately, this code uses Moose.

Moose?

The design patterns people have a term called Inversion of Control and a specific grouping of techniques in a family called Dependency Injection. You can get lost in the architecture astronaut nature of the discussion, but the basic principle is sound: avoid hard-coding dependencies in your code.

In other words, the most suspect part of this method is the line:

    my $i = Imager->new( file => $self->full_file );

... because of the very tight coupling between this module and Imager.

You may think now "Wait, but you need the behavior of Imager for the code to work correctly!" and you're correct to do so. That's absolutely right. This code needs the behavior of the Imager object, and this code is tied to the API of the Imager object, but this code itself does not have to be responsible for instantiating the Imager object.

If you already knew that, give yourself a gold star. If you're now thinking "Yeah, but why does that matter?" or "What?", read the previous paragraph again. It's subtle, but it's important.

Moose supports something I'll call synthetic attributes. They're attributes of your object built from the values of other attributes. You're likely to encounter them when they're built lazily. Consider if I added an imager attribute to this class:

has 'imager', is => 'ro', lazy_build => 1;

sub _build_imager
{
    my $self = shift;
    return Imager->new( file => $self->full_file );
}

That performs one very nice feature of decoupling the creation of the Imager object from the method I want to test. If that's the only value of the new synthetic attribute, it's still useful. Yet that attribute has greater serendipities:

sub test_save_to
{
    my $module = shift;
    my $image  = $module->new(
        filename => 'my_file.gif',
        state    => 'FETCH',
        validity => 10,
        imager   => undef,
        validity => 0
    ) );
    my $result = $image->save_to( 'some_dir' );

    ok ! $result, 'save_to() should return false without valid imager';
    ok ! $image->is_valid, '... invalidating image';
}

Extracting this hard-coded call into a synthetic attribute with Moose allows code which creates these image objects to provide their own values for the imager synthetic attribute. The test is now in control over what's provided. It can use a real Imager object or a mocked Imager object. It can allow the lazy builder to create a real object.

One small act of extraction (and, as usual, intelligent default behavior in Moose) has turned a method which is difficult to test into something much more tractable. Testing even the hard-coded call itself was possible in Perl 5, especially with the utilities of CPAN modules which let you scrounge around in namespaces, but little bit of care and a little bit of abstraction and decoupling and encapsulation make the code and the tests more robust.

(For full disclosure: I had a fleeting thought at first "With imager() as an accessor, I can subclass the class I want to test locally, override that method, and return what I want. That's not strict dependency injection by any means, but I can explain it in terms of DI as a metaphor." A moment later I realized how easily Moose supports DI if you structure your code in this way and was glad I don't have to overdesign the tests.)

This is the sort of writing I didn't have time to get into in detail in Modern Perl: the book, but is well within the scope of what the potential authors of the Moose book want to cover and very much in the spirit of what a new Perl Testing book will discuss. Hint, hint.

I amused myself far too much by reading A Class:: module I don't want to write and Can I please have *simple* modules?, for example.

My rule is simple: Moose has made me forget many, many little problems that I used to have in the same way that Perl has made me forget many, many little problems that I used to have. (I'd even forgotten I'd written an article called "Seven Sins of Perl OO" five years ago. With Moose, only one of those sins is still a problem.)

If you, like me, first encountered the term in the popular working programmer literature of the '90s (or if you've read Microserfs), you might have heard that the purpose of OO is to enable the deliberate reuse of unique and well-defined software components.

If you learned structured programming, whether earlier in the history of programming language development or because objects were a fad until Sun's marketing budget displaced IBM's marketing budget, you might have heard that the purpose of functions is to enable the deliberate reuse of unique and well-defined software components.

I write this knowing full well that in a few minutes I need to modify a piece of code which uses multiple CPAN modules, many of them defined, tested, deployed, and loved very well. With that in mind, I still believe that reuse as a primary design principle is often a fool's goal.

Last week's example of abstraction versus mock objects elicited thoughtful comments from Zbigniew Lukasiak and Andreas Mock relating to the single responsibility principle, design patterns, and dependency injection. These concerns are important in the sense that contrast and shading and hue are important concerns in creating great artwork.

Yet somehow the false notion that design is primarily an exercise in identifying, codifying, and extracting reusable components is pervasive, especially in literature intended for novices. (Perhaps novices sigh with relief when they finally get a single function or class working correctly, thinking to themselves "Thank goodness I shall never do this again!")

In 1999 I might not have created a helper class to solve my testing problem from last week. (Why multiply entities unnecessarily? I'm not writing ravioli code here!) I might have struggled through mocking DBIC row objects and priding myself on the thoroughness of my testing of all of the picky little details.

In 2011 I congratulate myself more on the code I don't have to write—not because I'm reusing implementation details but because I've minimized the details any particular piece of code has to understand.

That's why the Modern Perl book explains functions primarily as a way of naming and encapsulating discrete units of behavior and objects as discrete identities with single, well-defined responsibilities.

Reuse is well and good when it happens, but reuse happens best because of sensible design. It's not the primary goal of software design. Nor should it be. It's a serendipity born of the fundamental design concerns of naming and encapsulation.

Let us teach these useful truths to novices instead of the airy lies they will waste their time pursuing.

If you, like me, first encountered the term in the popular working programmer literature of the '90s (or if you've read Microserfs), you might have heard that the purpose of OO is to enable the deliberate reuse of unique and well-defined software components.

If you learned structured programming, whether earlier in the history of programming language development or because objects were a fad until Sun's marketing budget displaced IBM's marketing budget, you might have heard that the purpose of functions is to enable the deliberate reuse of unique and well-defined software components.

I write this knowing full well that in a few minutes I need to modify a piece of code which uses multiple CPAN modules, many of them defined, tested, deployed, and loved very well. With that in mind, I still believe that reuse as a primary design principle is often a fool's goal.

Last week's example of abstraction versus mock objects elicited thoughtful comments from Zbigniew Lukasiak and Andreas Mock relating to the single responsibility principle, design patterns, and dependency injection. These concerns are important in the sense that contrast and shading and hue are important concerns in creating great artwork.

Yet somehow the false notion that design is primarily an exercise in identifying, codifying, and extracting reusable components is pervasive, especially in literature intended for novices. (Perhaps novices sigh with relief when they finally get a single function or class working correctly, thinking to themselves "Thank goodness I shall never do this again!")

In 1999 I might not have created a helper class to solve my testing problem from last week. (Why multiply entities unnecessarily? I'm not writing ravioli code here!) I might have struggled through mocking DBIC row objects and priding myself on the thoroughness of my testing of all of the picky little details.

In 2011 I congratulate myself more on the code I don't have to write—not because I'm reusing implementation details but because I've minimized the details any particular piece of code has to understand.

That's why the Modern Perl book explains functions primarily as a way of naming and encapsulating discrete units of behavior and objects as discrete identities with single, well-defined responsibilities.

Reuse is well and good when it happens, but reuse happens best because of sensible design. It's not the primary goal of software design. Nor should it be. It's a serendipity born of the fundamental design concerns of naming and encapsulation.

Let us teach these useful truths to novices instead of the airy lies they will waste their time pursuing.

I like how KiokuDB persists plain old Moose objects. If you understand how your object graph works, there's little effective difference between a Moose object you create yourself or one you retrieve from your object store. Less so DBIC objects, in my experience. This isn't a flaw. It's merely a difference.

The difference became apparent when I started to port a fundamental test file as part of this migration. This test file exercises the document parser. As part of the application, the document parser uses a variant of the Readability algorithm to extract the meaningful portions of HTML documents. The test file itself uses a data-driven approach, where a t/files/documents/ directory holds multiple YAML files containing real data we've encountered along with the expected (correct) results.

#! perl

use Modern::Perl;
use Test::More;
use YAML::XS;

use lib 't/lib';

__PACKAGE__->main( 'App::DocParser', @ARGV );

sub create_docparser
{
    return App::DocParser->new(
        url       => 'http://www.example.com/some_url',
        url_base  => 'http://www.example.com/',
        @_,
    );
}

sub main
{
    my ($test, $module, @files) = @_;
    use_ok( $module ) or exit;

    if (@files)
    {
        $test->run_file_content_tests( $module, @files );
    }
    else
    {
        $test->test_find_content_files( $module );
    }

    done_testing();
}

sub test_find_content_files
{
    run_file_content_tests( @_, glob 't/files/documents/*.yaml' );
}

sub run_file_content_tests
{
    my ($test, $module, @files) = @_;

    for my $file (@files)
    {
        my $example   = YAML::XS::LoadFile( $file );
        my $docparser = create_docparser( %$example );

        like $docparser->content, $example->{content_regex},
        "DocParser should find matching content for $example->{desc}";

        is 0 + @{ $docparser->links }, $example->{link_count},
            '... with the right number of links';
    }
}

The previous version of this code created App::Document objects—instances of the same class persisted into the object graph with KiokuDB—and called functions in the DocParser namespace to find various pieces of context. A straightforward port of this test file to DBIC meant I'd have had to figure out how to instantiate dummy DBIC-alike objects representing documents to pass to the DocParser functions.

I thought about that for a few minutes. I pondered reaching for Test::MockObject::Extends. Then I didn't.

What's important in this test isn't that App::DocParser conforms to a specific interface governed by the persistence layer. That's irrelevant. Its only relevance is how well it identifies and extracts relevant information from given data.

I turned that namespace full of utility functions into a class. I gave that class the specific attributes on which it needs to operate, effectively giving instances of the class the responsibility to manage the data on which they operate. You can see the result: a simple constructor call creates a DocParser object without worrying about setting up test data in an example database or building a mock object framework which behaves sufficiently like a DBIC row object.

I really like the object responsibility pattern which Moose seems to encourage, where you create an object by passing all relevant data to its constructor and then operate on that data as needed (perhaps governed by laziness in attribute accessors). You get the benefit of being able to assume that a constructed object is in a safe and sane state as well as a decoupling of data dependencies.

Once I realized that this approach would lead to better code, it took twenty minutes to Moosify the document parser and one minute to change a couple of lines of code to port the test to the new framework. I count that as a sign of effective design.