Why not modify add_txn to reject undefs?

The code in review is:

while (my $stock = $stock_rs->next)
{
    my $pe_update = $self->analyze_pe( $stock );
    $stock_txn->add( $pe_update ) if $pe_update;

    my $cash_yield_update = $self->analyze_cash_yield( $stock );
    $analysis_txn->add( $cash_yield_update ) if $cash_yield_update;
}

... and the near duplication obscures (to me) the point of the code. Both Aristotle and Chas. are right—perhaps it's clearer to allow $transaction->add to do nothing when it receives nothing. I write "perhaps" because I see the appeal of that change, but I'm not sure I like it.

As usual with my software, the system has a fundamental design principle: either succeed in full or do nothing. It's fine to skip half of the analysis steps if the data just isn't there. (The project as a whole can succeed if it's only 60% correct; the joy of a margin of error. It's a lot more accurate than that.)

I take this principle to mean that robustness is more important than completeness. Skipping bad data and moving on is perfectly fine. The next run may improve transient errors, and catastrophic errors will require human intervention anyhow.

When these principles translate into design, I prefer to handle errors at the point of detection and not spread error handling throughout the system. All of these analysis methods should return something. When they succeed, they return a hash reference mapping column names to values in a database table. When these methods fail—whether the existing data isn't sufficient to calculate updated values or something else went wrong—they return;. As you well know, that's an empty list in list context and undef in the scalar context of the example code.

Why add nothing to a transaction when I know there's nothing to add? Yes, add() could check that it has nothing to do and do nothing, and that's fine, but it seems like that expands the behavior of add()'s API to include caller errors. Then again, the add() method must check that each hash reference contains a value for the transaction's bound primary key, or it will generate buggy output.

I suspect that both Aristotle and Chas. have in mind Postel's Law:

Be generous in what you accept and picky about what you emit.

The result might look something like:

while (my $stock = $stock_rs->next)
{
    $stock_txn->add(    $self->analyze_pe( $stock )         );
    $analysis_txn->add( $self->analyze_cash_yield( $stock ) );
}

This change has an advantage: it only necessitates a change in the add() method. All of the analyze_*() methods can continue to work as implemented.

Of course, there's a slight performance penalty to doing this. In my case, it's immaterial, but it wouldn't be present with macros. This is an IO-bound application anyhow, and the transaction manager exists to avoid very real, very measured bottlenecks.

Finally, Aristotle's mention of the null object pattern was about real objects, and not methods which return empty lists or hash references. If that's your style, good for you—but it's not mine in this case. While it's not obvious from the small snippets I've posted so far, the responsibility of the analysis methods is smaller in scope than the responsibility of the transaction objects. Coupling transaction management to the analysis methods—in as much that they have to know about transactions to return the right objects—would turn the design of the system inside out. The result would very likely not be an improvement.

cpanm and perlbrew should not run tests by default.

His points are reasonable, but his complaints are mostly about side effects and not the real problem. (I should clarify: the real problem I encounter.)

If running tests slow down installs, speed up the tests. (Do you want to get the wrong answer faster? Easy: it's 42. No need for a quantum computer to do the calculation in constant time. This algorithm is O(0).)

If running tests exposes the fragility of the dependency chain, improve the dependency chain.

If dependency test failures prevent the installation of downstream clients... this is a weakness of the CPAN toolchain. A well-written test suite for a downstream client should reveal whether bugs or other sources of test failures in a dependency affect the correctness of the client.

Note the assumptions in that sentence.

Anyone who's experienced the flash of enlightenment that comes from working with well tested code and who's shared that new zeal with co-workers has undoubtedly heard the hoary old truism that testing cannot prove the complete absence of bugs. It's no less true for its age, though it's also true that good testing only improves our confidence in the correctness and efficacy of our code.

For me, a 95% certainty that my code works and continues to work for the things to which I've tested it is more than sufficient. I focus on testing the things I'm most likely to get wrong and the things which need to keep working correctly. (I don't care much about pixel-perfect placement, but I do care that a book's index uses the right escapes for its data and markup.)

Without tests running on the machines themselves in the environments themselves where I expect my code to run, I don't have that confidence.

Put another way, I'm either not smart enough or far too lazy to want to attempt to debug code without good tests. That's why I write tests, and that's why I run them obsessively. That's good for me as a developer, and you're getting the unvarnished developer perspective.

I also care about the perspective of mere users. (Without users, we're amusing ourselves, and I can think of better ways to amuse myself than by writing software no one uses.).

Yes, an excellent test suite can help a user help a developer debug a problem. Many (most?) CPAN authors have had the wonderful experience of receiving a bug report with a failing test case. Sometimes this even includes a code patch.

Not all users are developers of that sort, nor should they be.

The CPAN ecosystem has improved greatly at automated testing and dependency tracking, but we can improve further. What if we could identify the severity of test failures? (We have TODO and SKIP, but they don't convey semantic meaning.) What if we could identify buggy or fragile tests? (My current favorite is XML::Feed tests versus DateTime::Format::Atom because it catches me far too often, it doesn't affect the operation of the code, and it's a stupid fix that's lingered for a few months.) What if the failures are transient (Mechanize relying on your ISP not ruining DNS lookups for you) or specific to your environment (a test suite written without parallelism in mind).

As Jonathan rightly implies, how do you expect an end-user to understand or care about or debug those things?

I'm still reluctant to agree that disabling tests for end-user installations is the right solution. I want to know about failures in the wild wider world. I want that confidence, but I can't bring myself to trade away that confidence for the sake of a little more speed of installation.

Yet his point about lingering points of fragility in the ecosystem are true and important, even if the proposed solution of skipping tests isn't right. Fortunately, improving dependency management and tracking and use and testing can help solve both issues: perhaps to the point where we can run only those tests users most care about and identify and report material failures in dependencies.

You may have read my Controlling Test Parallelism with prove and Parallelism and Test Suites. I still have Test::Harness parallelism enabled by default on most of the machines where I install my own Perls. While I haven't yet filed tickets and tried to write patches for modules which need a little help to run tests in parallel, I've only found a few lately that need work. That's nice—having a module install through cpanm in five seconds is a lot better than ten seconds or more. (I like cpanm because it's fast and quiet, and part of its speed comes from not printing to the console.)

I like instant feedback.

Like Steven, I noticed quite a while that installing a custom Perl through perlbrew takes a while, but then I remembered that a lot of work went into the Perl 5 test suite to make tests run in parallel. (We did something similar with Parrot several years ago, and it changed the way I work forever.)

To run core tests in parallel, set the environment variable TEST_JOBS=n, where n depends on your computer. I use a value of 9 on a quad-core machine; in practice, that tends to keep the CPU busy while not blocking anything too long on IO. You can set it globally in your shell's configuration file or create an alias or wrapper for perlbrew.

As most of the time spent compiling and installing Perl 5 through perlbrew goes to running the test suite, this has saved me a measurable amount of time.

This problem is an example of the pattern of Why Writing Your Own DSL is More Difficult Than You Think. Certainly Template Toolkit is among the better templating systems (I've written a couple myself), but it exhibits problems endemic to the process. (Then again, so does PHP. Now multiply that by the fact that some people use templating systems written in PHP and if you have to lie down for a while before the feeling passes, please accept my apologies.)

The semantics of Template Toolkit are great, when they work, but then everything's great when it works the way you expect. Robust software handles the cases you don't expect with aplomb, or at least without a boom.

A simple workaround for Template Toolkit is to avoid the fallback from potential method lookup to keyed hash access when dealing with an object. In other words, if $blessed_hash->do_something() fails, try $blessed_hash->{do_something}.

... except that that doesn't work when you want to call virtual methods on unblessed references, such as calling methods on arrays or hashes.

Another option is to change the syntax such that calling a method is visibly different from accessing a member of an aggregate. Perl 5 does this. It works pretty well, in the sense that if you use the right operator (access element versus invoke method), you've expressed your intent in a visually unambiguous fashion).

... except that people complain about the Perl 5 dereferencing arrow quite a bit. (Okay, you don't need an arrow to do this; as the Modern Perl book explains, the postfix indexed access or postfix keyed operators of {} and [] determine the type of operation effectively.)

... and except that one of the design goals of Template Toolkit was to be robust in the face of changing values provided to the template, such that it provides a loosely coupled interface for the data it expects. That's a fine goal, but it isn't free.

Here's the thing, though. The last time I looked, Template Toolkit compiles templates into Perl 5 code as an optimization. (The last template system I wrote did the same thing, but not as well. We should have used TT, but in our defense, TT didn't exist then.) This transliteration/compilation stage must be very, very cautious to allow standard Perl debugging and introspection tools to treat this generated code correctly. That is to say, I don't want to debug a big wad of generated code. I want to debug the code I actually wrote.

As usual, the solution is another layer of abstraction.

Perl 5 exists in two forms. The first is the source code you and I write. The second is the optree which the Perl 5 VM executes. There's nothing in between. You have one or the other. When your code runs, you have the optree, and the optree has references to the relevant location in the source code it came from, but the correspondence is often less useful than you might like.

While the generated code from Template Toolkit could include the correct file and line positions from templates, that's again less useful than you might like. (It's useful, but it doesn't solve every problem.)

If Perl 5 had instead an intermediate form separate from raw code and raw optrees, something more suitable to introspection and manipulation, we could produce tools which worked with this intermediate form to improve debugging, introspection, and better code generation.

We could even inject new code to add features (fall back to attribute access; prevent the fallback to attribute access) to code, even within lexical scopes. That is to say, we could manipulate how libraries behave from the outside in, and ensure that our changes would not leak out from our desired scopes.

It's certainly possible to replace the Perl 5 opcodes yourself, if you're comfortable reading Perl 5 source code, writing XS, relying on black magic, and dealing with strange issues of thread safety and manipulating global or at least interpreter-global values in a lexical fashion (while dealing with the fact that use is recursive in a sense)—but isn't Perl about not making people write C to do interesting things?

Certainly this isn't a technique you'd use every day, and it's not obviously a way to make Perl 5 run faster (though many optimizations become much easier), but the possibility for better abstraction and extension and correctness has much to recommend it.

And, yes, Lisp demonstrated this idea ages ago.

I want to inspect the call stack within a helper module, but I don't care about the call stack within the module. I want to use lots of little helper functions, because that's good design, but caller works against that. Looking up the stack means keeping track of the magic number of call frames within my module I currently use, and that's one more thing to update when I change things.

That's structural code highly coupled to the arrangement of other code, and if that doesn't wrinkle your nose with the subtle aroma of fragility and peril, I don't know what will.

Imagine if instead:

my $caller_object = capture_caller_state();

while (my $call_frame = $caller_object->next)
{
    next if $call_frame->is_eval;
    say $call_frame->location_as_string;
    my $next_frame = $call_frame->previous;
    ...
}

Imagine if you could pass this object around.

I know things get complicated if you pass this object up the call chain, but stack unwinding is a solved problem in that anyone capable of recognizing the problem should be able to figure out cheap and easy ways to fix it.

Alas, Perl 5.14 doesn't have this feature, and it's probably too late for 5.16 to get it, so for today I'm stuck imagining what might be.

(If you've never thought about this sort of thing before, you owe it to yourself to learn more about Continuation Passing Style, which is at least an order of magnitude more mind-bending at the start and at least two orders of magnitude more useful.)

You can already preorder Modern Perl: 2011-2012 from Amazon and it'll be available from other bookstores in the next couple of days. I expect copies to ship by early next week.

What's new?

I closed every bug filed against the previous edition—a handful of typos, a few confusing sections, and some suggestions on improvements. I took most of these suggestions.

I also revised almost every paragraph of the book, even at the sentence level. While I'm proud of the first edition, this new edition is clearer, shorter, more concise, and more accurate.

I removed some sections that didn't work. This didn't amount to much, but some of the things I recommended made little sense anymore and some of the experimental modules I mentioned turned out to be less useful than hoped. You won't miss them. This includes a big revision of the smart matching section in anticipation of changes in 5.16 and 5.18 (simplicity is the rule of the day; the simpler your use of the operator, the less risk you face for confusing behavior and edge cases).

I also added explanations for the new features in Perl 5.14, especially the /r modifier on regular expressions, and prepared for further features added in 5.16.

You might notice subtle improvements in formatting; the font size used for code examples is larger and callouts are much more attractive. We're still experimenting with the best way to format books to make readable text while avoiding brick-sized tomes, so feedback is more than welcome.

As with the first edition, we'll make electronic versions available for free from the Onyx Neon site. We hope to have ePub versions ready in the next two weeks. We'll also put raw HTML online on modernperlbooks.com at the same time (this also allows us to link to translations with greater ease). In addition, we're making most of our publishing tools available as CPAN distributions, including the formatting modules (Pod::PseudoPod::DOM) and the rendering pipeline (Pod::PseudoPod::Book). This will allow you to make your own version of the book in other languages if you desire, or to create new and attractive books in many formats. We'll make our own Kindle version, rather than using Amazon's conversion tools (they didn't work as well as we'd hoped), and we hope to get the book in the Nook store as well.

We plan to release a new version of the book late this year or early next year to cover 5.14 and 5.16. We'll almost certainly drop support and mention of 5.10.

We couldn't have done this book without you. Thanks for reading. Please share with your friends.

You can do it.

Sure, it's easy for me to say that. I've written a few things about Perl a few people have read. I have a few patches in a few projects and a couple of modules on the CPAN myself. (You're reading this, aren't you? So I have at least one reader. Thank you for your time!)

Even with all that in mind, two questions still come up far too often, and I think they prevent or delay us from providing code and documentation for other people to use freely:

• Does anyone care?
• Is it good enough (yet)?

You can see this attitude in the recurring debate over the responsibilities of authors. While some people say it's irresponsible or childish or unprofessional to upload code without full test coverage or complete documentation or an iron-clad promise to respond to every bug within 24 hours with a fix, an apology, a new release, and a sandwich (and yes, I exaggerate for effect), we're doing each other a disservice setting these demands on ourselves and each other.

Yes, we should do our best.

... but yes, we can start from something small and incomplete and refine it in smaller steps. It doesn't have to pass every test on every platform known to man if you can get feedback on where it fails and release a new version tomorrow. It doesn't have to have every feature you plan to add, if it does something useful that other people might want to use today. It doesn't have to have every option documented if it's usable right now.

I'm not suggesting that we allow ourselves to be irresponsible with what we share, but I am suggesting that we can allow ourselves the freedom to share a little earlier and a little more often. We have the ability to upload new code every day (every hour!) if we want.

So what if it's not perfect? Even if you waited until you thought you'd achieved perfect, you probably would miss, even by a little bit.

Start small. Do your research—work to your best quality—but let yourself hit smaller goals and make things a little bit better a little more often. Share earlier. This is a lot better for everyone.

Instead, I see Modern::Perl as enabling the core features I wish were on out of the box by default in Perl 5. While it'd be nice to pull in Try::Tiny and sometimes I might wish for fatal warnings, the former is not a core module and the second isn't something I use in every program.

Consequently the module hasn't needed much maintenance. Yet I've added a couple of missing features that should keep it useful and usable into the future. I uploaded a new version yesterday and will upload one more today with a little bit more polish.

First, it now requires autodie as a distribution dependency. It doesn't load autodie, but installing Modern::Perl on 5.10.0 will also install autodie. (Anything 5.10.1 and newer includes autodie in the core.) You don't have to use it, but now you can rely on any Perl considered modern to have autodie available.

Second, it now loads IO::File and IO::Handle so that you can call methods on lexical filehandles without having to load either manually. Perl 5.14 fixed that usability niggle, but Modern::Perl fixes this for people using 5.10 or 5.12. (Why both? I can never remember which one superseded the other in 5.12, but better safe than sorry. I welcome a patch to load one over the other with a version check—and please test carefully.)

Third, I added unimporting support so that you can write no Modern::Perl; within a scope to disable strictures, warnings, and language bundle features. It's an all or nothing switch and will remain that way, but I can see this being useful in specific cases, especially when updating older code in stages.

Finally I added date support to importing. If you write use Modern::Perl; you'll get the features of Perl 5.10 (with the caveat that if you're using Perl 5.11.3 or newer, you also get the unicode_strings feature, of which all you're likely to notice is that Unicode strings work better in more places).

Yet for forward compatibility, you should be using:

use Modern::Perl 2012;

... which enables 5.14 features. You can use 2009 and 2010 to get 5.10 features and 2011 to get 5.12 features. If you use the wrong date on the wrong version of Perl 5, you'll get an error, which is as it should be.

Next year I'm likely to drop support for Perl 5.10, in which case you'll probably get an error message that that year isn't modern enough, but I could be convinced to do a version check instead. I haven't decided. The tradeoff is between providing a minimal module suitable for use in programs which helps people write better code from the start and between telling people what they should and shouldn't use. Besides all that, the relevant code is only a couple of dozen lines of very simple Perl. Anyone reading this can reimplement it almost trivially.

Of course, this all comes about because the Modern Perl book goes to the printer today. We're very proud of the new 2011-2012 edition which concentrates on Perl 5.12 and Perl 5.14. It addresses all of the known typos and confusing parts of the previous edition, covers new features in Perl 5.14, and is, from all reports, even better than the first edition.

The book should be in stores in the next week or so, and we'll have electronic editions up this month for free download and redistribution. (We hope you tell lots of people to buy the print edition because it's great and more people need it on their desks, but sharing is caring and we support that.)

A few months ago I exchanged emails with JT Smith about my idea for a talk this year. I've mentioned in passing a few times a small side project my business is investing in. It's a side project, deliberately minimal, and—from the development side—definitely the kind of skunkworks, just get it working, maintain it as little as possible and let it run uninterrupted software that you're likely to find.

That doesn't mean it's quick or dirty. That doesn't mean it's not tested well, or that it has a slapdash design. All it means is that the most important criterion for any design or implementation decision is "is this the simplest thing that could possibly work" instead of "is this elegant" or "what's the standard modern Perl orthodoxy for this problem".

So far the results have been enlightening.

I don't want to give away too many of the details of my talk (if it's accepted), but here are two small hints which may or may not help you.

First, just because a good ORM such as DBIx::Class makes searching and manipulating existing data easy doesn't make it the best way to insert big batches of new data.

Second, while LWP and especially WWW::Mechanize are great tools for automating the behavior of a web client, sometimes wget or curl in a shell script is quicker, easier to parallelize, and more robust.

(As a bonus, consider also that if you're parsing semi-structured data out of HTML that removing all of the HTML is sometimes even easier than using a real HTML parser or even CSS selectors. Sure, semantic markup helps when you can rely on it, and sure, using a regex to remove HTML tags is a bad idea, but there are ways to turn HTML into plain text quickly and easily without doing anything on your own.)

In the first web bubble, the Silly side of Silicon Valley chased vanity metrics such as "the number of eyeballs on the site" and "brand awareness" and "unique visitors". Those numbers are only interesting when you can correlate them to producing value for customers and bringing in real cash in the form of revenue.

I've enjoyed the book The Lean Startup by Eric Ries because he offers a much better mechanism to track the success or failure of any attempt to produce real value to customers. While split testing (or A/B testing) is useful to see how small changes lead to different customer behaviors, Ries recommends cohort analysis, where you can see the behavior of real customers through the sales funnel and correlate the X-axis with individual changes to your business or product.

That means tracking customer behavior. If you're building some sort of software as a service product, and if the mechanism of delivery of that product is primarily a web site, you probably already know the punchline.

Assume I already know how to identify and log events for each salient customer action type. (I've built that kind of system before.) Assume I don't want to collect personally identifiable information (I don't). Assume I'm using Plack and its middleware heavily, and assume I'm happy using Catalyst as a web framework.

How can I identify unique users (with and without accounts) on a daily basis, anonymize them, but group their actions across the site such that my automated daily cohort graphs correspond with reality?

So far I've identified few points of possible contention. I can rely on browser cookies for unique identification of users if I know that user sessions have unique identifiers within a 24 hour period. (I could generate GUIDs for this, but that may be overdoing things.) I think< I also have to track the transition from anonymous visitor to authenticated user, but I might be able to convince myself that either replacing the current session or smple subtraction of successful login events from total number of unique anonymous visitors would give the right numbers.

(I also haven't dived much into how Catalyst 5.9 and Plack interact in terms of session and cookie handling. Everything's just worked, so I've ignored the details until now.)

I don't mind building such a system if necessary, but if all of the pieces are out there and available—or if someone's already built this and can give guidance—so much the better.

Have you solved this problem? If so, how did you do it? If not, how would you do it? Would you handle logging at the Plack level or the application level? Would you worry about tracking session changes? Does Catalyst need to know about this?