• Modern Perl 2011-2012 edition released (modernperlbooks.com)
• Developing parsers incrementally w/Marpa (blogs.perl.org)
• Parrot tickets migrated to @GitHub. (perlbuzz.com)
• Thank you, CPAN Testers (blogs.perl.org)
• The case of the overloaded curlies (blogs.perl.org)
• Perl more viable for webdev than ever (blogs.perl.org)

Like most data conversion projects, the challenges were less about the coding and more about making the decisions about how to massage the data between two similar systems.  For example, Trac has fields for Severity and Priority of tickets, but GitHub only has free-form tagging, so I had to create GitHub tags that correspond to Severity and Priority in Trac. GitHub's tracking system doesn't handle file attachments, so my conversion code had to make inline comments of the file attachments.

Most time-consuming of all was the conversion of users from Trac to GitHub.  We needed the issue history to have accurate user IDs on them, so I needed a big lookup table to do the job. While users like "coke" and "chromatic" have the same user IDs on both the Trac instance and GitHub, Trac user "jonathan" is "jnthn" on GitHub, and so on. Anyone I couldn't find a match for became generic user "Parrot".

The actual code to do all this is only about 200 lines of Perl code, which should be no surprise for someone who has the CPAN at his disposal. I used Net::Trac to read from the Trac instance, and the JSON module to write out JSON files in the GitHub API format. The bulk of the code is project-specific conversions to make little data tweaks like change severity to tags, and to make the output code a little more friendly in Markdown.

I have to specifically thank Rick at GitHub for helping us through this project.  I used a lot of his time with questions about how GitHub would handle my import format, and we had two test imports for us to see real results, so that I could adjust my conversion process. The final results are beautiful, and the Parrot team is excited to see this move made.

I've long been a fan of GitHub and how they help out the community, and this just adds to it.  This sort of aid to open source projects should stand as an example to other companies that work with open source.  Many companies give back to the communities of the projects on which their businesses are based.  It's fantastic to have a company willing to use human capital actually working with a project in which they have no direct involvement.  In helping us, GitHub gains nothing but the grateful thanks of the Parrot project.

The article's penultimate paragraph makes what should be an obvious point. (At least, it's obvious if you want to prevent terrorism as much as possible. If your goal is to spend lots of taxpayer money in a very flashy, showy way without worrying about efficacy, please continue.) In particular:

What the government should be doing is focusing on the terrorists when they are planning their plots. "That's how the British caught the liquid bombers," Schneier says. "They never got anywhere near the plane. That's what you want--not catching them at the last minute as they try to board the flight."

I read this article moments after sending an email commiserating about the silly (lack of) Unicode handling in a programming language which isn't Perl. Then something clicked.

One of my persistent desires for Parrot was to simplify the internals by reducing the amount of complexity and genericity in the core. In terms of Unicode, this means knowing the encoding of incoming data and the desired encoding of outgoing data, then transcoding to and from a single internal encoding. This way the core could operate on a single encoding and push the complexity of transcoding to the edges.

If Parrot hasn't changed this since I looked at it most recently, its string system requires each string to carry information about its encoding (which makes each string structure that much larger, increasing memory pressure) and each string operation to check for the need to transcode strings to mutually compatible encodings (which takes time for the comparison in every case, as well as time and memory for the transcoding in other cases).

Worse yet, string literals encoded in the source code of Parrot itself tend to have a specific encoding (ASCII or at least Latin-1 in the case of literals in the C code) and they ought to be constant, so transcoding in place isn't an option and, if you're working primarily with another encoding, that means always performing transcoding from that incompatible encoding.

It's not free to perform encoding at the edges, and you sometimes notice this when working with large chunks of data (though if you're processing multi-terabyte satellite images, treat them as binary and skip this encoding altogether), but it's the right thing to do.

The same principle applies for trusting incoming data. Secure it at the borders of the application. Don't spread those checks throughout the system. Harden the edges and don't let nonsense through. Fail early for suspicious things.

Otherwise you'll go mad trying to track down all of the possible interactions and possibilities of maliciousnesses that people could perpetuate if you lack a sane sanity policy. In other words, stop doing a lot of busy work to make it look like you know what you're doing. Do it right.

That's not the only thing Perl 5 needs, for some definition of need.

(I believe Perl 5 lacks a unified and coherent development vision, which makes this entire discussion both more interesting and less useful. Yet even an idealist has to deal with the world as it is sometimes.)

If you accept that Perl 5 needs (or "could use" or "would benefit from" or "really ought to have") a few more features, you have to answer at least two procedural questions in addition to the technical questions of "What is it?", "How should it work?", "How do we build it?", and "How do we know it works correctly?":

• Who will build it?
• Who will maintain it?

By way of analogy, consider the case of an appealing albatross in Parrot, specifically the compiler named IMCC. In Parrot's earliest days, the VM only ran bytecode. An assembler written in Perl 5 turned Parrot's assembly source code PASM into bytecode for Parrot to run.

Melvin Smith wrote an experimental intermediate compiler which added some syntactic sugar to PASM and produced Parrot bytecode either as output or directly in memory for Parrot to run. Parrot needed something like this.

It wasn't long before Melvin's IMCC found itself grafted onto the side of Parrot and used as the primary invocation and compilation mechanism. I hope I'm not mischaracterizing Melvin's opinion as including dismay that his code was a proof of concept which made assumptions and took shortcuts and wasn't exactly what he would have submitted for a shipping product. (I don't blame him one bit for writing prototype code—that's exactly what I would have done.)

Melvin stopped contributing to Parrot shortly after that point, but IMCC lives on to this day. (Again, this was prototype code which needed at least another round or two of severe refactoring before it was suitable for the sort of duties Parrot expected. As an example, I found a register use analysis algorithm which looked to have O(n12) complexity. That exponent of 12 is not a typo. Let me type it again: twelve. One dozen. Ten plus two. I managed to get it down to four in most cases.)

The problem is simple: someone (not Melvin!) dumped a big wad of code into the wrong directory in version control and now everyone has to maintain that code.

Perhaps marking this code as experimental would have helped, but other people (including me) added features and built onto it. (Dan's apologia for leaving Parrot mentions that his strategy of checking in messy code and hoping that would lure new people to help clean it up didn't work out as well in practice as he had hoped.)

In terms of Perl 5, it's important (as many people point out) to consider the question "Who will maintain this code?" That task shouldn't have to fall to the pumpking or a Nick Clark or Dave Mitchell by default, but it does.

The deeper question that Perl 5 needs to answer is "How can p5p make the whole of Perl 5 easier to maintain?"—not just to make it easier to add and support new features but to reduce the maintenance burden of existing features and to attract new contributors to help maintain code.

You can see this for yourself; open up almost any core module and see if you want to do the work to find and fix a bug. If that's not technical friction, I don't know what is.

• RT @briandfoy_perl Huh, the new look for (lists.perl.org) is really slick. Leo Lapworth really deserves his White Camel Award.
• RT @parrotvm We are converting our Trac wiki to @Github! (github.com) If you want to help, let us know!
• Perl, C and Dr. Dennis Ritchie (blogs.perl.org)
• Trial releases with Dist::Zilla (blogs.perl.org)
• Handy guide to core module activity (blogs.perl.org)
• Photos from YAPC::Asia in Tokyo (blogs.perl.org)
• Announcing namespace::sweep (blogs.perl.org)
• Why do you want new major Perl features? (blogs.perl.org)

The specific problem is that each milestone represents a date by which you absolutely must finish your work lest you have to wait for the next milestone. In one sense, that's a feature of regular release cycles: if a feature isn't ready for the current release, it can wait until the next release. In practice, uncautious project management can turn this virtue into a vice in two ways:

• If you get rewarded or punished based on meeting or missing calendar dates—not the appropriate level of release quality—you will prioritize calendar time over quality.
• If the periodicity between releases is too great, you'll prioritize getting a project in the current release over waiting until the next release.

Choosing the right periodicity is more art than science. (Project management is art, not science.) Yet you can know the period length is inappropriate when the end of a period approaches and you suffer the great stampede of features suddenly becoming good enough to land the week or weekend or day before your cutoff date for features landing for the next Great Big Release.

You can see the evolution of this understanding in Parrot over the past couple of years, and it's likely to culminate in a rethinking of Parrot's deprecation and support policy. Whereas Parrot promised to adhere to a six month cycle of publishing a deprecation or incompatible change notice before making the change or removal, that promise was simply untenable as waiting six months to make necessary changes left no one happy, least of all the end user implementors for whom the policy should have been most useful. That big stampede happened in Parrot as every six month cycle closed.

When we switched to a three month period, the big stampede still happened.

When I'm fortunate enough to work within a dedicated team of full-time developers using an agile approach, an iteration period of a week or two often seems to work the best. Anything longer than that still has stampedes, but delaying a branch merge by a week or so to get another day or two of polish isn't a big deal. Delaying a branch merge by a month or so—especially for branches in Parrot which should be trending smaller, simpler, and more encapsulated—pushes the edge of what's acceptable.

Granted, releasing stable and useful software still requires a lot of discipline from project management and developers. Slashdot comment screeching to the contrary, shorter release cycles don't have to mean that Unity-style big bang UI changes will land every month. (I suspect that Unity could have used another few months of development and testing and integration, if not another six month cycle). It's also often a simple matter of programming to release a feature present but disabled by default so as to get more testing on deployment and integration without forcing users to siwtch yet.

Yet the most important matter of discipline is unambiguous and ubiquitous project status information such as the reports from automated test suites, memory leak checkers, branch status reports, bug reports, and anything else that can measure the quality of the iteration's eventual release.

Certainly this requires a substantial investment on the part of projects to build and maintain this infrastructure, but it's the only reliable and repeatable way I know to produce great software. (If you've noticed that the much-vaunted Perl 5 monthly bleadperl releases are rather boring, that's the point! Most of the excitement takes place at project boundaries such as the interactions with CPAN modules.)

... but listen to a tale of two intertwined projects with a seemingly-sane and published deprecation policy that sounds great but doesn't actually work.

Parrot and Rakudo used to be much closer together. The Parrot repository had a subdirectory languages/ which held several language implementations running on Parrot. This was well and good (How did a lion get rich? It was the olden days!)—whenever Parrot underwent an API change, a simple ack of the source tree could find most places that needed to change, and running the test suites of the projects under languages/ could find the rest.

(Getting people to run the full test suite is a different story, but in that case at least guilt and a good bisection tool work post hoc magic.)

Then one day, Parrot left the Perl-centric world of perl.org and kicked languages/ out of the nest. languages/perl6 went to Rakudo.org and the troubles began.

You see, back in the olden days when a lion could get rich and the Perl 6 VM was like the Perl 5 VM but a little bit better, the way to extend Parrot was, obviously, write a whole wad of C code. Want to add a new data type? Write a wad of C code. Want to change how a core wad of C code works? Write a whole wad of C code. If hacking on Perl 5 had taught anyone anything (and I can't believe I can write this with a straight face), it's that finding hackers ready, willing, and able to write big wads of C code to work around or customize or extend other big wads of C code so that someone else doesn't have to write C code is a really easy prospect.

Anyhow, the Parrot execution and extension and embedding model has always assumed, as does Perl 5, that the VM should expose a big wad of C functions that extenders and embedders should use to manipulate big wads of C structs and other data types defined in C.

On one hand, customization and flexibility is good. On the other hand, people like me who know both C and Perl write in Perl when we can and C only when we have to for several very, very good reasons.

Imagine a ventriloquist with a Parrot puppet, except he doesn't stick his hand in the puppet to pull levers and flip switches and wiggle his fingers. He has a robot hand, operated by a joystick, and that robot hand is half robot and half organic and it tickles actual organs inside the puppet and the bird goes off and does its thing. Oh, and he's operating that joystick with his toes. Plus he's underwater, wearing a blindfold.

In other words, Rakudo has an intimate knowledge of the guts of Parrot and does some strange and bizarre things. (If you look through the revision history of those parts of Rakudo, you'll see I've fixed some of those strange and bizarre things and I've perpetuated others.)

When Rakudo and other languages left the nest, a deprecation policy came about. Imagine this vaudeville act:

HLL Developer: "Hey, you changed this API!"

Parrot Developer: "You weren't supposed to use it."

HLL Developer: "No one told me that, but regardless, you broke my project."

Parrot Developer: "Oh, sorry. Well here's how to fix it."

HLL Developer: "Wish you'd told me that months ago."

Parrot Developer: "Yeah, we'll do better next time. By the way, what in the world were you doing with that?"

HLL Developer: "No one told me to do it differently, so I just threw together something that works."

You can see where this is going.

Thus began the great Parrot version numbering wars. The end result was that Parrot would have two supported releases every year, one coming every six months. The initial deprecation policy promised no backwards incompatible changes without notification in at least one supported release.

In other words, if a project which was previously in languages/ had moved somewhere else where a well-meaning Parrot developer didn't notice that a simple change broke backwards compatibility, that project had the ability to request a reversion of the change, a documentation of the change as a compatibility change, and a delay of up to six months before making the change...

... depending on when the project noticed.

I've mentioned the idea of technical friction before, but get out your calendar.

This policy didn't only protect languages from unintended changes, it protected languages from desired changes. For example, if Rakudo wanted a new feature in Parrot, and if the best way to add that feature in Parrot meant breaking backwards compatibility (changing the order of search paths, for example), both Rakudo and Parrot developers had to mark their calendars.

The deprecation policy period is now three months, for hopefully obvious reasons.

Even so, now there are at least two projects on separate time tables with separate goals for features and improvements and deprecations. When Rakudo wants a feature from Parrot that Parrot doesn't yet support (or Rakudo developers believe it doesn't support fully or at all or in the precise way Rakudo wants), Rakudo tends to toe the joystick. When Parrot makes a change that exposes the fragility of tickling bird gullets with robot fingers, Rakudo gets to complain. The deprecation policy followed to the letter prevents Parrot from adding the features Rakudo demands at the same time that it prevents Parrot from improving the features that Rakudo uses.

You might read this and think that the problem is the deprecation policy. It's not—at least, that's not the root cause.

I hate the polite fiction calorie-free slogan that "Perl 6 is a research project", because it's a polite way to suggest that Perl 6 is an embarrassing pipe dream no true Perl hacker takes seriously, but implementing Perl 6 has required a lot of research and fits and false starts. (Before you complain next that it's taking a while, you write a performant grammar engine which allows lexical overriding of any grammatical rule or category.) When Parrot began, no one knew exactly how a Perl 6 VM machine should implement several important features. That was also true in 2005. That was also true of Parrot's 1.0 release in March 2009.

Making an effective Perl 6 means continually thinking and rethinking and simplifying and inventing new things.

Now tie an anchor around the feet of a live bird and pretend you're a ventriloquist.

Granted, Parrot's also long had the goal of running other dynamic languages efficiently and effectively. Its endgame is more than merely Perl 6 (and Perl 5). Yet a Parrot that won't run Perl 6 well has failed at its primary goal, and subsequent goals are less interesting.

The right solution is to invent a time machine and not kick Rakudo out of the nest. (The rightest solution is to invent a time machine and start implementing Perl 6 on Parrot from the first day as a first-class citizen and not a rat's nest of Perl 5 parsing madness which had, to my count, at least two replacements before Rakudo.)

The current Rakudo solution is to add another layer, another project external to both Parrot and Rakudo, which will sit in between the two and abstract away the Parrot specific parts in the hope that someday someone will add another VM backend so that Rakudo doesn't have to be tied to Parrot forever.

If the notion of working around a broken deprecation policy—broken because of too little coupling between two projects so intimately intertwined—is to add another layer of coupling to separate those two projects seems a little strange to you, you're not alone. By my count, that's three projects in a driver's seat built for one. If you can figure out which project is upstream and which project is downstream and how to debug with reasonable each which change where broke which test and who gets to fix it, you're a far, far better developer than I am.

M0 is one of those goals. This small set of opcodes represents the minimum necessary platform-specific code required to implement the rest of Parrot. M0 is a specification. Like Perl 6 is a specification, there can be multiple implementations. The demo implementation is a Perl 5 program, and my C implementation is the second implementation (and hopefully a likely candidate to become part of Parrot itself).

Like Perl 6, M0 is a specification and a test suite.

Unlike Perl 6, the M0 test suite is tied to the original implementation. That is to say, when you run the M0 tests, you exercise the Perl 5 demo M0 implementation.

A small task for the interested is to decouple the M0 test suite from the Perl 5 M0 implementation and make those tests available for multiple implementations. This probably means representing those tests in a form with expected inputs and outputs, perhaps running M0 source code through an assembler to create M0 bytecode, and running the tests against an arbitrary implementation.

Perl 6's test suite has an interesting capability whereby the list of tests contains metadata on which known implementation should skip or try to run but not expect to pass arbitrary test files. The #perl6 channel on irc.freenode.net can give you more details about this fudging process.

This project does easily parallelize across multiple interested people. Coordinating in #parrot on irc.perl.org will help.

As a bonus test, the C implementation should allow optional testing with Valgrind, if installed. All tests should run without memory leaks or invalid memory accesses. (I tested the C implementation by hand, but automating this is essential for further development.) This task is slightly more complex, but should be within reach for any Perl 5 programmer of modest experience.

M0 is one of those goals. This small set of opcodes represents the minimum necessary platform-specific code required to implement the rest of Parrot. M0 is a specification. Like Perl 6 is a specification, there can be multiple implementations. The demo implementation is a Perl 5 program, and my C implementation is the second implementation (and hopefully a likely candidate to become part of Parrot itself).

Like Perl 6, M0 is a specification and a test suite.

Unlike Perl 6, the M0 test suite is tied to the original implementation. That is to say, when you run the M0 tests, you exercise the Perl 5 demo M0 implementation.

A small task for the interested is to decouple the M0 test suite from the Perl 5 M0 implementation and make those tests available for multiple implementations. This probably means representing those tests in a form with expected inputs and outputs, perhaps running M0 source code through an assembler to create M0 bytecode, and running the tests against an arbitrary implementation.

Perl 6's test suite has an interesting capability whereby the list of tests contains metadata on which known implementation should skip or try to run but not expect to pass arbitrary test files. The #perl6 channel on irc.freenode.net can give you more details about this fudging process.

This project does easily parallelize across multiple interested people. Coordinating in #parrot on irc.perl.org will help.

As a bonus test, the C implementation should allow optional testing with Valgrind, if installed. All tests should run without memory leaks or invalid memory accesses. (I tested the C implementation by hand, but automating this is essential for further development.) This task is slightly more complex, but should be within reach for any Perl 5 programmer of modest experience.

M0 is one of those goals. This small set of opcodes represents the minimum necessary platform-specific code required to implement the rest of Parrot. M0 is a specification. Like Perl 6 is a specification, there can be multiple implementations. The demo implementation is a Perl 5 program, and my C implementation is the second implementation (and hopefully a likely candidate to become part of Parrot itself).

Like Perl 6, M0 is a specification and a test suite.

Unlike Perl 6, the M0 test suite is tied to the original implementation. That is to say, when you run the M0 tests, you exercise the Perl 5 demo M0 implementation.

A small task for the interested is to decouple the M0 test suite from the Perl 5 M0 implementation and make those tests available for multiple implementations. This probably means representing those tests in a form with expected inputs and outputs, perhaps running M0 source code through an assembler to create M0 bytecode, and running the tests against an arbitrary implementation.

Perl 6's test suite has an interesting capability whereby the list of tests contains metadata on which known implementation should skip or try to run but not expect to pass arbitrary test files. The #perl6 channel on irc.freenode.net can give you more details about this fudging process.

This project does easily parallelize across multiple interested people. Coordinating in #parrot on irc.perl.org will help.

As a bonus test, the C implementation should allow optional testing with Valgrind, if installed. All tests should run without memory leaks or invalid memory accesses. (I tested the C implementation by hand, but automating this is essential for further development.) This task is slightly more complex, but should be within reach for any Perl 5 programmer of modest experience.