tl;dr: Stuff I worked on:

• A new way of thinking about CPAN indexes
• Making CPAN.pm use a lot less memory
• Making CPAN.pm respect optional 'recommends' and 'suggests' prereqs
• Adding features and fixing bugs in toolchain modules
• Useful modules released: CPAN::Meta, CPAN::Meta::Requirements, Parse::CPAN::Meta, Search::Dict, Tie::Handle::Offset
• Other yak-shaving modules released: Pod::Wordlist::hanekomu, Task::BeLike::DAGOLDEN, CPAN::Test::Dummy::Perl5::RecSug

Why I love the QA hackathon

If you've been under a rock and still don't know what the QA Hackathon is: it's a sponsored conference in which a small band of dedicted Perl hackers spend three days madly coding to improve the quality of the Perl experience for everyone.

I really enjoyed the chance to meet people in person that I only know from on-line venues or rarely get to see face-to-face. Having so many people working on so many projects in one space made it really easy to benefit serendipitously from the work of others, or to have a chance conversation spark a new way to get things done.

Another thing that makes the hackathon awesome is how quickly blocking issues get fixed. Several times, someone would hit a bug in some related library, walk across the room to the person who could fix it, and it would get fixed and shipped before one could get a coffee.

This year, my work focused mostly on the evolution of CPAN.pm and the CPAN toolchain.

A new way of thinking about CPAN indexes

On the morning of the first day, I convened an informal group of half-a-dozen CPAN client maintainers, installer maintainers and other interested parties [1] to talk about how to re-think CPAN indexing. In particular, I wanted to separate the notion of the "index" from the "repository". The canonical CPAN index is a file on CPAN that maps Perl package names ("Foo::Bar") to a path to a distribution archive file on CPAN ("DAGOLDEN/Foo-Bar-1.23.tar.gz").

Historically, your CPAN client mirrored the index from the same CPAN mirror used to download tarballs. I think that's limiting in a few ways. First, that file keeps growing as CPAN grows and it takes a while to mirror the whole file when you only need the mapping for a few modules.

Some CPAN clients, like cpanminus, don't even use the package index directly, but query a web API that serves up answers from it, which is one way to separate the index from the repository. That would be a nice feature to have in all CPAN clients.

That still doesn't change the model of having only one official index that your client knows about. If you or your company want to manage the mapping, you've got to use various tools to modify the official index in some sort of minicpan or private CPAN repository (aka "DarkPAN"). It's possible, but not user-friendly.

After some debate, the group agreed on a new model. A CPAN client should support an ordered list of index resolvers and should query them in turn. This means you could specify that you want an online web resolver tried first, and only then the traditional index.

More powerfully, you could list a local overlay index as the first resolver. That would let you freeze the mapping to a particular version, or to swap in a development release that fixes a critical bug. The overlay index would only need to list the modules you want to change, because your CPAN client will fall back to the canonical index. You could even have an overlay index per-application or per-application-version for total control.

We also agreed that mapping shouldn't just be a distribution path on a CPAN mirror, but should evolve into a URL. This would allow overlay indexes pointing to locally patched distributions, or to the BackPAN, or potentially even to source repositories (if appropriate scheme handlers were written to check out the necessary files).

In summary -- CPAN indexes should become an open, flexible mechanism to give users more control over how module names are mapped to the files that can provide them.

After reaching that agreement, Nick Perez (nperez) volunteered to start working on a common library for index resolvers (to be called CPAN::Common::Index) and to build a resolver for it that uses MetaCPAN to provide the mapping data.

Meanwhile, I started work on a proof of concept for how CPAN.pm could be modified to use the new, common library instead of its traditional index lookup routines.

Evolution of CPAN.pm

It had been a while since I was deep in the guts of CPAN.pm, but after coming back up to speed, I tackled two big projects and found one crazy bug along the way.

New CPAN indexing and reduced memory footprint

A stock CPAN.pm client uses a ton of memory when the indexes are loaded — about 300 MB last time I looked. It's bloated because it keeps a read-only copy of the indexes loaded in data structures in memory and also keeps a mutable object for every index entry as well.

Some time ago, CPAN::SQLite was released to help solve that problem. It kept the indexes in a SQLite database and loaded data into memory on demand. I decided to use that same approach for the interface to the forthcoming CPAN::Common::Index library, with the goal of being able to load all data on demand, even directly from the package index file, using only core Perl modules.

Here's the trick: the package index file is line-oriented and is sorted by package name. Using the Search::Dict core module, I was able to do a binary search as a super-fast way to look up data for a package name.

The wrinkle in that plan is that Search::Dict wants a filehandle and uses it to seek around in the file, but the package index has an email-style header that confuses it. I could have copied it without the header, but that takes time and memory, too. PAUSE could publish an identical copy without the header, but that's extra work for PAUSE and potentially confusing if they ever get out of sync.

Instead, I wrote Tie::Handle::Offset and Tie::Handle::SkipHeader to hide the email header on a handle, so I could give that directly to Search::Dict. Unfortunately, Search::Dict died unless stat() on the handle gave a valid response, so I patched it to fall back to an alternate method if stat() failed. That revealed a bug in Perl, in which stat() warns when called on tied handles, even if there is a valid filehandle to check (filed as rt#112164).

Since that bug can't get fixed until Perl 5.17 and since we need a working Search::Dict for older Perls anyway, I patched Search::Dict to avoid using stat() on handles, and asked Ricardo Signes (rjbs) to give me a green light make a dual-life release to CPAN.

After chasing my tail on that for a while, I finally was able to get a proof of concept of on-demand index lookup on the package index file working, saving hundreds of megabytes of memory. It didn't use the CPAN::Common::Index library, since Nick was still writing it, but it expects the same API, so it will be easy to adapt once CPAN::Common::Index is ready (meaning that fast MetaCPAN lookups for CPAN.pm should be easy too).

My POC only covered the package index, but Andreas Koenig (klapperl) and Ricardo created a similarly sorted index of author data and we agreed to consider a similar approach for modlist data once we see how the package indexing works in practice.

I would have been happy if that was all I achieved at the hackathon but I still had some time left to get more done.

CPAN.pm support for 'recommends' and 'suggests' prereqs

The v2 CPAN::Meta::Spec formalized dependency specifications for different phases (configure/build/test/runtime) and for different levels of dependency (requires/recommends/suggests/conflicts). The 'recommends' level is for things that should be usually installed to make a module better except in really resource-constrained environments. The 'suggests' level is for really optional modules that might make a module better but really aren't necessary for regular use.

Even though those have been specified for a while, none of the CPAN clients supported them -- meaning it was a manual job to look at the META file, see the recommends/suggest and install them yourself. Ssually, no one bothers.

Since I was on a roll from the indexing work, I set up another CPAN.pm feature branch and implemented support for a 'recommends_policy' and a 'suggests_policy' to control whether those prereqs should be queued up along with the required ones. Even better, if those optional dependencies fail for any reason, CPAN.pm won't warn about missing dependencies and simply notes them as being optional when it reports the failures after processing a command.

Along the way, I found and fixed a CPAN.pm edge-case bug where a module listed in both "build requires" and "runtime requires" and that has a lower prereq in "build requires" would overwrite the higher requirement in "runtime requires". (yikes!) That might explain some bizarre CPAN.pm bug reports I've seen that we could never track down, so it was an extra win.

Unfortunately, ExtUtils::MakeMaker and Module::Build don't yet preserve 'suggests' dependencies during configuration, so this will only help with 'recommends', but fixes to the installers are in the works (Ricardo was working on EU::MM at the hackathon) and CPAN.pm will be ready whenever they are.

Adding features and fixing bugs

CPAN::Meta got a tiny bit of love. I released a version of Parse::CPAN::Meta with dependencies on the latest (less-buggy) versions of CPAN::Meta::YAML and JSON::PP. (I've already got CPAN Testers fail reports, so the tests apparently need some more work.).

Leon Timmermans (leont) added a new method to CPAN::Meta::Requirements for something he was working on, which was awesome because I wound up needing it for the CPAN.pm work only a couple hours after he sent me the pull request. Then I split out CPAN::Meta::Requirements from CPAN::Meta and released it, so CPAN.pm could depend on it without needing all of CPAN::Meta. CPAN::Meta also got some releases for these various changes.

In a startling display of synchronicity, both Curtis Poe (ovid) and Lars Dɪᴇᴄᴋᴏᴡ (daxim) reported a weird Module::Build bug within about an hour of each other. Apparently, errors in META file creation can result in existing META files being deleted, no new files being created and no error message shown about what happened. Leon and I figured out the problem and offered some workarounds — though we ran out of time at the hackathon to fix it in Module::Build itself.

Various other things I did

Several people — Leon, Michael Schwern, Olivier Mengué (dolmen), Lars, me, and a few others I now forget (sorry) — got together to discuss a draft of a "Build.PL API" draft. It defines what CPAN clients should expect interacting with a Build.PL/Build-based installer, which opens the door to future replacements for Module::Build, like Module::Build::Tiny.

Breno de Oliveira (garu) wanted to add CPAN Testers reporting to cpanminus, and along the way volunteered to write a unified, second-generation CPAN Testers client to replace the disparate behaviors of CPAN::Reporter and the reporting modules of CPANPLUS. I gave a small tutorial on CPAN Testers and the Metabase backing it to Breno and others interested in the topic.

As a minor note, I got annoyed at some Test::Spelling carping during all the releases I was doing, so I released a new Pod::Wordlist::hanekomu. If you use Dist::Zilla and the Test::PodSpelling plugin, check it out!

Cool things other people did

Some things I didn't work on that I thought were notable:

• To support CPAN::Common::Index, Nick wrote MetaCPAN::API::Tiny — a client for querying MetaCPAN that relies only on core Perl modules, which is exactly what we need for a new CPAN.pm index resolver
• The CPAN "package index" now updates every five minutes instead of every hour... which means other projects that rely on it, like MetaCPAN, are even closer to real time.
• I asked around if there was a command-line client for MetaCPAN and there wasn't. Then Chris Nehren (apeiron) asked me what I had in mind, whipped one up, and submitted it as an addition to the MetaCPAN::API distribution
• Ricardo worked on getting full support for CPAN::Meta::Spec v2 into ExtUtils::MakeMaker, including TEST_REQUIRES and ensuring all prerequisites types are preserved in MYMETA.json files
• Ricardo also got PAUSE to save package index files into git after each update, so we no longer lose historical information
• Peter Rabbitson (ribasushi) demonstrated a way to use git to store CPAN Testers reports to achieve massive delta compression (and make it easy for people to get copies of the raw data quickly and cheaply). I didn't have time at the hackathon to do much with it but hope to look into it more soon.
• Late in the afternoon on Sunday, Nick used his MetaCPAN::API::Tiny client for what was dubbed "CloudPAN", a crazy April-Fools proof-of-concept to hook module loading to load missing modules directly from source on metacpan. You'll never need to install pure-Perl modules again. ;-)

There was a lot more going on and a lot I missed, so if I omitted anyone's project, I mean no offense. (I'll read all the hackathon blogs to catch up.)

Conclusion and Acknowledgments

This was my third hackathon and was just as inspiring (and productive) as the last two. I'm excited about the evolution of CPAN.pm and hope to get my work tested further and then merged into the CPAN.pm master branch before long.

I have nothing but wonderful things to say about Laurent Boivin (elbeho), Philippe Bruhat (BooK) and the French Perl Mongers who organized a great event and provided wonderful hospitality, including an endless supply of food, drink and coffee machines to fuel our hacking.

I would also like thank the hackathon sponsors whose generosity made the hackathon possible and enabled me to attend. (If you'd like to donate, it's not too late and will help support next year's QA hackathon.)

These companies and organizations support Perl. Please support them: The City of Science and Industry, Diabolo.com, Dijkmat, DuckDuckGo, Dyn, Freeside Internet Services, Hedera Technology, Jaguar Network, Mongueurs de Perl, Shadowcat Systems Limited, SPLIO, TECLIB’, Weborama, and $foo Magazine

These people made individual donations (you rock!): Martin Evans, Mark Keating, Prakash Kailasa, Neil Bowers, 加藤 敦 (Ktat), Karen Pauley, Chad Davis, Franck Cuny, 近藤嘉雪, Tomohiro Hosaka, Syohei Yoshida, 牧 大輔 (lestrrat), and Laurent Boivin

Special thanks also to Torsten Raudssus (getty) and Duck Duck Go for the tee-shirt and Booking for the silly putty. :-)

Finally, thank you to all my fellow hackers! I had a great time and I hope to see you all again next year!

[1] CPAN index discussion group (with some people coming and going): me, Andreas Koenig, Florian Ragwitz, Michael Peters, Michael Schwern, Nick Perez, Olaf Alders, Olivier Mengué, Ricardo Signes, Tatsuhiko Miyagawa and probably even more I don't remember. (Please remind me if you were there and want to share the credit/blame.)

In the April CPAN Testers Summary, Barbie noted that CPAN Testers has nearly reached 12 million reports submitted.

I was recently looking at some Metabase statistics and noticed that it now contains over 6.3 million reports since going live. That means that over half of all reports were submitted by web instead of email. It also means that CPAN Testers has nearly doubled in a year.

The year has not been without hiccups, as the recent Amazon-related outage showed. I'll be spending some time this summer addressing some deficiencies and will be encouraging people to get involved in improving the CPAN Testers clients to make it easier for more people to participate.

Next year, will we see 24 million reports? Let's make it happen!

If you've been building up a backlog, please throttle your uploads.

I apologize for the interruption. Over the next month or two, I'll be making some changes to make sure that CPAN Testers is ready for a repeat of this particular failure mode (as well as other similar failure modes).

My first attempts at replication failed, so after a few emails back and forth, Chad sent me detailed instructions for replication. Here is an excerpt from his email (used with his permission):

I setup a new user, and deleted his .bashrc, leaving only the stock
Ubuntu 10.10 /etc/bash.bashrc in the environment. Then I created this
three-line ~/.bashrc :

perl5=/tmp/tmplib
lib="$perl5/lib/perl5"
eval $(perl -I"$lib" -Mlocal::lib="$perl5")

So, the environment now resolves to:

PERL5LIB=/tmp/tmplib/lib/perl5/x86_64-linux-gnu-thread-multi:/tmp/tmplib/lib/perl5
PERL_LOCAL_LIB_ROOT=/tmp/tmplib
PERL_MB_OPT='--install_base /tmp/tmplib'
PERL_MM_OPT=INSTALL_BASE=/tmp/tmplib

Notice that Chad describes the setup in detail, and even went to the trouble of replicating it with a "clean" user. This made it very easy to set up exactly the same situation on my development machine.

Next, Chad walked me through how to replicate the issue and even confirmed it on a fresh virtual machine!

Then I installed Parse::CPAN::Meta 1.42 (with CPAN 1.9402 and
local::lib 1.008), then I tested MB 0.37_04 which gave the same
errors. Then I quit the cpan shell, restart it, first do an explit
test of PCM 1.4401 before running a test of MB and all tests pass, as
before.

I also verified the same behavior on a virtual machine with a
fresh ubuntu 10.10 (with updates) and the same three-line .bashrc and
got the same behavior. I'm surprised that you cannot reproduce this.
At this point I don't believe there is anything left that is specific
to my environment.

With those clear instructions, I was able to replicate the issue.

Chad then took me through variations and outcomes:

I then upgraded CPAN to 1.94_65 but have the same errors on MB 0.37_04
unless I explicitly test PCM 1.4401 first.

And I have the same problems with MB version 0.37_05 as well, which
also works after an explicit test of PCM 1.4401

I tried to start working backwards a bit, MB 0.3624 is fine,
presumably because it doesn't depend on PCM.
However, MB 0.3701 fails, despite the fact that it only depends on PCM
1.42, which is the one that's installed, according to pmvers.  It
looks to be the same issue in each case: the existing PCM is not always detected.

Now I have a fact base that I can test and confirm. Finally, Chad did some further digging into the problem:

Then I looked at t/mymeta.t and traced back to CPAN::Meta and looked
at it dependencies in Makefile.PL to find that PCM 1.44 is listed both
as prereq and as a build prereq, which, being a novice, looked a bit
funny to me. Taking out the build prereq and leaving the prereq then
allowed me to test MB without errors.

And with that, I've got a decent workaround solution, as well as a hint as to what's going wrong in CPAN.pm. I haven't fixed CPAN.pm yet, but with this start, it's going to be much, much easier.

I want to thank Chad for his responsiveness and the incredible detail of his report. I hope it can be a lesson to anyone reporting bugs or responding to questions about failure:

1. Describe your environment in detail. If you can, replicate the error with a "clean" user or even on a clean install of the OS.
2. Describe the exact steps you took to replicate the failure. Take notes as you do it, so you can be as specific as possible.
3. Describe any variations you tried that did work, or any workarounds you used to deal with the problem.

That seems like common sense, but few bug reports or failure investigations I get are as thorough and constructive as the example above. Thank you, Chad!

leont pointed out on IRC that the CPAN Author FAQ is very hard to find and doesn't currently show up in search results, so if you would like to help out, please blog or twitter about it or something and help point some links in the right direction. :-)

Thanks!

This is nearly one-eighth of the 8.3 million reports contributed since the start of CPAN Testers.

The CT 2.0 server has been averaging just over 14 reports submitted per minute this week. If that submission rate holds up, September will see over 600,000 reports submitted and the total number of reports submitted will nearly double in a year's time.

I would like to say a big thank you to all the CPAN Testers that have been patient through bumps and lumps of the launch of CT 2.0. It looks like we're off a great start!

I've just updated the "quick-start" documentation on the CPAN Testers wiki: Quick HowTo for CPAN Testers 2.0

It's not perfect, but it's a lot better than it used to be. If you are still submitting reports by email, please read and follow the instructions before the end of August. If you have trouble, please ask for help on the cpan-testers-discuss mailing list or on #cpantesters-discuss on irc.perl.org.

• Free QA! -- a non-technical talk about the history and social architecture choices of the CPAN Testers project
• Perl 5, Version 13 -- a lightning talk summarizing notable changes in the Perl 5.13 development series

If you are already a CPAN Tester, a "profile file" has been pre-generated for you and you should submit a request that it be emailed to you. This pre-generated profile will identify your new CT 2.0 reports in a way that will make it easier to link them to the CT 1.0 old reports as those are converted.

If you are new to CPAN Testing, you will need to generate a profile file. Install Metabase::Fact and then run the metabase-profile program from the command line. It will prompt you to provide some information and output a profile file. The first time you use this profile file to submit a report, your profile will be automatically registered on the CT 2.0 server.

While the CT 2.0 server has launched, the clients that submit reports are, for now, the same ones used for CT 1.0 -- either CPAN::Reporter or CPANPLUS (or one of the smoke frameworks written for them like minismokebox or CPAN::Reporter::Smoker). The transport framework originally intended for email gets hijacked to send reports via http to CT 2.0 instead.

To send reports to CT 2.0, you will need to install the Test::Reporter::Transport::Metabase module and then configure your client to use that transport instead of the default one. Instructions are available on the CPAN Testers Wiki. While the docs are a bit sparse as I write this, I expect they will continue to evolve and improve as more people contribute to them.

If you're really stuck, please check the FAQ or ask for help on the CPAN Testers Discuss mailing list. If you are a CPAN Tester, I highly recommend joining the list to keep up on the latest issues and solutions in the migration to CT 2.0. If you do ask for help, please be sure to include the version number of all the relevant testing modules you're using along with your perl -V output.

Another good place to go for quick questions is the #cpantesters-discuss channel on irc.perl.org.

In other news, I'll be speaking about CPAN Testers at OSCON in my Free QA! talk. If you or someone you know will be there, please check it out or recommend it.