In the April CPAN Testers Summary, Barbie noted that CPAN Testers has nearly reached 12 million reports submitted.

I was recently looking at some Metabase statistics and noticed that it now contains over 6.3 million reports since going live. That means that over half of all reports were submitted by web instead of email. It also means that CPAN Testers has nearly doubled in a year.

The year has not been without hiccups, as the recent Amazon-related outage showed. I'll be spending some time this summer addressing some deficiencies and will be encouraging people to get involved in improving the CPAN Testers clients to make it easier for more people to participate.

Next year, will we see 24 million reports? Let's make it happen!

If you've been building up a backlog, please throttle your uploads.

I apologize for the interruption. Over the next month or two, I'll be making some changes to make sure that CPAN Testers is ready for a repeat of this particular failure mode (as well as other similar failure modes).

My first attempts at replication failed, so after a few emails back and forth, Chad sent me detailed instructions for replication. Here is an excerpt from his email (used with his permission):

I setup a new user, and deleted his .bashrc, leaving only the stock
Ubuntu 10.10 /etc/bash.bashrc in the environment. Then I created this
three-line ~/.bashrc :

perl5=/tmp/tmplib
lib="$perl5/lib/perl5"
eval $(perl -I"$lib" -Mlocal::lib="$perl5")

So, the environment now resolves to:

PERL5LIB=/tmp/tmplib/lib/perl5/x86_64-linux-gnu-thread-multi:/tmp/tmplib/lib/perl5
PERL_LOCAL_LIB_ROOT=/tmp/tmplib
PERL_MB_OPT='--install_base /tmp/tmplib'
PERL_MM_OPT=INSTALL_BASE=/tmp/tmplib

Notice that Chad describes the setup in detail, and even went to the trouble of replicating it with a "clean" user. This made it very easy to set up exactly the same situation on my development machine.

Next, Chad walked me through how to replicate the issue and even confirmed it on a fresh virtual machine!

Then I installed Parse::CPAN::Meta 1.42 (with CPAN 1.9402 and
local::lib 1.008), then I tested MB 0.37_04 which gave the same
errors. Then I quit the cpan shell, restart it, first do an explit
test of PCM 1.4401 before running a test of MB and all tests pass, as
before.

I also verified the same behavior on a virtual machine with a
fresh ubuntu 10.10 (with updates) and the same three-line .bashrc and
got the same behavior. I'm surprised that you cannot reproduce this.
At this point I don't believe there is anything left that is specific
to my environment.

With those clear instructions, I was able to replicate the issue.

Chad then took me through variations and outcomes:

I then upgraded CPAN to 1.94_65 but have the same errors on MB 0.37_04
unless I explicitly test PCM 1.4401 first.

And I have the same problems with MB version 0.37_05 as well, which
also works after an explicit test of PCM 1.4401

I tried to start working backwards a bit, MB 0.3624 is fine,
presumably because it doesn't depend on PCM.
However, MB 0.3701 fails, despite the fact that it only depends on PCM
1.42, which is the one that's installed, according to pmvers.  It
looks to be the same issue in each case: the existing PCM is not always detected.

Now I have a fact base that I can test and confirm. Finally, Chad did some further digging into the problem:

Then I looked at t/mymeta.t and traced back to CPAN::Meta and looked
at it dependencies in Makefile.PL to find that PCM 1.44 is listed both
as prereq and as a build prereq, which, being a novice, looked a bit
funny to me. Taking out the build prereq and leaving the prereq then
allowed me to test MB without errors.

And with that, I've got a decent workaround solution, as well as a hint as to what's going wrong in CPAN.pm. I haven't fixed CPAN.pm yet, but with this start, it's going to be much, much easier.

I want to thank Chad for his responsiveness and the incredible detail of his report. I hope it can be a lesson to anyone reporting bugs or responding to questions about failure:

1. Describe your environment in detail. If you can, replicate the error with a "clean" user or even on a clean install of the OS.
2. Describe the exact steps you took to replicate the failure. Take notes as you do it, so you can be as specific as possible.
3. Describe any variations you tried that did work, or any workarounds you used to deal with the problem.

That seems like common sense, but few bug reports or failure investigations I get are as thorough and constructive as the example above. Thank you, Chad!

leont pointed out on IRC that the CPAN Author FAQ is very hard to find and doesn't currently show up in search results, so if you would like to help out, please blog or twitter about it or something and help point some links in the right direction. :-)

Thanks!

This is nearly one-eighth of the 8.3 million reports contributed since the start of CPAN Testers.

The CT 2.0 server has been averaging just over 14 reports submitted per minute this week. If that submission rate holds up, September will see over 600,000 reports submitted and the total number of reports submitted will nearly double in a year's time.

I would like to say a big thank you to all the CPAN Testers that have been patient through bumps and lumps of the launch of CT 2.0. It looks like we're off a great start!

I've just updated the "quick-start" documentation on the CPAN Testers wiki: Quick HowTo for CPAN Testers 2.0

It's not perfect, but it's a lot better than it used to be. If you are still submitting reports by email, please read and follow the instructions before the end of August. If you have trouble, please ask for help on the cpan-testers-discuss mailing list or on #cpantesters-discuss on irc.perl.org.

• Free QA! -- a non-technical talk about the history and social architecture choices of the CPAN Testers project
• Perl 5, Version 13 -- a lightning talk summarizing notable changes in the Perl 5.13 development series

If you are already a CPAN Tester, a "profile file" has been pre-generated for you and you should submit a request that it be emailed to you. This pre-generated profile will identify your new CT 2.0 reports in a way that will make it easier to link them to the CT 1.0 old reports as those are converted.

If you are new to CPAN Testing, you will need to generate a profile file. Install Metabase::Fact and then run the metabase-profile program from the command line. It will prompt you to provide some information and output a profile file. The first time you use this profile file to submit a report, your profile will be automatically registered on the CT 2.0 server.

While the CT 2.0 server has launched, the clients that submit reports are, for now, the same ones used for CT 1.0 -- either CPAN::Reporter or CPANPLUS (or one of the smoke frameworks written for them like minismokebox or CPAN::Reporter::Smoker). The transport framework originally intended for email gets hijacked to send reports via http to CT 2.0 instead.

To send reports to CT 2.0, you will need to install the Test::Reporter::Transport::Metabase module and then configure your client to use that transport instead of the default one. Instructions are available on the CPAN Testers Wiki. While the docs are a bit sparse as I write this, I expect they will continue to evolve and improve as more people contribute to them.

If you're really stuck, please check the FAQ or ask for help on the CPAN Testers Discuss mailing list. If you are a CPAN Tester, I highly recommend joining the list to keep up on the latest issues and solutions in the migration to CT 2.0. If you do ask for help, please be sure to include the version number of all the relevant testing modules you're using along with your perl -V output.

Another good place to go for quick questions is the #cpantesters-discuss channel on irc.perl.org.

In other news, I'll be speaking about CPAN Testers at OSCON in my Free QA! talk. If you or someone you know will be there, please check it out or recommend it.

Apocalypse then helpfully batched several thousand reports and tried to overwhelm the server with parallel submissions. At about 450 reports per minute we finally started seeing some submission errors and now have some work to do to use the Amazon SimpleDB more efficiently or to queue submission peaks and process them offline.

In other interesting news, the CT2.0 server now supports Test::Smoke::Metabase reports, which Tux implemented during the Vienna QA hackathon and refined in the last couple weeks. He uploaded 150,000 or so historical Perl smoke reports as a test and is exploring using the CT2.0 server instead of the existing email system that supports Perl smoke testing. It’s the exact same problem (at somewhat lower volumes) that CT2.0 was designed to address and it’s nice to see some synergy in the work to date.