As a Senior Developer, you will be playing a central role in the design, development, and delivery of cutting-edge web applications for one of the most heavily-trafficked network of real estate sites on the web. You will work in a small, collaborative environment with other seasoned pros and with the direct support of the company’s owners and senior management. Your canvas and raw materials include rich data sets totaling several million property listings replenished daily by hundreds of external data feeds. This valuable data and our powerful end-user tools to access it are deployed across several thousand real estate search sites used by more than a million home-buyer leads and growing by 50K+ users each month. The 1M+ leads using our search tools are in turn tracked and cultivated by the several thousand real estate professionals using our management software. This is an outstanding opportunity to see your creations immediately embraced by a large community of users as you work within a creative and supportive environment that is both professional and non-bureaucratic at the same time, offering the positives of a start-up culture without the drama and instability.

If that sounds like interesting work to you then take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

Firstly I’d like to point out that, funnily enough, I’m not very familiar with Java or Perl6. It’s entirely possible that I’ll make all sorts of errors in the following details. If you spot any do please let me know.

Background

The Java language ecosystem is big and mature after years of heavy investment of time and money.

It doesn’t have a central repository of Open Source modules like CPAN (though Maven repositories like these are similar I guess). It does, however, have a number of mature high quality class libraries, and a very large number of developers familiar with those libraries (more on that below).

Goals

The primary goal of the java2perl6api project is to make it easy to create Perl 6 class libraries that mirror Java equivalents. By mirror I mean share the same method names and semantics at a high level (though not at a low-level, more on that below).

Secondary goals are to do that well enough that:

• the documentation for Java classes can serve as primary the documentation for the corresponding Perl 6 classes. The Perl 6 classes need only document the differences in behavior, which these should be minimal and ‘natural’. The same applies to books describing the Java classes.
• Java developers familiar with the Java classes should feel comfortable working with the corresponding Perl 6 classes.
• and, hopefully, some way can be found to convert test suites for the Java classes into Perl 6 code that’ll test the corresponding Perl 6 classes. (I appreciate that this is a non-trivial proposition, but there are viable approaches available, like xmlvm.) Even if that can’t be done, extracting and translating tests manually is less work, and more effective, than creating them from scratch for a new API.

Why?

Firstly, creating good APIs is hard. Java APIs like JDBC 3.0 and NIO.2 are the result of years of professional effort and demanding commercial experience. Why not build on that experience?

I appreciate that Java APIs are often limited by the constraints of the language, such as the lack of closures, and that Perl 6 can probably express any given set of semantics more effectively than Java. My point here is that some Java APIs embody, however inelegantly, years of hard won experience that we can benefit from. I’d rather make new mistakes than repeat old ones.

Secondly, there are many more Java developers than Perl developers. Many many more if job vacancies are any indication:

I think we’d be foolish not to try to smooth the path for any Java developers who might be interested in Perl 6. The java2perl6api project is just one small aspect of that.

I really hope someone starts writing a “Perl 6 for Java Developers” tutorial. Perl 6 has the potential to become a very popular language¹. Getting just a tiny percentage of Java developers (and Computer Science majors and their teachers) interested in it could be a big help.

Thirdly, any future DBI for Perl 6 and Parrot needs a much better foundation than the very limited and poorly defined one that underlies the Perl 5 DBI. I plan to adopt the JDBC 3.0 API and test suite for that internal role. (You could call this a “Test Suite Driven Strategy”.) I’ll talk more about that in a future blog post.

The History java2perl6api

I’ve been kicking around various ideas for integrating Java and Perl6/Parrot for years. I think I first decided to use JDBC as the inspiration for the DBI-to-driver API in 2006.

You may remember back in 2004, around the 10th anniversary of the DBI, the Perl Foundation setup a “DBI Development Fund” that people could donate to. I’ve never drawn any money from that fund. I want to use it to oil other peoples wheels.

In 2007 Best Practical sponsored Perl 6 Microgrants through the Perl Foundation. I asked if I could piggyback my idea for a Java to Perl 6 API translator onto their microgrant management process but using money from the DBI Development Fund. TPF and Best Practical kindly agreed. I posted a description of the task and Phil Crow volunteered and was awarded the microgrant in April 2007.

At OSCON in July 2007 I gave lightning talk called “Database interfaces for open source languages suck” which explained the rationale for using JDBC as a foundation for the DBI-to-driver API and mentioned Phil’s java2perl6 project.

Development ground to a halt around the end of 2007 for various reasons. It picked up again for a few months after OSCON 2009 (where I gave a short lightning talk asking for help) then stalled again in October. Partly because we seemed to have hit a limitation with Rakudo and partly because I was focussed on Devel::NYTProf version 3 and then version 4, which took way more time than I expected.

There’s life in the project again now. We’ve dodged the earlier problem, put the code on github, brought it into sync with current Rakudo Perl 6 syntax, and generally instilled some momentum.

The Current java2perl6api

Let’s take a look at a simple example.

To generate a perl6 file that mirrors the API of the java.sql.Savepoint class you’d just execute java2perl6api like this:

$ java2perl6api java.sql.Savepoint
loading java.sql.Savepoint
wrote java/sql/Savepoint.pm6 - interface java.sql.Savepoint
checking java/sql/Savepoint.pm6 - interface java.sql.Savepoint

That’s loaded and parsed the description of the java.sql.Savepoint class (from the javap command), generated a corresponding perl6 module, and run perl6 to validate it.

The generated module (with some whitespace and cruft removed) looks like this:

use v6;
role java::sql::Savepoint {
    method getSavepointId (
    --> Int   #  int
    ) { ... }
    method getSavepointName (
    --> Str   #  java.lang.String
    ) { ... }
};
=begin pod
=head1 Java
  Compiled from "Savepoint.java"
  public interface java.sql.Savepoint{
      public abstract int getSavepointId() throws java.sql.SQLException;
      public abstract java.lang.String getSavepointName() throws java.sql.SQLException;
  }
=end pod

The pod section shows the description of the class that javap returned. The java2perl6api utility parsed that Java interface and generated the corresponding Perl6 role. The ‘java.sql.Savepoint’ has been mapped to ‘java::sql::Savepoint’. The generated methods are stubs using ... (the “yada, yada, yada” operator). The types int and java.lang.String have been mapped to Int and Str. Because the only types used were built-ins, no type declarations were added.

Currently java2perl6api handles the above plus overloaded methods (which generate multi methods), multiple implements clauses (which generate multiple does clauses). There’s also partial support for class/interface constants (which currently generate exported methods).

The default behavior is to recursively process any Java types referenced by the class which aren’t mapped to Perl 6 types. So executing java2perl6api java.sql.Connection, for example, will generate 48 Perl 6 modules! (Because java.sql.Connection refers to many types, including java.sql.Array which refers to many types including java.sql.ResultSet which refers to java.net.URL which refers to java.net.Proxy etc. etc.) The --norecurse options disables this behavior.

Normally you’ll want to use the recursion but instead of letting it drill all the way into the Java types, you would supply your own ‘typemap’ specification via an option. That tells java2perl6api which Java types you want to map to which Perl 6 types. So instead of recursing into the java.net.URL type to generate a java/net/URL.pm6 file, for example, you can tell java2perl6api to use a specific Perl 6 type. Perhaps just Str for now.

How this relates to JDBC / DBDI / DBI v2

I want to start applying java2perl6api to the JDBC classes now to create a “Database Driver Interface” or “DBDI” for Perl 6.

Starting with the DriverManager class and the Connection interface I’ll use java2perl6api to generate corresponding Perl 6 roles with heavy stubbing out of types. Basically anything I don’t need to think about right now will be mapped to the Any type.

I’ll start fleshing out some basic implementation logic for each in a Perl 6 class that does the corresponding role. I’ll probably use PostgreSQL as the first driver and the guts of MiniDBD::Pg as inspiration.

The first minor milestones will be creating connections, then execute non-selects, then selects then prepared statements. Somewhere along the way I expect they’ll be a Perl 6 DBDI driver implemented for the Perl 6 MiniDBI project. The next key step would be to start refactoring the code heavily so anyone wanting to implement a new driver should only have to implement the driver specific parts. (There are some JDBC driver toolkits that can provide useful ideas for that.)

What needs doing

There’s a TODO file in the repository that lists the current items that need working on.

One fairly simple item is to add a --prefix option to specify an extra leading name for the generated role. So java.sql.Savepoint with a prefix of DBDI would generate a DBDI::java::sql::Savepoint role.

Another item, less simple but more important, is to automatically discover the values of constants and embed them into the generated file. Probably the best way to do that is to extend the parser (which uses Parse::RecDescent) to parse the verbose-mode output of javap, which includes those details.

There are plenty of others.

How you can get involved

Firstly, come and say “Hi!” in the #dbdi IRC channel on irc.freenode.net.

The code is on github. You can get commit access by asking on the #perl6 channel.

There’s also a mailing list at dbdi-dev@perl.org which you can subscribe to.

I look forward to hearing from you!

1. 
   When I say “Perl 6 has the potential to become a very popular language” I do so with typical British Understatement.

Our work involves manipulating and warehousing external data feeds and developing web interfaces to create home search tools for prospective buyers and lead management tools for real estate agents. We’re looking for a skilled coder to join our small team of talented engineers in Ann Arbor. We hope to find an experienced programmer who is a good fit with our team, well-versed in multiple languages, able to learn quickly and work independently. We work in a Linux environment, and tools and languages we use include Perl, Ruby on Rails, PostgreSQL, and GIT. Perl experience is a significant plus, but your current comfort level with any of these specific tools is less important than overall technical aptitude and ability to learn quickly and fit in well with the current team.

That’s a little thin on details partly because the work is varied. If you think you might be interested, take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

We’re looking for a skilled coder / database wrangler to play a key role within our Operations and Engineering teams. The various responsibilities of the job include working with the large databases underlying our real estate search tools, setting up services for new clients, communicating with clients to evaluate bug reports, troubleshooting technical issues escalated by our client services team, and interfacing with the engineering team on systems maintenance and development. The scope of work that we do involves managing hundreds of external data feeds that feed into in-house databases totaling several million property listings. These listing databases power hundreds of real estate search sites used by more than a million home-buyer leads, who are tracked and cultivated by the thousands of Realtors using our management software. This position is critical to the robustness of these systems.

If that sounds like interesting work to you then take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

I have two Twitter accounts, Skud for personal stuff (since March 2007), and fbase which I manage for work, as community director at Freebase.com. Each has around 700+ followers, and regularly gets direct messages and @replies. I actively follow a couple of hundred people as Skud; the fbase account follows a bunch of people but I only check it occasionally, via the website, so that can be ignored for now.

I also have a identi.ca account as skud. Many of my open source contacts primarily use identi.ca, and I would like to use it to keep in touch with them. I’ve also created a freebase account on identi.ca.

So, here are my requirements:

• I want to use a single client to manage all my tweeting and denting without having to login and out.
• My desktop environment is OSX. I would accept a desktop client, or a good web- or browser-based one. I’ll consider Linux-based clients running under Parallels, but it’s a bit of a PITA to set up.
• I want to be able to tweet/dent in one place and have it show up in both by default
• … but be able to direct it to just one location if appropriate, on a tweet-by-tweet basis.
• I want to minimise accidents where I accidentally tweet something personal on my work account (eek!)
• I want to follow people on both services (Twitter and identi.ca) in close-to-realtime, and be able to respond/join conversations/retweet in a timely fashion.
• I want notifications of @replies and direct messages on both services, for all my accounts.
• I also need to follow various searches in close-to-realtime, mostly to respond in a timely way to people talking about or asking questions about Freebase, but I also like to track mentions of my real name, conferences I’m attending, etc.
• I don’t absolutely need a highly graphical client, but I do appreciate the way seeing people’s user icons helps me remember who’s who.
• I don’t need anything special for mobile use; I have a twitter client on my phone that I’m sufficiently happy with.

Currently I’m using Tweetdeck, and it’s not too bad. Here’s my current setup (direct messages removed):

As you can see, it’s kind of out of control! It spreads across my desktop, and I need to scroll sideways to see everything, even with narrow columns. Plus, it doesn’t handle identi.ca, and it doesn’t give me very good cues to remember whether I’m tweeting as Skud or fbase, which means I’ve occasionally made embarrassing blunders.

With regard to UI, I suppose I would like something closer to an email client or feed reader, that allows me to set up folders/tags to manage all the stuff I’m following, and shows counts of new messages in each folder. The ability to collapse groups of stuff and see an aggregate count would be ideal; obviously I’d start by grouping “personal” and “Freebase”.

The other pretty good client I tried was an early version of Nambu, which had the UI I wanted but unfortunately doesn’t seem to want to let me login to identi.ca. Here’s a screenshot of my setup:

Downsides: the aforementioned failure to work with identi.ca, and I would prefer a little more flexibility in what I show in my sidebar. Like, I’d like to completely hide/ignore “Home” for the fbase account or at least automatically mark everything there as read.

I’m waiting on a beta code for the new version of Nambu and hoping it will work properly with identi.ca, but in the meantime, does anyone have suggestions of what else might work for me?

ETA: This is an advanced question I’m asking here. Yes I’ve looked in the obvious places. I’m looking for a specific recommendation for a fairly specialised set of requirements here, so unless you know the feature set of a client pretty well, chances are that just mentioning one with which you have only a passing acquaintance is unlikely to be helpful.

I have two Twitter accounts, Skud for personal stuff (since March 2007), and fbase which I manage for work, as community director at Freebase.com. Each has around 700+ followers, and regularly gets direct messages and @replies. I actively follow a couple of hundred people as Skud; the fbase account follows a bunch of people but I only check it occasionally, via the website, so that can be ignored for now.

I also have a identi.ca account as skud. Many of my open source contacts primarily use identi.ca, and I would like to use it to keep in touch with them. I’ve also created a freebase account on identi.ca.

So, here are my requirements:

• I want to use a single client to manage all my tweeting and denting without having to login and out.
• My desktop environment is OSX. I would accept a desktop client, or a good web- or browser-based one. I’ll consider Linux-based clients running under Parallels, but it’s a bit of a PITA to set up.
• I want to be able to tweet/dent in one place and have it show up in both by default
• … but be able to direct it to just one location if appropriate, on a tweet-by-tweet basis.
• I want to minimise accidents where I accidentally tweet something personal on my work account (eek!)
• I want to follow people on both services (Twitter and identi.ca) in close-to-realtime, and be able to respond/join conversations/retweet in a timely fashion.
• I want notifications of @replies and direct messages on both services, for all my accounts.
• I also need to follow various searches in close-to-realtime, mostly to respond in a timely way to people talking about or asking questions about Freebase, but I also like to track mentions of my real name, conferences I’m attending, etc.
• I don’t absolutely need a highly graphical client, but I do appreciate the way seeing people’s user icons helps me remember who’s who.
• I don’t need anything special for mobile use; I have a twitter client on my phone that I’m sufficiently happy with.

Currently I’m using Tweetdeck, and it’s not too bad. Here’s my current setup (direct messages removed):

As you can see, it’s kind of out of control! It spreads across my desktop, and I need to scroll sideways to see everything, even with narrow columns. Plus, it doesn’t handle identi.ca, and it doesn’t give me very good cues to remember whether I’m tweeting as Skud or fbase, which means I’ve occasionally made embarrassing blunders.

With regard to UI, I suppose I would like something closer to an email client or feed reader, that allows me to set up folders/tags to manage all the stuff I’m following, and shows counts of new messages in each folder. The ability to collapse groups of stuff and see an aggregate count would be ideal; obviously I’d start by grouping “personal” and “Freebase”.

The other pretty good client I tried was an early version of Nambu, which had the UI I wanted but unfortunately doesn’t seem to want to let me login to identi.ca. Here’s a screenshot of my setup:

Downsides: the aforementioned failure to work with identi.ca, and I would prefer a little more flexibility in what I show in my sidebar. Like, I’d like to completely hide/ignore “Home” for the fbase account or at least automatically mark everything there as read.

I’m waiting on a beta code for the new version of Nambu and hoping it will work properly with identi.ca, but in the meantime, does anyone have suggestions of what else might work for me?

So I thought I’d summarize my interpretation of the comments on that thread, as much for my own benefit as yours, and see if this post flushes out any further information.

We’ll start with the smaller/personal projects and work up from there…

John Cappiello

John mentioned that he was working on something. I sent John an email to ask for an update and he said it had “morphed away from a CRM into something not really overlapping much at all”.

Gábor Szabó

Gábor Szabó mentioned in the thred that he has “a simple CRM I use in-house that I plan to release as open source one day. It is written in Perl. While it is very minimalistic if you are interested I can show it and we can discuss what additional features TPF might need.”

Affinity

Affinity was mentioned in the thread, but I can’t find any references to it being open source.

CustomX

The web pages are in German, so here’s a translation to English of their web site, which gets visually mangled in the process, and a much more readable translation of their 2006 whitepaper.

It’s available under the Perl Artistic License. The base language is German. I don’t know what level of internationalization/locaization they support.

They currently use mod_perl v1 and say it’s “not tested” for v2, which seems to suggest a lack of maintenance. Databases supported include Postgres, Oracle, MySQL, and Informix. It’s extensible via plugins.

I couldn’t find any indication of a development community.

Wice CRM Groupware

Another German CRM system, so here’s a translation of their home page. It’s in use by a range of companies, and they even have a glossy magazine (though it looks like a once-off produced in 2005).

“Wice is a LAMP system with Linux as operating system, Apache as web and application server, MySQL as the database and the system is developed as an Apache module in mode_perl.”

“We have Wice a plugin architecture, which we almost arbitrary adjustments, enhancements and interfaces can be realized [...] There are also numerous standard plugins, such as a web, an integrated CMS, or a Customer Self Service Center.”

The source code is not ‘free’, however. You get the source code only when you licence the software.

Donor.com

“Our software has been serving non-profits for 30+ years. We have spent the past 5 years wrapping all of our C-based business logic with Perl to expose everything as Web Services (50 modules and 300+ methods so far). Our presentation layers are the WebGUI CMS (also Perl) and a cross-client GUI (Win/OSX/Linux) written in wxPerl. Our systems integrate everything from CRM, inventory management, event tracking, credit card processing, bulk email, direct mail, magazine subscriptions, sponsorships, etc. Almost everything in our system is configurable. We have not yet open-sourced all of our code, but we did just transition our ownership to a new 501(c)3, the DonorWare Foundation, to help facilitate that in the future.”

Jim Brandt replied to comment in the thread saying “I got another response from the folks who run donor.com. Turns out they are heavy perl users, so we’re looking at their system right now to see if it will meet our needs. I’ll post with more details once we know.” No news yet.

Josh Berkus expressed some pros and cons of using donor.com.

Other Perl CRM Systems?

What have I missed? Are there any other open source CRM systems out there implemented in Perl?

For comparison, here are some details (mostly skimmed from Wikipedia) of open source CRM systems not implemented in perl…

CiviCRM

Designed to meet the needs of advocacy, non-profit and non-governmental groups. Implemented in PHP. The license for the current version is the GNU AGPL 3.

CiviCRM may be deployed either standalone or alongside Drupal and Joomla! content management systems. Both the Drupal and Joomla! professional associations use CiviCRM. The standalone version is intended to work alongside other CMSs.

CiviCRM is used by many large NGOs including Amnesty International, Creative Commons and the Wikimedia Foundation for their fundraising.

They’re currently considering a new architecture:

In short, we propose using PHP Doctrine for the ORM layer [...] we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.

Link to the CiviCRM project page at Ohloh.net.

vtiger CRM

An open source fork of SugerCRM. Implemented in PHP, initially released in 2003. Seems to be supported by AdventNet, who make a non-open source Java based CRM called Zoho. Licences: vtiger Public License 1.1 and SugarCRM Public License 1.1.2.

Hosted at sourceforge. Link to vtiger project page at Ohloh.net.

epesi BIM

Implemented in PHP. Links to the developers wiki, forums, and Ohloh.net project page.

A search on Ohloh.net for tag:crm yields 649 projects with the CRM tag!

It turns out that Ohloh is a great way to look for projects. The Analysis Summary on each project page gives a useful overview of some key metrics: “Large, active development team”, “Few source code comments” etc.

Sorting the list in various ways and taking the projects (other than those above) that appear on the first page of each, gives this list, in no particular order:

OpenERP

“Python. Formerly known as TinyERP. OpenERP is an Open Source enterprise management software. It covers and integrates most enterprise needs and processes: accounting, hr, sales, crm, purchase, stock, production, services management, project management, marketing campaign, management by affairs, …

Once in production, it automates and help you to control all activities: sales triggers manufacturing orders, accounting entries are updated by stock operations, incoming mails are tracked in the system, the integrated document management system helps your team to collaborate, …”

Open For Business Project (Apache OFBiz)

Java. “The Open For Business Project is a set of tools and enterprise applications including ERP, CRM, e-commerce, SCM, MRP, and CMMS/EAM. It uses a service oriented and events driven architecture and tools to automate all aspects of application development and maintenance.”

Omni ERP

Python. “Omni ERP is an innovative business application platform; it is completely based in open source technologies and brings a whole new level of modularity and flexibility to the business solutions environment.
It uses a new approach combining new software engineering techniques like plugin based architecture, inversion of control and aspect oriented programming with a refreshing new RIA UI to bring a new level of experience to the SME market”

ADempiere

Java. “ADempiere Business Suite ERP/CRM/MFG/SCM/POS done the Bazaar way in an open and unabated fashion. Focus is on the Community that includes Subject Matter Specialists, Implementors and End-Users. We are a community fork of Compiere”

ERP5

Python. “ERP5 is a full featured high end Open Source / Libre Software solution published under GPL license and used for mission critial ERP / CRM / MRP / SCM / PDM applications by industrial organisations and government agencies.

It is distributed to linux community via packages for numerous distributions (Mandriva, Debian, Ubuntu,…) and a dedicated Live CD.”

JFire

Java. “JFire is an ERP, CRM, eBusiness, and SCM/SRM solution for business enterprises. It uses JavaEE, JDO, and Eclipse RCP, and is designed to be highly customizable. It is a complete and extensible solution that fulfills business needs like user management, online trade with business partners, points of sale, various distribution channels forming a distribution network, store management, etc.”

CK-ERP

PHP. “CK-ERP is an open source accounting / MRP / ERP / CRM system that runs on top of multiple middlewares. It provides accounting and back office functionalities to SMEs and utilizes the underlying middleware to administer accounts/groups.

It comprises 22 modules – Administration, Multilingual Facility, Contact Management, Customer Relationship, Customer Self Service, Vendor Relationship, Material Requirement Planning, Warehouse, Inventory, Service, Accounting Ledger, Bank Reconciliation, Accounts Payable, Accounts Receivable, Purchase Order, Sales Order, Quotation, POS for Cashier, POS for Manager, Human Resources, Staff Self Service and Payroll.

Operating platform can either be LAMP or LAPP. Backend database engine can be anyone of MySQL, PostgreSQL and SQLite”. Oddly, their home page says “minimal documentation will be made available to users of CK-ERP” and their roadmap is old.

Blue ERP

PHP. “Blue ERP is an open source, web based ERP application. Its goal is to provide a flexible and user friendly interface that can work out of the box and be modified to suit specific needs easily.

The main goals of the projects are:* provide a feature full ERP application
* be open in licence and in spirit – in blue ERP everything is open, especially the development
* be user friendly by providing adequate documentation and assistance to users to encourage widespread adoption”

OTRS::ITSM

After all the PHP systems, it’s nice to find one implemented in Perl! Though the description suggests it’s geared towards the needs of an IT Service organization.

“ITIL compliant Open Source IT Service Management Solution (ITSM). Requires the OTRS (Open Ticket Request System) framework.

Besides all OTRS features, new ones include Incident, Problem and Configuration Management, integrated Configuration Management Data Base; process spanning Communication management: within the IT Service organization, towards Customers/Users/Management and Suppliers/Providers; powerfull statistical features for (Trend-)Analysis, key figured Reporting, ITSM-Planning/-Controlling; flexible configuration, customizing and expandability regarding your individual requirements.”

[That's quite enough "CRM" systems! Ed.]

I’ve learnt to rewrite the ‘-’ characters because although they look like ‘-’ characters they’re really a unicode HYPHEN: U+2010. Some other chars get mangled too, but that’s the most frequent problem for me.

So I was delighted to see a blog post by marcus ramberg called Fixing the POD synopsis in OSX wherein he fingers nroff as being the problem and gives a simple solution:

alias perldoc='perldoc -t'

Trouble is using perldoc -t means you loose the nice bold text that nroff gives you. So I went digging…

It seems the problem only affects people using UTF-8 and that nroff has a -T option that lets you specify an output encoding to use. So if perldoc ran ‘nroff -Tascii’ instead of plain ‘nroff’ that would avoid the hypen problem and let us keep the bold text.

It turns out that perldoc has an option to specify the nroff command to use, so the solution is simple:

alias perldoc="perldoc -n 'nroff -Tascii'"

This is, of course, still a hack. My main worry is that pod docs using non-ascii characters may get mangled. A much better fix would be to arrange for the ascii characters to not get mapped to unicode at all. So I went digging again…

nroff calls groff -m tty-char ... and tty-char refers to a tty-char.tmac file that defines the character mappings. The groff man pages point me to groff_tmac man pages which tell me I can get groff to look for .tmac files elsewhere by passing a -Mdir option or setting the GROFF_TMAC_PATH environment variable.

I looked at the default file, /usr/share/groff/1.19.2/tmac/tty-char.tmac on my Mac, and… decided it was time to go to sleep! The formatting is probably simple enough but I’m out of tuits.

So, what’s needed is for someone to determine what change is needed to the tty-char.tmac file, or the files it refers to, to avoid unwanted conversions to unicode. Then put a modified file into a directory and either add a -Mdir option to the nroff alias above, or set the GROFF_TMAC_PATH environment variable. Setting the env var has the benefit of ‘fixing’ all the man pages.

So, anyone want to dig deeper? (For all I know the solution can already be found on google…)

Once you start measuring something, and advertising the results, people with an interest in particular outcomes naturally start to look for ways to influence those results. (It’s the Observer Effect writ large.)

The fact that TIOBE’s methodology, which I’ve discussed previously here and here, is simplistic makes it particularly open to gaming. Anyone, or any community, with access to many web pages can simply add the magic phrase “foo programming”, where foo is their language of choice, to get counted.

And it seems that’s exactly what the Delphi community did at the end of 2008¹. They made a concerted effort and it seems to have paid off. (I’d be very interested in hearing about similar behaviour in other language communities.)

Is that behaviour gaming? The author of the post who exhorted is readers to “Update your Delphi related blog or site to say Delphi programming on every page in visible text (update the template). Stand up and be counted. You can make a difference!” doesn’t seem to think so, as he also said “I am not suggesting we game the system, just that we help TCPI get an accurate count.“

An accurate count of what, exactly? That’s always been the fundamental question with TIOBE. It should be obvious that most web pages that talk about “delphi programming” wouldn’t actually contain the phrase “delphi programming”. The same applies to every other language. That’s the paradox at the heart of the TIOBE Index. And yet, somehow, TIOBE seem to think that counting pages containing the phrase “delphi programming” lets them claim that:

The ratings are based on the number of skilled engineers world-wide, courses and third party vendors.

Eh? How can they possibly defend that claim? Certainly their documented definition doesn’t support it, or even mention it.

I presume they’re thinking that CV’s, job postings, and adverts are most likely to contain the magic phrase. It should be obvious, again, that the number of CV’s, job postings, and adverts referring to a given programming language would naturally only be a small fraction of the total web pages referring to the language. (And only distantly related to the “popularity” of a language.) Yet that “small fraction” is what TIOBE measure and make bold claims about.

The fact that TIOBE is making a comparison based on a small fraction makes it even more troubling that TIOBE CEO Paul Jansen appears to support language communities changing their pages to include the magic “foo programming” phrase. In an email quoted on delphi.org he says:

For your information, I think your action has already some effect. Tonight’s run shows that Delphi is #8 at this moment. There is a realistic chance that Delphi will become “TIOBE’s Language of the Year 2008″

He’s endorsing the artificial insertion of the magic phrase. Clearly this distorts the TIOBE index in favour of language communities that infect as many pages as possible with the magic phrase.

That sure seems like an invitation to game the system! It’s likely to lead to other language communities doing the same, and so to further devaluation of the TIOBE Index.

(For alternatives to TIOBE you could look at sites like http://www.langpop.com/, James Robson’s Language Usage Indicators, or my popular comparison of job trends blog post with ‘live’ graphs.)

I have, on a couple of occasions, used the phrase “perl programming” in blog posts for my own amusement, and linked it to my original TIOBE or not TIOBE – “Lies, damned lies, and statistics” post. I haven’t suggested that others do the same. TIOBE’s endorsement of artificial insertion changes that. Now it seems like we’re going to get a dumb “race to the bottom” to see which language community controls the most web pages.

If, as a result, the TIOBE Index is affected significantly, then I simply hope they’ll drop their pretentious claims and state clearly exactly what they’re counting, how they’re doing it, and what it means: not much.

1. Many thanks to Barry Walsh for his blog post that alerted me to this.

After almost exactly two years working for Shopzilla I’ve moved on.

I’ve enjoyed my time with Shopzilla, met some great people, and worked on some challenging problems. I’m especially grateful to them for giving me the opportunity to develop and release some very useful Open Source software: Gofer, DashProfiler, and especially NYTProf v2.

When I started with ValueClick back in 1999 it was a small company. Over the years I was with them it grew rapidly, both from within and via mergers. When I left in 2006 it had become a very large company. For me Shopzilla was always a very large company.

I really like to make a significant positive impact where I work. I believe I did that at ValueClick and Shopzilla, but it’s always easier at a smaller company. There’s far less inertia to overcome.

So I’m delighted to be working now at a small company with a great team, product, technology, approach, and growth: TigerLead.com

I’m not keen on job titles, but I needed to put something on my LinkedIn profile. After exploring some options I settled on “Entropy Minimizer”!

I’m also delighted to be working with the PostgreSQL database, at last, as I’ve not had a chance to work with it before. (Although I used to be good at tuning Ingres, the distant forerunner of PostgreSQL, back in the days when RDBMS were novel and we thought a 50MHz cpu was fast.)