Ringing “Half-sheets” with Perl 6

a talk about this program

The problem statement

I have data in a source in the cloud. I want to create a PDF using that data.

The background

One of my hobbies is English Change Ringing.

Change ringing is a particular type of ringing, which arose in England, which instead of ringing melodies, rings permutations.

It’s difficult to explain, but I’ll try to give an overview in the next few slides.

Make the sound travel

When a bell is struck by either a clapper or a hammer, the noise travels primarily in the direction the mouth of the bell is facing. For bells hung “dead” (such as carillon bells), this means that most of their volume goes down, into the ground. If, instead, you put a bell on a wheel, you can get it to strike when pointing outwards, thus make it audible much further away.

However, it takes a lot of energy to keep a bell on a regular wheel going. In England, from the 1200s to the 1600s, they started adding extra pieces of wood (called the “stay” and the “slider”) that let the bells be stored mouth up, and developed a way of ringing the bells that used potential energy to save a lot of effort.

The bells as St Bees Priory, Cumbria, in mouth up position. Photo by Dougism, from Wikimedia

There are some nice animations here showing bells moving.

There is one disadvantage though; when the bell is rung this way, it takes around 2 seconds from when it is struck to when it can be struck again. This makes ringing melodic tunes difficult (if not impossible).

Something to do with all these bells…

Instead of attempting to ring melodies, English ringers developed a system of ringing which permutes the order of the bells.

You start ringing a descending scale. You then start permuting the order of the bells, using a memorized “method”, so that each time all of the ringing bells have stuck, it is in a different order. The method eventually will return you to the descending scale, without repeating any permutation.

(Technically, there are two different styles of change ringing: “method” ringing, and “called changes”. I am only describing “method” ringing here.)

Methods

A method is a way of changing the order of the bells. Some of them are simple, and some of them are complex. Every methods starts from “rounds” (the descending scale) and proceeds through a set of changes until it arrives back at rounds. At various points in the change, a call can be made. A call (usually “bob” or “single”) changes the pattern for one change, after which the ringers resume the pattern (although in a new position). Calls allow methods to be extended to make “touches”, which are longer (or shorter) than the “plain course” of a method.

For example, here is a very simple method, called “Plain Bob Doubles” (“doubles” means that it is rung on five bells).

Plain Bob Minor, generated by Blueline
Plain Bob Minor (in a different representation), from Ringing.org

Or, a more complex one, called “Cambridge Surprise Major” (“major” is rung on eight bells; “surprise” is a category of methods “that have internal places made when the treble changes dodging pairs”).

Cambridge Surprise Major, generated by Blueline
Cambridge Surprise Major (in a different representation), from Ringing.org

So what does it sound like

Here are a bunch of YouTube videos of ringing. We can watch a few of them, if you want…

Performances

In ringing, if you ring a “touch” that comprises all of the possible permutations for the number of bells you are ringing on, it is called ringing an “extent”.

From math, we know that the number of permutations of a given number of items is n!. So, 3 bells = 6 changes, 4 bells = 24 changes, etc.

Ringing an extent of 7 bells is 5040 changes. This takes about 3 hours (give or take). Three hours of ringing, with no breaks or substitutions, is a reasonable “hard” thing to do (for many people, it is not really hard). So this became known as a “peal”. On 8 or more bells, anything over 5000 changes is considered a peal.

(The extent on 8 (40,320 changes) has been rung in tower (as opposed to handbells) in 1963; it took 18 hours, on very light bells.)

Three hours, though, can be hard to fit into a day, and if the ringing is spotty, very annoying for the neighbors. Much more common is a “quarter-peal”, which is anything from 1260 changes (1250 on 8 or more bells) to 5040 changes (5000 on 8 or more bells). This usually takes around 45 minutes to ring.

BellBoard

Having spent three hours (or forty-five minutes) concentrating on counting to the exclusion of everything else, you want to make sure people know you did this. Towers have local documentation (in the form of peal books, etc.), but it is nice to get recognition from others. (Historically, that should read “be able to brag to others, and force them to buy you beer, or get into fights with them”; ringing has calmed down a lot in the last several centuries.)

As a result, we share our accomplishments. Since 1911, one way to do this has been The Ringing World, a weekly publication that has columns, letters, and is about half composed of peal and quarter-peal notices.

In addition, they now run a site called “BellBoard”, which is a web-based system that lets you submit your records, and have them included in the print version of The Ringing World. (Presumably, you can also send them in by mail.)

Dedications

Often, quarter-peals or peals are dedicated to someone or something. These are included as footnotes in the records, and are for things like deaths, birthdays, weddings, etc.

When my grandfather died in 2009, my sister and I put together a quarter-peal in his honor. After we rang the quarter-peal, I made an overly formal announcement of it, in the style of something from the erzatz 18th century, printed it on nice paper, put in in a presentation folder, and gave it to my grandmother.

I did this on several other occasions as well (e.g.). But it was a pain to type everything out each time, and I never really got into the habit of doing it on a regular basis.

Bragging to the tourists

We have two towers in Boston: the Church of the Advent, and Christ Church in the City of Boston (which most people know as “Old North”).

Old North has tours that come through the ringing room, where they learn a little bit about ringing.

(Old North has the oldest set of change ringing bells in North America, installed in 1745. Paul Revere was one of the first ringers!)

We want to show the tourists that we are still ringing, and give them a sense that this is a living tradition. We have put up pictures, and try to change them from time to time. It was suggested that we could put up copies of interesting performances we do.

This gave me the impetus to try and create a program to automate creating ringing announcements. Which is what I'm going to talk about now.

One last thing…

Well, almost. Here are some links to things about ringing that might be of interest, if you want to digress:

The data

So, let’s start with the data.

The data is kept on BellBoard, https://bb.ringingworld.co.uk. For example, this quarter-peal in honor of a Boston ringer who died in World War I, or this Easter peal with a number of firsts.

Although I have written programs to scrape web pages for data, it kind of sucks. However, this page documents their “API”. From this, we can see that we can get the data in XML fairly easily.

XML examples

Here is some example XML.

From the quarter-peal:

<?xml version="1.0"?>
<performance xmlns="http://bb.ringingworld.co.uk/NS/performances#" id="P1244942">
  <association></association>
  <place towerbase-id="5852">
    <place-name type="place">Boston</place-name>
    <place-name type="dedication">Christ Church, Old North</place-name>
    <place-name type="county">Massachusetts</place-name>
    <ring type="tower" tenor="13-3-5 in F" />
  </place>
  <date>2018-09-02</date>
  <duration>45m</duration>
  <title><changes>1260</changes> <method>Stedman Doubles</method></title>
  <ringers>
      <ringer bell="1">Laura Dickerson</ringer>
      <ringer bell="2" conductor="true">Elaine M Hansen</ringer>
      <ringer bell="3">Phoebe House</ringer>
      <ringer bell="4">Joshua R Burson</ringer>
      <ringer bell="5">Michael Tartell</ringer>
      <ringer bell="6">Bryn Marie Reinstadler</ringer>
      <ringer bell="7">Austin J Paul</ringer>
      <ringer bell="8">Katarina Whimsy!</ringer>
    </ringers>
  <footnote>First of Stedman: 4, 5</footnote>
  <footnote>First in tower: 6, 8</footnote>
  <footnote>Rung in memory of Boston Guild ringer Private Murray Edward Gordon Mackman     (26 July, 1890  - 1 September, 1918)</footnote>
  <donation>£3.50</donation>
  <timestamp>2018-09-03T19:02:49</timestamp>
</performance>

And from the peal:

<?xml version="1.0"?>
<performance xmlns="http://bb.ringingworld.co.uk/NS/performances#" id="P1225038">
  <association>North American Guild</association>
  <place towerbase-id="5852">
    <place-name type="place">Boston</place-name>
    <place-name type="dedication">Christ Church, Old North</place-name>
    <place-name type="county">Massachusetts</place-name>
    <ring type="tower" tenor="13-3-5 in F" />
  </place>
  <date>2018-04-01</date>
  <title><changes>5088</changes> <method>Yorkshire Surprise Major</method></title>
  <composition ref="2266172" />
  <composer role="composed">Donald F Morrison</composer>
  <ringers>
      <ringer bell="1">Cally D Perry</ringer>
      <ringer bell="2">T David Westmoreland</ringer>
      <ringer bell="3">Alison Stevens</ringer>
      <ringer bell="4">Leland Paul Kusmer</ringer>
      <ringer bell="5" conductor="true">Edward J Futcher</ringer>
      <ringer bell="6">John Bihn</ringer>
      <ringer bell="7">Elaine M Hansen</ringer>
      <ringer bell="8">Austin J Paul</ringer>
    </ringers>
  <footnote>Rung for Easter.</footnote>
  <footnote>First peal in the method 3,4,6</footnote>
  <timestamp>2018-04-02T22:33:37</timestamp>
  <rwref>5586.476</rwref>
</performance>

The PDFs

From the XML data, I want to generate PDFs. However, sometimes I want to modify the data before creating the PDF; for instance, to change the date to read “Easter, 2018”, or to fix the extra spacing in the quarter-peal footnote.

This means that I need some intermediate format, ideally text-based for easy editing, that can be turned into a PDF easily.

In particular, these were the requirements that I came up with:

Text based intermediate format options

Here are the various things that I considered:

HTML + css
Although there is support for using HTML + css to create printed material (see this article, for example), there isn’t a lot of support for this yet.
SVG + css
Although would be cool, it suffers from the same problem as HTML, which is that it is difficult to turn into a PDF easily.
(La)TeX
Probably a good choice, although my impression is that getting the kind of page control I want is more complicated. However, it’s a huge download (1G).
DocBook
Even less clear how to do proper layout than HTML, and have you ever tried to get it installed?
roff
Already installed on my computer; probably not too hard to learn; has good page control…

I could probably have made TeX work, but I decided to look at roff, because it was already on my computer.

(There are probably a bunch of other solutions, but they didn’t occur to me...)

roff

roff is a text-based format, currently mostly used for generating man pages on Unix, but that has a history of page design. It’s fairly flexible, and the GNU version supports output directly to PDF.

There are some annoyances that I ran into: in particular, it is difficult to get it to work with system fonts (that is, it is impossible to work with system fonts, you need to translate them into some special format and put them in a particular place, which I don’t have a license to do with the font I wanted to use). Also, support for Unicode is complicated (although this is partially because PDF support is complicated as well...).

roff template

The first step I took was to develop a roff file to generate the ringing half-sheet.

Here is a commented version of the template.

As you can see in the initial comment, you turn this into a PDF using the command:

groff -Tpdf -Kutf8 demotroff.groff > demotroff.pdf

And here is what is generated.

Guild logos

You’ll notice that there is a logo in the upper-right hand of the generated file. This is a precreated PDF image that is embedded in the output PDF.

To create these PDFs, I took existing logos, and manually recreated them in SVG. For instance, here is the Boston Change Ringers logo as an SVG (and the SVG source).

To convert them into PDFs, I wound up using a web-based conversion service (as I mentioned, going from SVG to PDF is not easy). In particular, I used FileFormat.Info, which works ok. (Its support for non-PS-standard fonts is a little lacking; the Boston Change Ringers font was converted incorrectly. In basically everything fonts (or, more accurately, typefaces) are complicated.)

Here is the converted BCR logo as a PDF.

Perl 6; the glue beneath my wings

So, now I have a roff template file, all I need to do is fill it in. I usually reach for Perl 5 at this point, but I thought that this would be a nice project to use Perl 6 for.

(And, thus, this is why this talk is happening.)

The prologue

#!/usr/bin/env perl6 
use v6;
sub croak { note $^msg; exit(1); } # because Perl 6 doesn't have the Perl 5 "\n" magic for die 
 
use HTTP::UserAgent;
use XML::XPath;
use Template::Mustache;

There isn’t much here to talk about. croak() fixes what I see as a problem in the Perl 6 version of die().

MAIN()

The MAIN() subroutine is a formal entry point to the program, if you want one (you don’t need to provide it). The advantage of using it is that you can specify command line parameters.

sub MAIN ( Str  :p(:$performance)!,
           Bool :g(:$groff) = False,
           Bool :f(:$force) = False,
           Str  :i(:$image)? where ( !$image.defined or ($image eq 'none') or "{$image}.pdf".IO.f or croak("{$image}.pdf does not exist for inclusion as image") )
) {
	my $file = "groff/$performance.groff";
	if $file.IO.e and !$force {
		croak "File $file already exists. Will not overwrite.";
	}
 
	my $xml    = get-performance-xml($performance);
	my %parsed = parse-performance-xml($xml);
	my $output = create-groff(%parsed, $image);
 
	if %parsed<pid> ne $performance {
		croak "Retrieved performance has different id: requested $performance and received %parsed<pid>. Will not continue.";
	}
 
	# save the data 
	$file.IO.spurt: $output;
	say "$file written.";
 
	if $groff {
		shell "/usr/local/bin/groff -Tpdf -Kutf8 $file > pdf/$performance.pdf";
		say "groff command run.";
	}
}

get-performance-xml()

This retrieves the data from BellBoard.

sub get-performance-xml ($p) {
	my $data = HTTP::UserAgent.new.get("https://bb.ringingworld.co.uk/view.php?id={$p}", Accept => 'application/xml');
	$data.is-success or croak("HTTP error retrieving post: {$data.status-line}.");
	return $data.content;
}

parse-performance-xml()

This takes the XML from BellBoard, and pulls out the various variables that we will need.

sub parse-performance-xml ($xml) {
	my $xpath = XML::XPath.new(xml => $xml);
	my %data;
 
	# gather the straightforward items 
	my %spec = (
		'pid'         => 'substring(/performance/@id,2)',    # the performance ID sadly starts with a 'P' in the XML 
		'guild'       => '/performance/association/text()',
		'date'        => '/performance/date/text()',
		'tower'       => '/performance/place/@towerbase-id',
		'towernamepl' => '/performance/place/place-name[@type="place"]/text()',
		'towernamede' => '/performance/place/place-name[@type="dedication"]/text()',
		'towernameco' => '/performance/place/place-name[@type="county"]/text()',
		'nchanges'    => '/performance/title/changes/text()',
		'method'      => '/performance/title/method/text()',
		'composer'    => '/performance/composer/text()',
		'details'     => '/performance/details/text()',
		'notes'       => '/performance/footnote/text()',
	);
 
	for %spec.kv -> $k, $v {
		my $r = $xpath.find($v);
		given $r.WHAT {
			when XML::Text { %data{$k} = $r.text().trim(); }
			when Str       { %data{$k} = $r; }
			when Array     { %data{$k} = $r.map({ .text().trim(); }).list; }
			default        { if defined($r) { croak("Unknown type {$_.perl} for key $k"); } }
		}
	}
 
	# gather the ringers 
	for | $xpath.find('/performance/ringers/ringer') -> $r {
		my $ringer = $r.contents().map( {.text().trim();}).join(' ');
		if $r.attribs<conductor> { $ringer ~= ' \*[conductor]'; }
		%data<ringers>{$r.attribs<bell>} = $ringer;
	}
 
	return %data;
}

create-groff()

This takes the data extracted from the XML, uses it to fill out the template, and generates the output roff.

sub create-groff (%perf, $image) {
	my %rdata;
	%rdata<pid> = %perf<pid>;
 
	%rdata<urpic><img> = do given %perf<guild> {
		when defined($image)            { $image  };
		when 'North American Guild'     { 'nagcr' };
		when 'MIT Guild of Bellringers' { 'bcr'   };
		when 'Boston Change Ringers'    { 'bcr'   };
		default                         { 'none'  };
	};
	if (%rdata<urpic><img> eq 'none') { %rdata<urpic> = Nil; }
	if %perf<guild> { %rdata<guild><guild> = %perf<guild>; }
 
	%rdata<date> = Date.new(%perf<date>, formatter => &date-formatter);
	my $towername = "%perf<towernamede>, %perf<towernamepl>, %perf<towernameco>";
	if %perf<tower> ~~ (5851|5852) {
		%rdata<tower>{'t' ~ %perf<tower>}<towername> = $towername;
	}
	else {
		%rdata<tower><tdef><towername> = $towername;
	}
	%rdata<performance_type> = do given %perf<nchanges> {
		when         $_ < 1250 { 'performance' };
		when 1250 <= $_ < 5000 { 'quarter-peal' };
		when 5000 <= $_        { 'peal' };
		default                { 'weird non-number of changes' };
	};
 
	%rdata<method><method> = "%perf<nchanges> %perf<method>";
	if %perf<composer> { %rdata<method><composed><composer> = %perf<composer>; }
	if %perf<details>  { %rdata<method><details><details> =  %perf<details>; }
 
	for %perf<ringers>.keys.sort(&infix:«<=>») -> $n {
		%rdata<ringers>.push: { num => $n, ringer => %perf<ringers>{$n} };
	}
	%rdata<ringers>[0]<num> = '\*[treble]';
	my $num = numbells(%perf<method>);
	if ($num % 2 == 0) || (%rdata<ringers>.elems >= $num) {
		%rdata<ringers>[* - 1]<num> = '\*[tenor]';
	}
	if %perf<notes>.elems {
		%rdata<notes><footnotes> = [ %perf<notes>.map({ %( 'note' => $_ ); }) ];
	}
	my $out = Template::Mustache.render($=finish, %rdata, :literal);
	$out ~~ s:g/ \n ** 2..* /\n/;    # clean up blank lines, which are anathema to troff 
	$out .= trans([ '&lt;', '&gt;', '&amp;', '&quot;' ] => [ '<', '>', '&', '"' ]); # fix XML entities 
	return $out;
}

date-formatter()

This lets me format dates the way I want, with affixes. I found the algorithm on StackOverflow.

sub date-formatter ($self) {
	my $year = $self.year;
	my $month = qw<nul January February March April May June July August September October November December>[$self.month];
	my $day = $self.day;
 
	# see https://stackoverflow.com/a/13627586/1030573 
	my $day_m10  = $day mod 10;
	my $day_m100 = $day mod 100;
	my $affix    = '\*[th]';         # default affix 
	if    ($day_m10 == 1) && ($day_m100 != 11) { $affix = '\*[st]'; }
	elsif ($day_m10 == 2) && ($day_m100 != 12) { $affix = '\*[nd]'; }
	elsif ($day_m10 == 3) && ($day_m100 != 13) { $affix = '\*[rd]'; }
 
	return "$month $day$affix, $year";
}

numbells()

Earlier, I wanted to only make the last bell marked as the “tenor” if in an “even” method, or if there were more bells ringing than the number of bells “inside” the method. To know how many bells are inside bells, you need to transform the last word of the method name back into a number.

Note that this will not work for “called changes”, I haven’t yet added support for that. In order to properly indicate the tenor, I may need to manually fix the .groff file.

sub numbells ($method) {
	my $stage = ($method ~~ m/(\w+)$/).Str;
	my @counts = qw<nul impossible impossible Singles Minimus Doubles Minor Triples Major Caters Royal Cinques Maximus>.map: &fc;
	return @counts.first(fc($stage), :k) // 16;
}

The template

This is just the roff template we went through earlier, but removing all of the comments, and putting in the various template data.

Because of how Template::Mustache works [mustache], I can create certain “custom” towers (see the area around the {{# towers}} block), which lets me do something different for e.g. Old North.

=finish
\# in 'root' dir: groff -Tpdf -Kutf8 groff/{{& pid}}.groff > pdf/{{& pid }}.pdf
\X'papersize=5.5in,8.5in'
.pl 8.5i
.po 0.5i
.ll 4.5i
.sp |0.5i
.nr def_ps 10
.nr def_vs 15
.nr sml_ps 9
.nr sml_vs 13
.nr flr_ps 20
.nr flr_vs 20
.fam N
.ft NR
.ps \n[def_ps]pt
.vs \n[def_vs]pt
.lg
.kern
.nh
.de GUILD
.ps \\n[def_ps]pt
.vs \\n[def_vs]pt
.nop \f[I]for the\f[]\h[|0.5i]\f[B]\\$1\f[]
..
.de STANZA
.ps \\n[def_ps]pt
.vs \\n[def_vs]pt
.sp
.in 0
.ft NI
.nop \\$1
.br
.in 0.5i
.ft NR
..
.de ftsmall
.ps \\n[sml_ps]pt
.vs \\n[sml_vs]pt
..
.de finalflourish
.ps \\n[flr_ps]pt
.vs \\n[flr_vs]pt
.sp
.in 0
.ce
.nop \f[ZD]\m[red3]\N[167]\m[]\f[]
.br
..
.ds st         \f[B]\v[-.25v]\s[-4]st\s[+4]\v[.25v]\f[]
.ds nd         \f[B]\v[-.25v]\s[-4]nd\s[+4]\v[.25v]\f[]
.ds rd         \f[B]\v[-.25v]\s[-4]rd\s[+4]\v[.25v]\f[]
.ds th         \f[B]\v[-.25v]\s[-4]th\s[+4]\v[.25v]\f[]
.ds treble     \s[-3]TREBLE\s[+3]
.ds tenor      \s[-3]TENOR\s[+3]
.ds conductor  \f[I]\s[-1](conductor)\s[+1]\f[]
.nf
{{# urpic}}
\h[|3.0i]\X'pdf: pdfpic {{& img }}.pdf -L 1.5i 1.5i'
.sp 0.5v
{{/ urpic}}
{{# guild}}
.GUILD "{{& guild }}"
{{/ guild}}
.STANZA "on"
{{& date }}
.STANZA "at"
{{# tower }}
{{# t5851 }}
The Church of the Advent
{{/ t5851 }}
{{# t5852 }}
Christ Church in the City of Boston
.ftsmall
\f[I](called \[lq]Old North\[rq])\f[]
{{/ t5852 }}
{{# tdef }}
{{& towername }}
{{/ tdef }}
{{/ tower }}
.STANZA "was rung a {{& performance_type }} of"
{{# method }}
{{& method }}
{{# composed }}
.ftsmall
composed by {{& composer }}
{{/ composed }}
{{# details }}
.fi
.ftsmall
{{& details }}
.nf
{{/ details }}
{{/ method }}
.STANZA "by the ringers"
.in 0
.ta 1iR 1.2i
{{# ringers }}
	{{& num }}	{{& ringer }}
{{/ ringers }}
.br
{{# notes}}
.STANZA "with notes"
.fi
{{# footnotes }}
{{& note }}
.sp 0.25
{{/ footnotes }}
.nf
{{/ notes}}
.finalflourish

All at once

Here is the program as a single file.

Output

Earlier, I mentioned two performances, a quarter-peal and a peal.

Here are the various steps for each:

September quarter-peal

Easter peal

Next

There are some things that I still need and want to do with this...

Questions?

Fin