Suggestivity and Idioms in APL

Two major principles emerge in APL programming, suggestivity and idioms. I recently encountered a nice illustration of how these two concepts work together. This demonstrates how the use of compositionality in the service of suggestivity ca help to improve and enhance the communicative and expressive power of idiomatic notation, which is, ultimately, the goal of any good notation.

Suggestivity is a language design principle put forth by Ken Iverson in his Turing Award Lecture, Notation as a Tool of Thought. The principle says that suggestive expressions suggest new ideas and themes to a reader by encouraging variations to a given expression that may be alternative solutions to the same problem or may be new expressions that address a different problem but that is now understood to have some connection to the previous problem by virtue of the similarity between the solutions expressed as variations of one another.

A few things make suggestive notations work better. Compositionality is one element, since only the ability to mix and match known ideas enables the necessary freedom to explore how solutions may share something in common. Another feature that does not receive as much attention as compositionality is structural transparency. Without seeing the underlying structure of the solution to any given problem, it becomes very difficult to see the ways in which two ideas (i.e., expressions) can be related. Without structural transparency, expressions become so atomic that one cannot deconstruct the ideas within it to allow variations to be suggested to the reader.

This is where idioms become quite valuable. Since the ease of working with astructure is directly related to its complexity and size, a solution that is structurally transparent but very verbose and large will be harder to see and manipulate using the human mental faculties. APL idioms have the benefits of concision and structural transparency.

Let's look at a nice example of suggestivity at work through idioms. Suppose we have a Boolean vector that masks or selects groups of elements. This is a common structure that appears often in APL programs. we often want to speak about different parts of groups, or to identify the groups themselves. Consider the following example:

1 1 1 1 0 1 1 1 0 1 0 1 1 0 0 0 1 1 1 1

These groups begin at the following indices:

0 5 9 11 16

We often want to know these starting points for each group. How might we do this? Well, the starting position of each group is the point at which we see a 0 1 in the vector, not including the start. One approach that directly encodes this idea might be the following:

⍸(⊃⍵)@0⊣¯1⌽0 1⍷⍵

In this expression, we look for all the 0 1's that may appear, and then account for the first element of the input. But is this the best way to do this? If we examine this in terms of suggestivity, we can see some room for improvements. For one thing, the handling of the first element is somewhat heavy-handed and not particularly compositional or general. The main issue with it is how much it makes the first element a special case, and how the solution is completely different than the main logic 0 1⍷⍵. Worse, it is also much more complex than the main logic, and so obscures the main point of the code. We, at least as a rule, do not want the edge cases of our code to take up more space and energy than the main point of the code. What happens when we want to find the end of a group instead of the start? Now we may get something like this:

⍸(⊃⌽⍵)@(¯1+≢⍵)⊢1 0⍷⍵

Notice how much variation there is between this expression and the previous. While the core idea is still there, not much of the structure remains, and the code feels quite different than what came before. This is a good indicator that the solution is not as highly suggestive as it might be.

We can improve this approach by recognizing that we can handle the edge cases by using a catenation:

⍸0 1⍷0,⍵
⍸1 0⍷⍵,0

Notice immediately how much better this feels. We have taken advantage of the ⍸ to eliminate the need to drop or take elements to match the original shape of ⍵. Without that, we would need to do the following:

¯1↓0 1⍷0,⍵
¯1↓1 0⍷⍵,0

This feels much more suggestive. The structural similarity between these two expressions is much greater than in the previous examples.

What other variations are suggested by this expression? There are some. If we were to use 1 1⍷⍵ instead of 0 1 or 1 0, then we might get the groups minus the last element. But it is harder to imagine what use 0 0 might be in this case. With some extra manipulation, we may get something, but much less obviously so than 1 0, 0 1, or 1 1. How about changing 0,⍵ or ⍵,0 around? In the 0 1 case, using ⍵,0 instead of 0,⍵ will not give us much, since it will only have the effect of ignoring the first element, and in so doing is no different than using 0 1⍷⍵ on its own. The same holds true for the 1 0⍷. We see a similar sort of lack of symmetric usefulness and clarity when we use 1 instead of 0 for our catenation.

When we examine this solution in terms of suggestivity, then, we see that there is some suggestivity in the notation, but it is not as strong or as general as we may like. Can we do better? As it turns out, yes, we can.

Any APLer who has been around a while will already know where I am going with this. The common idiom for finding the start of Boolean groups in a vector goes like this:

⍸2<⌿0⍪⍵

And the idiom for finding the ends of the groups looks like this:

⍸2>⌿⍵⍪0

Here, we can already see some advantages. We can easily see how this can scale to higher ranks, and if we ignore the handling of ⍸, we can apply these idioms to different axes simply by using / and , instead of ⌿ and ⍪. Additionally, the shape of 2>⌿⍵⍪0 or 2<⌿0⍪⍵ matches the input shape without needing to drop elements like we did with the ⍷ solutions.

The suggestivity begins to really explode when we consider using things other than < or >. Unlike the 0 1 and 1 0 inputs, using > and < with ⌿ immediately suggests meaningful variations, all of which have interesting connections made more concrete by their structural similarity. If we can use the < and > relations, what happens if we use the other Boolean relations? Here is a table of the combinations we might have:

Y             0 1 1 1 0 0 0 0  0 1 1 1 0 0 0 1  1 1 1 1 0 0 0 0  1 1 1 1 0 0 0 1  
{2<⌿0⍪⍵}¨Y   0 1 0 0 0 0 0 0  0 1 0 0 0 0 0 1  1 0 0 0 0 0 0 0  1 0 0 0 0 0 0 1  
{2<⌿1⍪⍵}¨Y   0 1 0 0 0 0 0 0  0 1 0 0 0 0 0 1  0 0 0 0 0 0 0 0  0 0 0 0 0 0 0 1  
{2<⌿⍵⍪0}¨Y   1 0 0 0 0 0 0 0  1 0 0 0 0 0 1 0  0 0 0 0 0 0 0 0  0 0 0 0 0 0 1 0  
{2<⌿⍵⍪1}¨Y   1 0 0 0 0 0 0 1  1 0 0 0 0 0 1 0  0 0 0 0 0 0 0 1  0 0 0 0 0 0 1 0  
{2≤⌿0⍪⍵}¨Y   1 1 1 1 0 1 1 1  1 1 1 1 0 1 1 1  1 1 1 1 0 1 1 1  1 1 1 1 0 1 1 1  
{2≤⌿1⍪⍵}¨Y   0 1 1 1 0 1 1 1  0 1 1 1 0 1 1 1  1 1 1 1 0 1 1 1  1 1 1 1 0 1 1 1  
{2≤⌿⍵⍪0}¨Y   1 1 1 0 1 1 1 1  1 1 1 0 1 1 1 0  1 1 1 0 1 1 1 1  1 1 1 0 1 1 1 0  
{2≤⌿⍵⍪1}¨Y   1 1 1 0 1 1 1 1  1 1 1 0 1 1 1 1  1 1 1 0 1 1 1 1  1 1 1 0 1 1 1 1  
{2=⌿0⍪⍵}¨Y   1 0 1 1 0 1 1 1  1 0 1 1 0 1 1 0  0 1 1 1 0 1 1 1  0 1 1 1 0 1 1 0  
{2=⌿1⍪⍵}¨Y   0 0 1 1 0 1 1 1  0 0 1 1 0 1 1 0  1 1 1 1 0 1 1 1  1 1 1 1 0 1 1 0  
{2=⌿⍵⍪0}¨Y   0 1 1 0 1 1 1 1  0 1 1 0 1 1 0 0  1 1 1 0 1 1 1 1  1 1 1 0 1 1 0 0  
{2=⌿⍵⍪1}¨Y   0 1 1 0 1 1 1 0  0 1 1 0 1 1 0 1  1 1 1 0 1 1 1 0  1 1 1 0 1 1 0 1  
{2≠⌿0⍪⍵}¨Y   0 1 0 0 1 0 0 0  0 1 0 0 1 0 0 1  1 0 0 0 1 0 0 0  1 0 0 0 1 0 0 1  
{2≠⌿1⍪⍵}¨Y   1 1 0 0 1 0 0 0  1 1 0 0 1 0 0 1  0 0 0 0 1 0 0 0  0 0 0 0 1 0 0 1  
{2≠⌿⍵⍪0}¨Y   1 0 0 1 0 0 0 0  1 0 0 1 0 0 1 1  0 0 0 1 0 0 0 0  0 0 0 1 0 0 1 1  
{2≠⌿⍵⍪1}¨Y   1 0 0 1 0 0 0 1  1 0 0 1 0 0 1 0  0 0 0 1 0 0 0 1  0 0 0 1 0 0 1 0  
{2≥⌿0⍪⍵}¨Y   1 0 1 1 1 1 1 1  1 0 1 1 1 1 1 0  0 1 1 1 1 1 1 1  0 1 1 1 1 1 1 0  
{2≥⌿1⍪⍵}¨Y   1 0 1 1 1 1 1 1  1 0 1 1 1 1 1 0  1 1 1 1 1 1 1 1  1 1 1 1 1 1 1 0  
{2≥⌿⍵⍪0}¨Y   0 1 1 1 1 1 1 1  0 1 1 1 1 1 0 1  1 1 1 1 1 1 1 1  1 1 1 1 1 1 0 1  
{2≥⌿⍵⍪1}¨Y   0 1 1 1 1 1 1 0  0 1 1 1 1 1 0 1  1 1 1 1 1 1 1 0  1 1 1 1 1 1 0 1  
{2>⌿0⍪⍵}¨Y   0 0 0 0 1 0 0 0  0 0 0 0 1 0 0 0  0 0 0 0 1 0 0 0  0 0 0 0 1 0 0 0  
{2>⌿1⍪⍵}¨Y   1 0 0 0 1 0 0 0  1 0 0 0 1 0 0 0  0 0 0 0 1 0 0 0  0 0 0 0 1 0 0 0  
{2>⌿⍵⍪0}¨Y   0 0 0 1 0 0 0 0  0 0 0 1 0 0 0 1  0 0 0 1 0 0 0 0  0 0 0 1 0 0 0 1  
{2>⌿⍵⍪1}¨Y   0 0 0 1 0 0 0 0  0 0 0 1 0 0 0 0  0 0 0 1 0 0 0 0  0 0 0 1 0 0 0 0  
{2∧⌿0⍪⍵}¨Y   0 0 1 1 0 0 0 0  0 0 1 1 0 0 0 0  0 1 1 1 0 0 0 0  0 1 1 1 0 0 0 0  
{2∧⌿1⍪⍵}¨Y   0 0 1 1 0 0 0 0  0 0 1 1 0 0 0 0  1 1 1 1 0 0 0 0  1 1 1 1 0 0 0 0  
{2∧⌿⍵⍪0}¨Y   0 1 1 0 0 0 0 0  0 1 1 0 0 0 0 0  1 1 1 0 0 0 0 0  1 1 1 0 0 0 0 0  
{2∧⌿⍵⍪1}¨Y   0 1 1 0 0 0 0 0  0 1 1 0 0 0 0 1  1 1 1 0 0 0 0 0  1 1 1 0 0 0 0 1  
{2⍲⌿0⍪⍵}¨Y   1 1 0 0 1 1 1 1  1 1 0 0 1 1 1 1  1 0 0 0 1 1 1 1  1 0 0 0 1 1 1 1  
{2⍲⌿1⍪⍵}¨Y   1 1 0 0 1 1 1 1  1 1 0 0 1 1 1 1  0 0 0 0 1 1 1 1  0 0 0 0 1 1 1 1  
{2⍲⌿⍵⍪0}¨Y   1 0 0 1 1 1 1 1  1 0 0 1 1 1 1 1  0 0 0 1 1 1 1 1  0 0 0 1 1 1 1 1  
{2⍲⌿⍵⍪1}¨Y   1 0 0 1 1 1 1 1  1 0 0 1 1 1 1 0  0 0 0 1 1 1 1 1  0 0 0 1 1 1 1 0  
{2∨⌿0⍪⍵}¨Y   0 1 1 1 1 0 0 0  0 1 1 1 1 0 0 1  1 1 1 1 1 0 0 0  1 1 1 1 1 0 0 1  
{2∨⌿1⍪⍵}¨Y   1 1 1 1 1 0 0 0  1 1 1 1 1 0 0 1  1 1 1 1 1 0 0 0  1 1 1 1 1 0 0 1  
{2∨⌿⍵⍪0}¨Y   1 1 1 1 0 0 0 0  1 1 1 1 0 0 1 1  1 1 1 1 0 0 0 0  1 1 1 1 0 0 1 1  
{2∨⌿⍵⍪1}¨Y   1 1 1 1 0 0 0 1  1 1 1 1 0 0 1 1  1 1 1 1 0 0 0 1  1 1 1 1 0 0 1 1  
{2⍱⌿0⍪⍵}¨Y   1 0 0 0 0 1 1 1  1 0 0 0 0 1 1 0  0 0 0 0 0 1 1 1  0 0 0 0 0 1 1 0  
{2⍱⌿1⍪⍵}¨Y   0 0 0 0 0 1 1 1  0 0 0 0 0 1 1 0  0 0 0 0 0 1 1 1  0 0 0 0 0 1 1 0  
{2⍱⌿⍵⍪0}¨Y   0 0 0 0 1 1 1 1  0 0 0 0 1 1 0 0  0 0 0 0 1 1 1 1  0 0 0 0 1 1 0 0  
{2⍱⌿⍵⍪1}¨Y   0 0 0 0 1 1 1 0  0 0 0 0 1 1 0 0  0 0 0 0 1 1 1 0  0 0 0 0 1 1 0 0  
{2⊢⌿0⍪⍵}¨Y   0 1 1 1 0 0 0 0  0 1 1 1 0 0 0 1  1 1 1 1 0 0 0 0  1 1 1 1 0 0 0 1  
{2⊢⌿1⍪⍵}¨Y   0 1 1 1 0 0 0 0  0 1 1 1 0 0 0 1  1 1 1 1 0 0 0 0  1 1 1 1 0 0 0 1  
{2⊢⌿⍵⍪0}¨Y   1 1 1 0 0 0 0 0  1 1 1 0 0 0 1 0  1 1 1 0 0 0 0 0  1 1 1 0 0 0 1 0  
{2⊢⌿⍵⍪1}¨Y   1 1 1 0 0 0 0 1  1 1 1 0 0 0 1 1  1 1 1 0 0 0 0 1  1 1 1 0 0 0 1 1  
{2⊣⌿0⍪⍵}¨Y   0 0 1 1 1 0 0 0  0 0 1 1 1 0 0 0  0 1 1 1 1 0 0 0  0 1 1 1 1 0 0 0  
{2⊣⌿1⍪⍵}¨Y   1 0 1 1 1 0 0 0  1 0 1 1 1 0 0 0  1 1 1 1 1 0 0 0  1 1 1 1 1 0 0 0  
{2⊣⌿⍵⍪0}¨Y   0 1 1 1 0 0 0 0  0 1 1 1 0 0 0 1  1 1 1 1 0 0 0 0  1 1 1 1 0 0 0 1  
{2⊣⌿⍵⍪1}¨Y   0 1 1 1 0 0 0 0  0 1 1 1 0 0 0 1  1 1 1 1 0 0 0 0  1 1 1 1 0 0 0 1  

This should make it clear how many interesting things come out of this. Almost all of the binary relations that we can imagine in use here find some meaningful interpretation. Even some functions that we may not think of as related or relevant can suddenly emerge as equivalent under this domain. What I find most interesting here is how many of these are obviously useful in our context of Boolean groups. Many of these idioms are actually viable in practice, too, unlike some things that may be theoretically valuable, but rarely find the opportunity for use in a real application.

Further, the visual symmetries that abound here are useful when multiple of these idioms find use in the same application, since they serve to create visual clarity in the code because of their regularity.

Using these idioms in practice also enahnces the readability of your code. Even if you use one of the less used combinations above, the similarity to the more common cases means that a reader who is familiar with the common case has very little to do to understand the new case, since most of the underlying logic is exactly the same, and signalled to be so by the idiom's structure. All of this is a powerful application of the principle of suggestivity when used to help guide one's aesthetics and style when writing code.

I think these sorts of benefits of compositionality are the unique purview of languages like APL that are uniform and somewhat consistent in their primitive design, but most improtantly, are extremely concise. I do not believe that you can leverage this kind of suggestivity with languages that are more verbose. Their verbosity encourages too much abstracting away of underlying structure, and this leads to hiding the connections between these functions. Additionally, even if we do retain structural transparency, the resulting code is too large to make experimentation and comparison easy. It is fundamentally easier to try out many single line expressions than it is to try out many 15 - 30 line expressions. Even expanding a single line into 5 or so lines makes experimentation and iteration significantly more costly. There is a huge value in the reduction of the cost of variation down to a few characters, in the same way that any technique that reduces the length of the feedback cycle improves our ability to iterate and therefore learn.

Thus, learning how to leverage idioms and suggestivity as a method fo iteration and incremental improvement is one of the best ways to level up the quality and clarity of your code.