Teaching Programming Languages by Experimental and Adversarial Thinking

Presented on May 11, 2017
Presenter: Kyle

Preview

Kyle will be ‘lead[ing] the charge on this noble expedition’ about teaching PL. He will be presenting “Teaching Programming Languages by Experimental and Adversarial Thinking,” from SNAPL 2017.

Summary

The basic idea is to use adversarial thinking in order to find implementation differences between interpreters of a mystery language. The mystery language is intentionally poorly-defined, and given an ambiguous English-based specification. The different interpreters each use different possible implementations of the same vague English specification. For example, two different interpreters may take different approaches to an array access at a negative value (array[-1]), where one might throw an exception, and another may access elements of the righthand side of the array. Given these interpreters (but not their source code), students must experimentally find programs which illustrate different behavior on the interpreters. Students then must write an English description of their hypothesis of how the different interpreters work (e.g., “interpreter A gives an error on negative array access, and interpreter B accesses off the righthand side of the array”). Based on the results, students were generally able to derive programs showing different behavior, though it wasn’t until about halfway through the class where the student hypotheses were generally on the mark.

Advantages

According to the authors, this approach has the following advantages:

  1. This mimics how people generally learn programming languages, which includes lots of experimentation. This is intended to improve this sort of skillset so students can more easily grasp new languages in the wild.
  2. Not all assignments need to take this approach; it can be done with all assignments or as few as one assignment.
  3. There is not an expensive “buy-in”; students don’t need lots of time to get up to speed (in contrast to, say, students needing to learn Scala in CS162 before they can realistically do anything else)

Students generally had a positive reaction, and nearly all claimed that this improved their toolbox of skills to learn new languages.

Disadvantages

  1. Even for one assignment, this requires an incredible amount of work. This demands a parser, multiple interpreters, and associated grading infrasturcture for each assignment. The authors did this for 56 languages, and I honestly don’t know how all this can be maintained without lots of extra help (which they had).
  2. The authors didn’t actually demonstrate that this actually helps anything; students self-report positively, but it isn’t clear if this builds up usable skills (more on this later)
  3. Without some sort of auxilliary assessment (e.g., exams), students do not have strong motivation to learn proper terminology and theory. For example, students need to derive hypotheses about scoping without really knowing what scoping is. Students are never required to actually learn what scoping is (heaven forbid if their hypotheses are wrong), nor do they need to learn proper terminology, without an auxilliary assessment.

Discussion

Several points came up during the discussion.

  1. The authors emphasise an approach of teaching languages as if they were natural objects, which we collectively received with a negative opinion. This seems to deny you tons of information. It feels like there should be some halfway point.
  2. The authors fundamentally assume that people learn languages in a relatively experimental fashion. Everyone agreed with this except for Kyle, who prefers deep dives into documentation. The end result for Kyle (and only Kyle) is that these assignments felt painful, though everyone else thought they were fun. Kyle did not feel like he was learning anything or gaining any usable skills; it felt like being given arithmetic problems to solve. Kyle proclaimed he’d rather write a fuzzer and have it automatically discover differences, as opposed to exploring somewhat randomly by hand.
  3. While this seems like a good approach for building up practical skills, it is heavily focused on pragmatics. It is difficult to teach any real theory without additional assessments in this style.
  4. The assignments used for the class did not go very deep into any topics. The most complex assignment was based on simple classes. To me (Kyle), it is not clear how this would work for features which students did not already have at least some understanding of. Without some understanding, there is no way to know what kinds of programs are interesting or not. Indeed, most of the challenge with solving these assignments was figuring out which programs are interesting (to which Kyle proclaimed he’d rather read up on background and then surgically write tests).
  5. Overall, these assignments are laser-focused on open exploration.

Attachments