The paper starts out by explaining what software pipelining / Modulo Scheduling is. One point of definitional confusion is that it appears to actually be presenting the Modulo Variable Expansion¹ algorithm rather than vanilla Modulo Scheduling (which is a special case of MVE).

Then it defines the problem by introducing fancy variables to say “for every iteration count, the result of the pipelined and the original loop should be the same”:

• is the new loop body […]
• is the loop prolog […]
• is the loop epilog […]
• is the minimum number of iterations that must be performed to be able to use the pipelined loop.
• is the amount of unrolling that has been performed […]

[…]

Likewise, the generated code behaves either like if , or like

if , where

The verification problem therefore reduces to establishing that, for all , the two basic blocks

are semantically equivalent.

Here, is the trip count, is the result of symbolic execution, and is the original, single basic-block loop body.

Nice. Now if we could just . But there are countably too many of those to symbolically execute, so let’s read on.

The key discovery of this paper is that this undecidable property is implied by (and, in practice, equivalent to) the following two decidable conditions:

In essence: exiting the pipelined loop and running the original loop body ( times if it was unrolled) should be the same as continuing it and exiting. You throw in some conditions about the induction variable and the loop bound, and you have the algorithm in section 5.

That is the core insight of the paper. The rest of it concerns itself with their symbolic execution method, the proof, and some caveats about the approach.

The interesting one is that passing the validation condition is sufficient but not necessary for the transformation being correct (i.e. being equivalent to Equation 1). The paper says they “strongly believe it holds in practice as long as the software pipelining algorithm used is purely syntactic”. Which is a somewhat unsatisfactory thing to say in a theoretical CS paper, but it sounds reasonable.

They implemented a software pipeliner and this validator for CompCert, and have not found it to reject correct transformations, which gives this claim a bit of empirical weight.

The rest of the caveats are pretty much classic caveats to any symbolic execution technique.

I hope to get a chance to implement this, and the fact that software pipelining algorithms are usually purely syntactic makes the job easier: from a very high level, we are comparing terms of def-use chains.

Read the paper here.

What is Marie Kondo’s favorite tool? Clang-tidy

What wizard reduces binary size? The wizard of Oz

Why didn’t Frodo understand the ring? He doesn’t readelf

Why did a prison hire a compiler? To do escape analysis

What does an elf use to move houses? Relocations

What does the program say when it jumps to an invalid address? Every day we stray further from the light of GOT

Why does the compiler love origami? It’s constant folding

What is a compiler’s favorite sport? Inlining

But of course an operating system ultimately rooted in the 70s will have accumulated its fair share of legacy. Of course, not all problems are rooted in this: we still are perfectly capable of making new and exciting mistakes.

std::vector<pid_t> pids = get_all_pids();
for (pid_t pid : pids) {
  if (matches(pid)) {
    kill(pid);
  }
}

pid_t pid = fork();
if (pid < 0) {
  oh no;
  abort();
}
if (pid == 0) {
  something in child;
} else {
  int status;
  YOUR_EINTR_WRAPPER(waitpid(pid, &status, 0));
}

$ ./a.out 
bash: ./a.out: No such file or directory

If how this relates to Emacs is a mystery, just trust me that this will make sense. But first to more elementary things.

I may not be Golang’s biggest fan in general (lack of generics, verbose syntax, simplistic type system, etc), but I’ve written a bunch of it in the last couple of years and found an unexpectedly useful feature. Golang provides a package called regexp/syntax that proves to be as useful as its documentation is sparse.

In essence, this package exposes the user to the finite state machines built by the regexp compiler. This can be used to do analyses on regular expressions such as “does this regexp ever match a particular character after matching n characters?” or “does this regexp match any strings starting with a particular prefix?”. These might sound like constructed examples, but both of them actually popped up in my dayjob.

r, err := syntax.Parse(`.*`, syntax.Perl)
if err != nil {
    panic(fmt.Sprintf("Parse: %v", err))
}

p, err := syntax.Compile(r.Simplify())
if err != nil {
    panic(fmt.Sprintf("Compile: %v", err))
}
fmt.Println(p)

  0 fail
  1 anynotnl -> 2
  2*    alt -> 1, 3
  3 match

func GetSuccessors(i *syntax.Inst) []uint32 {
    if i.Op == syntax.InstMatch || i.Op == syntax.InstFail {
        return []uint32{}
    }
    o := []uint32{i.Out}
    if i.Op == syntax.InstAlt || i.Op == syntax.InstAltMatch {
        o = append(o, i.Arg)
    }
    return o
}

func HasLoop(p *syntax.Prog) bool {
    var tovisit []uint32
    tovisit = append(tovisit, uint32(p.Start))
    seen := make(map[uint32]bool)
    for len(tovisit) != 0 {
        i := tovisit[0]
        if seen[i] {
            return true
        }
        seen[i] = true
        tovisit = tovisit[1:]
        tovisit = append(tovisit, GetSuccessors(&p.Inst[i])...)
    }
    return false
}

func Match(p *syntax.Prog, pc uint32, input []rune, idx int) bool {

    var prev rune = -1
    if idx > 0 && idx <= len(input) {
        prev = input[idx-1]
    }
    var cur rune = -1
    if idx < len(input) {
        cur = input[idx]
    }

    i := p.Inst[pc]
    switch i.Op {
    case syntax.InstAlt:
        fallthrough
    case syntax.InstAltMatch:
        return Match(p, i.Out, input, idx) || Match(p, i.Arg, input, idx)

    case syntax.InstNop:
        fallthrough
    case syntax.InstCapture:
        return Match(p, i.Out, input, idx)

    case syntax.InstEmptyWidth:
        if !i.MatchEmptyWidth(prev, cur) {
            return false
        }
        return Match(p, i.Out, input, idx)

    case syntax.InstMatch:
        return true
    case syntax.InstFail:
        return false

    case syntax.InstRune:
        if cur == -1 || !i.MatchRune(cur) {
            return false
        }
        return Match(p, i.Out, input, idx+1)
    case syntax.InstRune1:
        if cur == -1 || cur != i.Rune[0] {
            return false
        }
        return Match(p, i.Out, input, idx+1)
    case syntax.InstRuneAny:
        if cur == -1 {
            return false
        }
        return Match(p, i.Out, input, idx+1)
    case syntax.InstRuneAnyNotNL:
        if cur == -1 || cur == '\n' {
            return false
        }
        return Match(p, i.Out, input, idx+1)

    default:
        panic("Invalid instruction.")
    }
    panic("Fell off the switch.")
}

func Even(p *syntax.Prog, pc uint32, idx int, visited map[int]map[uint32]bool) bool {

    i := p.Inst[pc]
    switch i.Op {
    case syntax.InstNop:
        fallthrough
    case syntax.InstCapture:
        fallthrough
    case syntax.InstEmptyWidth:
        return Even(p, i.Out, idx, visited)
    case syntax.InstRune:
        fallthrough
    case syntax.InstRune1:
        fallthrough
    case syntax.InstRuneAny:
        fallthrough
    case syntax.InstRuneAnyNotNL:
        return Even(p, i.Out, (idx+1)%2, visited)

    case syntax.InstMatch:
        return idx == 0
    case syntax.InstFail:
        return true 

func copyVisited(visited map[int]map[uint32]bool) map[int]map[uint32]bool {
    n := make(map[int]map[uint32]bool)
    for k, v := range visited {
        if n[k] == nil {
            n[k] = make(map[uint32]bool)
        }
        for k2, v2 := range v {
            n[k][k2] = v2
        }
    }
    return n
}

    case syntax.InstAlt:
        fallthrough
    case syntax.InstAltMatch:
        branchvisited := copyVisited(visited)
        return Even(p, i.Out, idx, visited) && Even(p, i.Arg, idx, branchvisited)

    default:
        panic("Invalid instruction.")
    }
    panic("Fell off the switch.")

    if visited[idx][pc] {
        return true
    }

    if visited[idx] == nil {
        visited[idx] = make(map[uint32]bool)
    }
    visited[idx][pc] = true

Let’s go explore the argument presented and why it is flawed.

Python 3 Is Not Turing Complete

With the right definition of Python 3, this might actually be technically correct. Any computer language executed on a real physical computer can never be truly Turing complete. Because all resources in a computer are finite, you could theoretically enumerate all possible states. If you have studied a bit of computation theory, Turing Machines are more powerful than that (also, you will know that functions that you can enumerate are usually “boring” from a computation theory point of view).

In computer science a fundamental law is that if I have one Turing Machine I can build any other Turing Machine.

I am not completely sure what that even means. If you have a Turing Machine that just does nothing, I can not use that to build any Turing Machine. There is a Turing Machine (also known as the Universal Turing Machine) that can simulate all other Turing machines. Note that this requires the Turing Machines it simulates to be encoded in some suitable way.

If I have COBOL then I can bootstrap a compiler for FORTRAN (as disgusting as that might be). If I have FORTH, then I can build an interpreter for Ruby.

If you have Python 3, you can definitely use that to write an interpreter of Python 2 in it, if you really really want to. But even that has no direct correspondence to the point of Turing Completeness. Note how this talks about specific encodings of a program (i.e. programming languages), while the concept of Turing Completeness concerns itself with mathematical (partial) functions.

A mathematical function is just a mapping from input to output, while a program is a procedure to produce said output. It is completely possible that a Turing Complete system is unable to express a program another one can. The Game of Life is Turing Complete, but it will definitely will never be able to parse your Python program from your hard-disk.

Currently you cannot run Python 2 inside the Python 3 virtual machine. Since I cannot, that means Python 3 is not Turing Complete and should not be used by anyone.

No, no. It really really does not mean either of these things.

Note

Yes, that is kind of funny way of saying that there’s no reason why Python 2 and Python 3 can’t coexist other than the Python project’s incompetence and arrogance. Obviously it’s theoretically possible to run Python 2 in Python 3, but until they do it then they have decided to say that Python 3 cannot run one other Turing complete language so logically Python 3 is not Turing complete. I should also mention that as stupid as that sounds, actual Python project developers have told me this, so it’s their position that their own language is not Turing complete.

I would really like to see these conversations. Either these Python developers are misquoted here, or were wrong (or, as explored above: technically correct – but that has nothing to do with any of the arguments that were made).

The rest of the argument goes on in similar fashion, incorrectly combining concepts of theoretical computer science with implementations, citing how other virtual machines manage to be more general purpose, and so on, and so forth; the latter of which, of course, is true – but invariantly true for CPython 2 and CPython 3.

This was done by building a class hierarchy with an abstract class Expr on the top level, two classes ExprAnd and ExprOr that represent conjunction and disjunction respectively and both contain lists of Expr. Various other classes where used to represent various atoms of the formulas, the particular code was about solar physics so the atoms happen to be things like Wavelength(x, y) for selecting observations of a particular wavelength.

The base class Expr defines __and__ and __or__, that is the methods that need to be implemented to overload the & and | operations that just return ExprAnd and ExprOr of the object and the left hand operand.

ExprAnd and ExprOr also implement the operation corresponding to their node type by just returning a new ExprAnd or ExprOr with the left hand operand added to the list, respectively. They also implement __rand__ and __ror__, so their special implementation is also applied if they are used at the right hand side of the operation – or that was the plan. ExprAnd commutes | operations into the conjuction to keep conjunction as the top level connective.

The problem that I asked to help with was that foo & (bar & baz) did not seem to evaluate into ExprAnd(foo, bar, baz), but rather ExprAnd(foo, ExprAnd(bar, baz)), that is, the automatic transformation to CNF was not working.

Python actually implements logic that when you have an operation foo & bar where the type of bar is a subclass of the type of foo and implements its own version of __rand__, the expression will be evaluating using that. So when you read __rand__ (or any of the operators) is used if the left hand side does not implement the operator, that is wrong. For instance,

In [18]: class Foo(object):
    ...:     def __add__(self, x):
    ...:         print "Foo ", self, x
    ...:         

In [19]: class Qux(Foo):
    ...:     def __radd__(self, x):
    ...:         print "Qux ", self, x
    ...:         

In [20]: Foo() + Qux()
Qux  <__main__.Qux object at 0x112f9f750> <__main__.Foo object at 0x112f7e510>

This behaviour was what I was aiming to exploit by making ExprAnd (for instance) implement both __and__ and __rand__ and make Python use the most specific implementation. Sadly, it does not work this way. If you now introduce a third class that inherits from Foo and does not implement either of the methods itself, the base class’ will be used, for instance

In [21]: class Bar(Foo):
    ...:     pass
    ...: 
In [22]: Bar() + Qux()
Foo  <__main__.Bar object at 0x112f80890> <__main__.Qux object at 0x112f93a90>

The semantics of Python operator overloading will only prefer the __rand__ method of the right hand side if it is an object of a subclass of the type of the right hand side object. Note that this does not take into account in which class the implementation of the left hand side was actually done, which explains the behaviour above.

If you look at the class layout described above, this was exactly the case that I was hitting. Disclaimer: What follows below is not how it was actually fixed in the code.

import inspect

def fn_mro(cls, fn):
    mro = inspect.getmro(cls)
    return [getattr(x, fn).im_func for x in mro if hasattr(x, fn)]

def most_specific(fn):
    def func(self, other):
        name = fn.__name__
        self_fn = getattr(self, name).im_func
        other_fn = getattr(other, name).im_func
        if self_fn == other_fn:
            return fn(self, other)
        if self_fn not in fn_mro(other.__class__, name):
            # We did not inherit this from something in other's MRO.
            if other_fn not in fn_mro(self.__class__, name):
                # They didn't either. PANIC!
                raise TypeError
            return fn(self, other)
        else:
            return getattr(other, name)(self)
    return func

In [7]: class Foo(object):
    @most_specific
    def __add__(self, x):
        print "Foo ", self, x
   ...:         

In [8]: class Qux(Foo):
    @most_specific
    def __add__(self, x):
        print "Qux ", self, x
   ...:         

In [9]: Foo() + Qux()
Qux  <__main__.Qux object at 0x103103690> <__main__.Foo object at 0x103103f50>

In [10]: class Bar(Foo):
   ....:     pass
   ....: 

In [11]: Bar() + Qux()
Qux  <__main__.Qux object at 0x103103e50> <__main__.Bar object at 0x10310b050>

The first question can be answered by something that should be familiar to everyone working in IT: bad naming. The “completeness” in the two theorems actually refers to two different things: completeness in the completeness theorem relates a formal proof system to model theory and says that a statement is true only if it can be proved (in a proof system for which completeness holds), that is – read follows from implies is provable from ; the completeness in the incompleteness theorem refers to whether a theory contains every sentence or its negation, i.e.  – read for each , either or follow from . It is not hard to see that those two statements have no direct relationship to each other – one of them talks about a theory (the ), the other one about a proof system.

The second question is more subtle and can be explained by the slightly metaphysical definition of truth and existence in classical logic. We can rewrite the statement of the completeness theorem as “for each true statement, there exists a proof”. To do so, let’s define to be the predicate that is true iff is a valid proof for . Then a formalisation would be . Decidability, which could be expressed as “there exists a procedure with which, for every true statement, we can find a proof”, on the other hand, could be expressed as (for the reader following at home – this definition works because f can just return an invalid proof for ) .

For a sound proof system (one that only proves true sentences, i.e. ), completeness of a proof system would imply decidability if you left out the part about computability. The difference is that explicitly quantifying over the function and requiring it be computable means there has to be a procedure, while a simple exists quantifier introduces an implicit function. The notion of procedure is a lot stronger than function; a function as defined in mathematics is simply a mapping from one set to the other (whereof there are uncountably infinite for infinite sets), while procedure requires a formal (finite) representation that could be executed by a computer (whereof there are countably infinite).

This mirrors the distinction between classical and intuitionistic logic. In classical logic, there is a platonic idea of truth (and thus existence): it is legitimate to refer to an object that one has no procedure on how to produce, if one can infer it has to exist (whatever it is). Intuitionistic logic, however, is a lot stricter and requires there to be a procedure on how to produce the objects.

The introduction talks about various other interesting approaches other people have taken to recognize keystrokes, including matching on the distinctive sound different keypresses produce or their electromagnetic signature. There is also prior work in using radio interference patterns, but the authors of this paper took the different approach of only using over-the-counter equipment for doing so. I was not aware at all that there was this much research going on on essentially remote key-logging.

Their approach, however, exploits something known as CSI values – sadly, no, that does not stand for Crime Scene Investigation. CSI stands for Channel State Information and is a set of values that WiFi devices use to optimize their transmission among multiple channels.

They observed that it is possible to find signatures for different movements of the fingers in those CSI values, but they faced the problem that they needed to determine when a key-press started and ended, and noise was also a problem. They would denoise the signal by first applying low-pass filtering and then applying Principal Component Analysis on the data retrieved from the CSI of various channels – they noticed that while the changes were not the same in all of them, they were strongly correlated. So they applied PCA, discarding the highest variance component.

Mean Absolute Deviations are then used to find parts of the timeseries that are high in variance and correspond to keypresses by comparing them to empirically obtained thresholds. The thus found timeseries are then used to train a k Nearest Neighbour model, which would classify each unknown sample with the class that the majority of its k nearest neighbours from the training data-set correspond to. They use Dynamic Time Warping as a distance metric, which tries to align two data-sets by non-linearly warping the time axis until they match. This allows them to compare the signatures of keypresses, even if they are pressed longer or shorter than in the training data.

This model is then trained with 30 data-samples per user per key, and numbers for accuracy are given. This is with the caveat that the experiment was done under controlled conditions, in particular the users were asked to type one key at a time and not to move their heads or other body parts. Training the model independently for each user, they managed to achieve 93.5 % keypress recognition accuracy in continuously typed sentences.

Find the paper here.

Turing-complete means a language can be used to express a (probably non-proper – if you manage to find a system that is more expressible than the Turing machine, I am sure there is a Turing award or two waiting for you) superset of the pure mathematical functions that a Turing machine can compute. This does not talk about I/O (which is an important part of an actual computer) or the complexity these functions are executed in. So, I am afraid we will not see an operating system written in the Game of Life.

This definition also means an actual computer is technically not Turing complete, given it cannot simulate a Turing machine’s infinite tape.

So, concluding: Turing-complete does not mean a language is useful to do something on an actual computer, while an actual computer is not actually Turing-complete. The two concepts are quite orthogonal.

Elaborating more (it was late yesterday): I/O can be “simulated” with appropriate encoding in the output, but that is one of the points: arithmetic is Turing-complete, so I can, for all Turing-computable functions, produce an integer that somehow encodes its result. This is an extremely important theoretical construct in computation theory, but also useless in a practical context, because I want to write actual data into actual memory of an actual machine.

The paper I read for this post is about GraphLab which is a framework for expressing parallel computations based on a graph model that allows to exploit the sparse structure of machine learning algorithms, that is to say that steps of the computation only operate on a subset of the state. In this graph data is associated with each vertex and edge, and a user-supplied update function is used to compute new values. This update function can only access and modify data in the neighbourhood of the node, that is the data of the node, its adjacent edges and vertices.

In addition to the data stored in the graph, there is a shared data table, which contains data which, as the name suggests, can be accessed by all applications of the update function – it can not be modified by them though. To modify data in the shared data table, a fold function that iteratively calculates a value from the data in the vertices and the previous value is specified by the user. Optionally, the user can scale the result of the fold or supply functions to merge results of different fold operations.

By imposing the restriction that the update function only operate on the neighbourhood of a node, the graph encodes the data dependencies between different parts of the computation and allows to determine which updates can be applied concurrently. GraphLab offers different consistency models: full consistency makes sure no data can be accessed concurrently, edge consistency ensures no two updates that access shared edges are executed at the same time, and vertex consistency only ensures only one update is applied to a particular node at a given time. From all I can see, vertex consistency only works correctly for trivially parallelizable problems, that is problems consisting of independent threads of computation that do not interact.

A scheduler determines the sequence of sets of updates that should appear to be applied to the graph at the same time. So, for example, if the sequence is [, ] where and are sets of functions applied to vertices, it is ensured that (modulo the consistency model) it appears that all the operations in were applied to the graph at the same time, then all operations in . Different machine learning algorithms require different schedulers.

Note how I said “appear to be applied”, which means that this is not necessarily how they are actually applied. As an example, if vertex consistency is used, and and , then , , , can be evaluated at the same time, while it still appears that was applied after . There is a very strong connection to how databases handle transactions – if the update function does not modify values that are not exclusive to it by the consistency model, we get serializability – that is, the parallel execution produces the same result as some sequential one. Database transactions are also designed to be serializable, and two of them can commit concurrently if they involve disjunct sets of rows.

The paper concludes with case studies of machine learning tasks implemented in GraphLab, which I will skip in this post.

Find the paper here: GraphLab: A New Framework For Parallel Machine Learning.

A lot of the book is about a world in which there are two types of people, knights and knaves. Knights always speak the truth, and knaves always lie. This particular puzzle is about finding out information about the one werewolf among people A, B and C that make the following statements.

• A: C is a werewolf.
• B: I am not a werewolf.
• C: At least two of us are knaves.

We are trying to find answers to the following two questions:

• Is the werewolf a knight or a knave?
• If you have to take one of them as a travelling companion, and it is more important that he be not a werewolf than that he not be a knave, which one would you pick?

First we will formalize the fact that there is exactly one werewolf as the two statements “there exists at least one werewolf” and “the werewolf is unique” (we assume that the universe contains only persons, because they are the only objects in this riddle, so we do not need a predicate for “is person”):

Then, we introduce the Knight predicate (and an equivalence of negative Knight to Knave for convenience)

Next we can formalize the three statements, which are true if and only if the person that says them is a Knight.

If we call the set of equations (Equation 1) to (Equation 6) , then we want to find out whether

Then, to answer the second question, we want to check

So, with that out of the way, let us introduce the tool that will make sure we do not need to do any thinking: the Z3 SMT solver. SMT stands for satisfiability modulo theory and means that you can give it first-order formulas of some theories (like equality, integers, computer arithmetic, …) and it will try to find a satisfying model. Given infinite domains there is no guarantee that will be found (especially the negative answer is particularly challenging – if there is a positive one it will be found by enumeration eventually). If you want to prove using an SMT solver, in general what you do is apply the deduction theorem to obtain (where is the conjunction of the elements of ). All possible models satisfy this if and only if there is no model satisfying the negation, i.e. , which can be rewritten as . In other words, we ask the SMT solver if is satisfyable, and if the answer is no, holds.

Now all that’s left is translating this specification to Z3’s language. First thing we need to do is define a datatype for our objects, that are Persons. By using declare-datatype we get objects that are only equal to themselves, so we can use the equality predicate as above.

(set-option :timeout 600)
(declare-datatypes () ((Person A B C)))

Then, we need to define our three predicates Knight, Knave and Werewolf.

(declare-fun Knight (Person) Bool)
(declare-fun Knave (Person) Bool)
(declare-fun Werewolf (Person) Bool)

Then we can formalize the formulas of into Z3 syntax pretty much verbatim. The statement that there is exactly one werewolf can be expressed with the two assertions that a model has to satisfy

(assert (exists ((x Person)) (Werewolf x)))
(assert (forall ((x Person)) (implies (Werewolf x) (forall ((y Person)) (implies (not (= x y)) (not (Werewolf y)))))))

The three statements by A, B and C become the following. Note that = is used for both logical equivalence and equality between objects of the universe in Z3.

(assert (forall ((x Person)) (= (Knight x) (not (Knave x)))))
(assert (= (Knight A) (Werewolf C)))
(assert (= (Knight B) (not (Werewolf B))))
(assert (= (Knight C) (exists ((x Person) (y Person)) (and (not (= x y)) (and (Knave x) (Knave y))))))

Now we have asserted all we know about the problem, we can begin trying to prove (Equation 7) – (Equation 11) to see which one the solutions is correct. We use the push and pop operations of Z3 to temporarily add assertions between the two operations. I simplified to

(push)
(assert (not (exists ((x Person)) (and (Werewolf x) (Knave x)))))
(check-sat)
(pop)

(push)
(assert (not (exists ((x Person)) (and (Werewolf x) (Knight x)))))
(check-sat)
(pop)

(push)
(assert (Werewolf A))
(check-sat)
(pop)

(push)
(assert (Werewolf B))
(check-sat)
(pop)

(push)
(assert (Werewolf C))
(check-sat)
(pop)

Z3’s answer to the input we gave it is

unsat
sat
unsat
sat
sat

This means that (Equation 7) and (Equation 9) hold in all models, and the others do not. This means that the werewolf is a Knave, and we should take person A as our travelling companion.

While a rather simple example, I think this is a nice demonstration of what one can do with an SMT solver.

Before going into the implementation, it is probably worth spending a few words on the sequent calculus. The basic construct of the sequent calculus is a sequent, which looks like this: . This is the assertion that, from the set of assumptions , we can prove . The sequent calculus consists of rules that allow to simplify the sequents in a way that the simplified sequents are valid iff the original one is. These rules can be repeatedly applied until only atoms exist in the sequent, at which point it is obvious whether it is valid or not (e.g.  is obviously correct, while is not, with and being logical atoms).

One detail I have omitted so far is that I initially wanted to write a prover that works for intuitionistic instead of classical logic. The difference between the two logics is that, intuitionistic logic is stricter than classic logic in that it requires proofs to be constructive. This means that the proofs can be used to construct an object that satisfies the assertion. For instance, is obviously true in classical logic, but intuitionistically one cannot prove this without knowing which of the two sides hold. If you are only here for the Haskell, it is probably not important that you completely understand this.

My initial attempt (I am very inexperienced in Haskell) was to define two functions to encode the proof with the first operation being on left or the right side of the sequent. It would then call the respective other function on the simplified sequents. At this point I only wanted to encode intuitionistic proofs, so I had the left hand side of the proof as a list of expressions, and the right hand side as a single expression. I had the constructor Nil, only to be used on the right side, to express that it is empty. proveLeft tried to apply a rule to the first expression in the list.

data Expr = Implication Expr Expr | Var String | Neg Expr | Nil deriving (Eq,Show)
data Sequent = Sequent [Expr] Expr deriving (Eq,Show)

proveLeft :: Sequent -> Bool
proveRight :: Sequent -> Bool

I ran into all sorts of problems, especially with the negation operator on the left side, which can only be applied when the right hand side is empty. Atoms had to be explicitly skipped by putting them to the end of the list. Because of this it was hard to figure out when to terminate, or, as with intuitionistic logic one has the drop the right hand side if the proof is “stuck”, when to do that.

I tried to make this work for a bit, but then realized if I ever got it to work, it will be extremely brittle and probably broken in one corner case or another. I then tried attacking the problem in smaller parts – the first one of which was transforming a sequent into simpler sequents. I also thought always using the first expression in the list as the one to operate on was not helpful, so I came up with the following functions (with rather poor names). I also decided to do classical logic first – in a sense, it is simpler because the left and the right hand side are both sets of expressions.

expandLeft :: Sequent -> Expr -> Maybe [Sequent]
expandRight :: Sequent -> Expr -> Maybe [Sequent]

The return value is Maybe [Sequent] so the function can return Nothing if the Expr cannot be operated on, or Just [Sequent] if it can be. For optimization purposes I decided to replace the [Expr] from before with a custom type containing three [Expr] – one for atoms (we need not bother trying to apply rules to them), one for negation (for classical logic, this is not needed, but later for intuitionistic we will want to skip those if there is something on the right hand side), and other composite expressions that we can always apply rules to. This is not strictly needed because of the Maybe [Sequent], but it felt a bit wasteful to call a function that only returns Nothing for every atom – would be interesting to do profiling on this.

data Expr = Implication Expr Expr | Var String | Neg Expr deriving (Eq,Show)
data ExprSet = ExprSet [Expr] [Expr] [Expr] deriving (Eq,Show)
data Sequent = Sequent ExprSet ExprSet deriving (Eq,Show)

The rest of the program involved a lot of using functions from Data.Maybe to put the pieces together.

stepRight :: Sequent -> Maybe [Sequent]
stepRight s@(Sequent lhs rhs) = listToMaybe$mapMaybe (expandRight s) ((getComposite rhs) ++ (getNeg rhs))

This runs expandRight for all expressions on the right hand side, and returns the result first one of those that returns a Just [Sequent], or Nothing if all of them return Nothing. The right hand side is the same idea.

Then, the same idea can be used to apply steps on the left side until there are none more to be taken, then on the right hand side. LK stands for the classical version of the sequent calculus there.

stepLK :: Sequent -> Maybe [Sequent]
stepLK s = listToMaybe$catMaybes [stepLeft s, stepRight s]

Even though I start with a single sequent, a step on a sequent can result in more than one. We need to apply stepLK to all sequents in a list until it returns Nothing for all of them. To do so a function steps is defined that returns either the derived list of Sequents after taking a step, or Nothing if no steps can be taken any more.

getFirst :: (Maybe a, a) -> a
getFirst ((Just x), _) = x
getFirst (_, y) = y

steps :: [Sequent] -> Maybe [Sequent]
steps xs = if all (isNothing . fst) nextIter then Nothing
                                             else Just$concat$map getFirst nextIter 
        where nextIter = map (\x -> (stepLK x, [x])) xs

Then, to bring this to the logical conclusion, this function is applied until it returns Nothing, and then it is checked whether all sequents in the call before that were axioms.

iterToNothing :: (a -> Maybe a) -> a -> a
iterToNothing fn x = case fn x of Nothing -> x
                                  Just y  -> iterToNothing fn y

solve :: Sequent -> Bool
solve s = all isAxiom$ iterToNothing steps [s]

Turning this into a solver for LJ (the intuitionistic version of the calculus) is fairly easy at this point. The rules operating on the negation on the left side have to be guarded by a condition that the right hand side be empty, and the stepsLK function has to be changed to drop the right hand side if there are no more steps for a sequent but it is not an axiom yet. In the implementation this leads to two functions being different between the LK and LJ cases, but I have yet to find a way to properly select which of the two to use at runtime: plumbing the functions through the computation by passing them as arguments does not seem the nicest way.

This paper presents no original insights, but is a very readable introduction into the Curry-Howard isomorphism and concepts related to it. It does not go into the complex technicalities but manages to convey a general intuition and interesting historical context. The appendix of the online edition (the edition I read) also includes an insightful correspondence between the author (Philip Wadler) and Howard about the various people involved in its history.

It introduces the concept in three simple statements.

• propositions as types

• proofs as programs

• simplification of proofs as evaluation of programs

What does this mean? This means that there is a bijection between proofs in a formal proof system and programs in a formal system of computation, where propositions in the proof correspond to types in the program.

This is a very general statement, and the paper in question shows it for one particular pair of formal proof system and formal system of computation, in particular natural deduction for intuitionistic logic and singly typed lambda calculus. It then notes that it indeed holds in various other cases, the list of which I will not enumerate here.

If one thinks about the Brouwer–Heyting–Kolmogorov interpretation of intuitionistic logic, which is also introduced in the paper, this correspondence becomes intuitively plausible. The BHK interpretation defines the logical connectives by the existence of procedures to turn proofs of a proposition into another proof.

For example, a proposition holds if there is a procedure that turns proofs of into proofs of . So if we look at a bijection between propositions and types, this means that there is a procedure that turns an object of type into one of type . Also other propositions are expressed as objects in the BHK interpretation. For example, holds if there is a tuple , where is a proof for and is a proof of , so it can be seen as a tuple. can similarly be seen as the option type for an object that is either of type or .

The paper does a much better job of explaining this all in a bit more detail than I ever could in the course of a blogpost here, so I will not attempt to, but I hope this will bring across the very basic idea behind the concepts.

I also recommend browsing the titles in the reference section if you are interested in the content provided by this paper, I have highlighted [23], [33], [40], [42], [54], [56] and [59] (which does not mean the other ones are worse, just that they did not immediately catch my eye) for further study (probably I will only get to the papers, not the textbooks).

Find the paper here: Propositions as Types.

From all I can tell, the paper does a very unfortunate mistake when defining the meaning of the two concepts, as it wrongly gives the same definition for both of them. The paper defines a set of natural numbers being in both and as , where is a computable predicate (otherwise every set would be in because every set is representable by a predicate, albeit not by a computable one). Of course, this does not make any sense, and Wikipedia offers a much more believable definition thereof: “Also, a formula is equivalent to a formula that begins with some existential quantifiers and alternates times between series of existential and universal quantifiers; while a formula is equivalent to a formula that begins with some universal quantifiers and alternates similarly.” In this context formula and set is more or less interchangeable, as a formula describes the set of values that satisfy it.

After clearing up this slight confusion, it also becomes possible to understand the rest of the paper. It defines the concept of a -complete and -complete set as being a set that all sets of the respective class can be reduced to and that is in that class. Let, for example, be -complete, then for every set there is a computable function such that .

A few complete sets are given (without proof). I will pick out the example of where is defined in this paper as the set of words for which the Turing machine of index halts. For those unfamiliar with the theory of computation this may be odd, but as every Turing machine can be described by finite string over a finite alphabet, there is a way to enumerate the Turing machines, that is, assign an index to each of them.

It took some time wrapping my head around why this set is , but essentially it boils down to the fact that it can be expressed by where expresses (the Gödel number of) a trace of a computation of and is a predicate that is true iff is a trace of a halting computation of on input ; without fancy terms the formula means: an is in if and only if there exists an execution trace of that halts on the input . Well, less fancy terms, I guess.

If you know about computability theory you see I hand-waved a few things. For instance, I just assumed that is a computable predicate, but it is not hard to imagine that checking whether a trace of a computation on a Turing machine on an input actually is a halting computation is indeed computable (you check whether the last state is a halting state and then check whether all steps of the trace are allowed in the Turing machine). It is also important to note that the restriction to computable predicates is needed for the definition to make any sense, this is because if non-computable predicates are allowed, every set would trivially be expressible using zero quantifiers by a predicate .

The paper then shows that the set of (indices of) Turing machines that work in time relative to the size of their input is -complete. They first show that the set is in by showing a formula that represents it. It might be confusing to the reader that their formula is . At first look one might think “I thought meant there is one ”. This is a technicality that is not elaborated on in this paper: the universe is assumed to be integers, which are countable. This allows to encode n-tuples as a single integer in various way (e.g. using powers of prime numbers), which are then “decomposed” again by the predicates. The proof that it is complete is done by reducing a set already known to be complete to it (I will omit details to not completely blow the scope of this post).

After this it is also proven that the set of Turing machines that work in polynomial time is -complete, the details of which I also refer to the paper. The main idea of the proof is again to prove that it is in by showing a formula that expresses it and then reduce an already known to be complete set to it.

There are more proofs about properties of Turing machines and their place in the arithmetic hierarchy which I will not elaborate on in the interest of brevity, if you are interested in this sort of stuff I recommend reading the whole paper. One of them sadly references another paper which is behind a paywall, so it might be hard to really grasp what is going on (one has to trust the statement asserted with a reference to that paper is true).

Anyway, the second section of the paper shows corollaries that can be deduced from the previously proved statements about the arithmetical hierarchy. For example, that there is a Turing machine that works in time but cannot be proved to. This is simply because Turing machines that provably do so can be enumerated by definition – proofs are finite strings over finite alphabets, hence can be enumerated. The set of Turing machines that have the property has previously been proven to be -complete, which cannot be enumerated. This is, intuitively, because infinitely many elements have to be checked before giving an answer to the .

I will not enumerate the rest of the proofs and corollaries, so take a look at the paper if this incited interest.

Link to the paper: Arithmetical Hierarchy and Complexity of Computation.

It is, as is fairly obvious, about Brewer’s conjecture. To be honest, I have never actually heard of this before (and I might just have mistaken it for Brouwer, the logician, back when I downloaded this paper, but I guess I will never know). It seems to state the fact that it is impossible to achieve consistency, availability and partition-tolerance at the same time.

The first section of this paper talks about these properties in a fully asynchronous system – that is a system where every agent must function only based on the messages it has received. Most statements made are fairly self-evident. The main theorem of this part states that if the network is partitioned into two or more parts, a write made in one of them will not be reflected in reads in others’. This isn’t a very deep observation, in my opinion, as the side condition that every request be served eventually is obviously inconsistent with partition-resistant consistency. It cannot even be both available and consistent even in the case where all messages are delivered, because a node has no way of delaying the response to the client until a point where it knows it would have received all messages preceding the request by the client.

The second section is a bit more interesting. It analyzes these properties in a what it calls partially synchronous network. These are networks in which every node has a clock running at the same rate and there are bounds on network latency – mind the difference to systems having the absolute result of clocks synchronized to some degree, like TrueTime used in Google’s Spanner. I am sure a physicist would tell me that having two clocks run at exactly the same rate is technically impossible, but we can probably get close enough for practical purposes (and this is a model anyway).

Anyway, in these networks the main theorem of the previous section obviously still holds. Even such a probably-physically-impossible system is incapable of magically bridging arbitrarily long network partitions in finite time. However, the variant where all messages are delivered does not hold, because assuming a central node that manages the system, a node can always serve a request by waiting for the appropriate amount of time before returning because there are bounds on the network latency (of course in practice there is not much point in having a distributed data-base managed completely by a central node).

A weaker form of consistency is then proposed, which is based on consistency while all messages are delivered, and a bound on re-convergence time after a interval in which not all messages have been delivered. This notion is formalized and then an algorithm that achieves this is shown.

Find the paper here: Brewers Conjecture paper.

Raft, the algorithm proposed in the paper, is a solution to the replicated state machine problem. The basic premise thereof is a cluster of computers running independently but working together in what appears to be a single consistent state machine. Without auxiliary conditions, this is a quite simple goal to achieve: one could simply choose one of these computers to actually do the work and leave the rest idle. Of course this is not the point of the exercise, so we want the state machine to be available as long as the majority of these computers is still up, while still giving only correct result. The paper refers to these properties as availability and safety, respectively.

The problem is simplified vastly by requiring that one of these computers is master at any given time, and clients only talking to this master. This is achieved by a fairly simple protocol where, upon detecting the current master has failed, a server initiates an election by talking to all other servers, asking them to vote for it. Other servers will vote only for other servers that are more up-to-date. If more than half of the servers vote for it, it becomes the new master. The protocol also shows that simplicity was the main design goal of Raft: to prevent servers from competing for the vote, which may lead to elections that need to be repeated for lack of winner, the server simply waits for a random time before calling the election, which makes it unlikely that two or more servers do so at the same time.

This server is then responsible for dealing with users requests. If a user wants to update the state, this change is written into the so-called log of the master and replicated onto the other servers called followers. Under normal operation, it will be considered to be committed and persistently in the system when it is written to more than half of the cluster (some caveats apply). Any data that was committed will be available as long as more than half the servers are up at any given point. The master always makes sure that when writing an entry on another server, the entry immediately before the one written matches the one in its logs. By induction, this means the follower’s log is complete up to the newly written entry.

If any of the servers except the master fails, nothing interesting happens. The master will just no longer be able to talk to it, it will no longer write data, but updates will be resent in case it comes back. Simple failover on master failure is equally straightforward. The master dies, the remaining servers elect a new master that has the same state as the previous master (because, remember that the updates are replicated to more than half of the servers). This new master takes the place of the old one, and nothing particularly interesting happens either.

Of course, no one would care if the paper was simply about trivial failover. The interesting case arises, of course, due to more complicated series of master failures. But because of the simplicity of the algorithm this is, in fact, not much harder to understand than the trivial cases above. In this blog-post I have hand-waved “more up to date“ and “committed” a bit earlier, these two concepts are important to understand the general case. All log entries are identified by two integers: the term (the term increases every time the mastership changes), and a sequence number within that term. A server can only win the mastership election if either its term is higher than more than half of the others, or it is in the same term but has a longer (or equally long) log in it. This makes sure that the elected master always has a complete log of committed entries. An entry is committed only after the current master has written a log-entry in its term to ensure that more than half of the followers catch up with the master.

Find the paper here: Raft Algorithm Paper.

Anyway, I picked this paper more or less at random from the seemingly endless list, and it, as the title says, proposes two algorithms for maintaining order in a list. As is usual, the first part of the paper is dedicated to giving a general introduction into the problem, its history and the solutions that have previously been proposed to solve it. Then, they justify the first of their two algorithms which, even though it does not offer theoretical improvements (i.e. lower complexity bounds – which is impossible considering that the best solution has worst-time on all operations), has two characteristics that make it useful: their analysis is easier to follow, and their practical experiments suggest that its real-world performance is superior. In fact the theoretic performance is arguably worse: the proposed data structure has amortised time complexity, while the best known one has worst-case. Anyway, as is sometimes the case, there are cases where the theoretically worse data-structure has better real-world performance; as computers always work with finite data, asymptotic performance is sometimes an oversimplification.

The first algorithm is a solution to the Order-Maintenance Problem, which has “the objective … to maintain a total order subject to insertions, deletions and precedence queries”. It does that by assigning tags to elements such that the order of the tags corresponds to the order of the elements. Then, the analysis proceeds on the virtual trie that can be obtained using the binary expansion of the tags. Roughly the algorithm makes sure that the density (that is, the number of “occupied” tags per available tag) does not exceed a threshold for every node in the implicit trie. The analysis is a bit handwavey at times, for instance it states that “Thus, we pick a u [comment: maximum tag] that works for any value from n/2 to 2n. If at any point we have too few or too many list elements, we rebuild the data structure for the new value of n. This rebuilding introduces only a constant amortized overhead.” I can see this statement being true in analogy to how amortized analysis of dynamic arrays works (that is, always doubling the size of the array when it is full results in amortized ), but it is not immediately obvious that the statement is true, because there could be subtle differences.

An important detail of the complexity analysis is that it does not do any assumption about which tag is chosen for a newly inserted element. If there are multiple tags free between the elements and adjacent to the newly inserted one, one would assume that it is best to chose the new tag such that it is as far from both as possible. This not being represented in the theoretical analysis could be a sign that the bound is simply not tight enough (i.e., that one could achieve a better bound by considering this), but their experimental evaluation suggests otherwise. It seems to make no difference where the new element is inserted, in their own words “Consecutive inserts trigger relabels after every insert, but the number of elements relabeled is small. … average inserts have fewer triggers, and the number of elements relabeled is larger.”

The second algorithm is a solution to the File-Maintenance Problem, which is similar to the Order-Maintenance Problem but also requires that it is possible to traverse the elements in order. The solution keeps the elements in order in an array. Again, the algorithm works on an implicit tree where elements of the array are bundled into one leaf node, for some . The algorithm then makes sure that two invariants are preserved, such that there is always space to insert a new item. Firstly, adjacent intervals (on every level of the implicit tree) must only differ by a certain amount. Secondly, every interval (on every level of the implicit tree) must have sufficient space to accommodate new items. The algorithms that make sure this is the case are too sophisticated to be covered in this summary, so consult the full paper for these details.

Find the paper here.

In the introduction he argues that while people may expect him to talk about UNIX, because this is what he is most famous for, he will not because he feels that he is getting undeserved credit for the work of a multitude of people where for his contributions luck and fortunate timing played as much part as skill. He also promises to show the “cutest program” he has written, but I am not sure how that is followed up on in the paper.

It talks about self-printing programs before concerning itself with self-hosting compilers, that is, compilers for a language that are written in that same language, for instance a C compiler written in C. This reminded me a bit of assembling an IKEA desk yesterday that required a table for assembly, so you have the same chicken-and-egg problem. The solution, of course, is relatively straightforward and consists of writing an initial version of the compiler in another language (or getting an table that is not this IKEA desk – or indeed in this case not following the instructions, but that is irrelevant to a discussion about computers), and then using existing binaries of the compiler to compile subsequent versions.

The example he shows is a bit more interesting. He considers a high-level idealized section of C compiler that treats string escape sequences, basically as a sequence of ifs like

if (c == 'n') {
    return '\n';
}

This is interesting for itself, as once again this is a self-reference that expects this to be compiled by a thing that already understands what '\n' means. Now consider the case when you want to add a new escape sequence that is not available in your current compiler. Obviously the above will fail, as your compiler will complain that it does not know what to do with '\n'. However, what you can do is add the following to the compiler, creating a version that understands the meaning of '\n'.

if (c == 'n') {
    return 10; // 10 is the ASCII code for \n.
}

You can now, in any subsequent version of the compiler that is compiled by this (or a newer one), refer to '\n' instead of 10. He claims that ‘It is as close to a “learning” program as I have seen’, which I would take with a grain of salt (especially honouring the fact that this paper was written in the 70s) considering recent advances in machine learning, whereas this is more of a matter of where a particular piece of data was introduced (the compiler, the compiler’s compiler, …) than anything that has genuinely to do with learning – the fact that it should return 10 for c == 'n' is in the compiler’s binary, either way.

Then he, as is quite obvious, applies these ideas to information security. How, in face of this ridiculous matryoshka doll situation, can one be sure what a program does – in particular, that it does not do anything malicious? The only way would be to go up the chain of compilers until the root, which does not appear tractable. He suggests the alternative is trusting the people that wrote the program, thus the title of the paper (as the trust, indeed, has to be recursive – you implicitly trust the compiler that was used to compile your compiler).

He then ends with a few political comments about persecution of “hackers” and the two-facedness of considering them wiz kids and criminals at the same time, which sadly still seem current now over 40 years later.

Find the paper here.

This is actually, for computer science, a fairly old paper, written by no less than Stephen Cook, about whom you might have heard, for instance regarding the Cook(–Levin) theorem, which in fact is closely related to the content of this paper. Essentially, Cook’s theorem states that the propositional case of satisfiability is NP-complete.

For those not familiar with complexity theory (very roughly the theory for determining the relative hardness in terms of computational effort of different problems), NP-complete problems are those that are mostly believed to be hard to solve (that is, there exists no polynomial algorithm), but no proof exists for that statement. Solving – as in coming up with an algorithm of polynomial running time, one would immediately solve the whole class. This is due to the observation that is also expressed in the paper that there exist polynomial reductions between the problems (a reduction is a transformation of an instance of one problem into an instance of another, whereby one can solve the initial problem).

The first half of the paper talks about the complexity of theorem proving in the propositional case, which is essentially the same as satisfiability in classical logic (something is a tautology iff its negation is unsatisfiable)

The second half of the paper concerns itself with the complexity of theorem-proving first-order (i.e. predicate) logic and presents lesser known ideas. It is generally known that theorem proving is undecidable, that is, there is no algorithm that can proof all valid theorems of first-order logic. Thus, the definition of complexity is a bit more evolved than in the propositional case. How can one analyze an algorithm that can possibly go on forever without halting? Cook essentially proposed setting an order on the Herbrand instances (essentially instantiating every function and predicate in every possible way) to analyze the complexity relative to the highest index of the instances used in a proof (this is a bit of a handwavey explanation). He presents two bounds on this measure.

Find the paper here (actually, this is a re-typeset version, because the original one was written on a typewriter and this is, in my opinion, nicer to read).