Sheaves

Tensor associated to a database

Sun, 16 Mar 2025 00:00:00 +0000

In this post (the 2nd in this series on Tensors and Language Models), we will be interested in collections of tables or dataframes that look something like:

We will call these databases. This example database comes from the paper on quantifying the knowledge capacity of attention layers.

By the end of this post, we will see how to define a tensor associated to a database. But first, we will begin by defining a matrix associated to a function $f: X \to Y$.

Matrix representation of a function

Suppose we have a set $X = {a,b,c}$, another set $Y = {m,s}$ and a function $f \colon X \to Y$ given by

\[\begin{align} f \colon X &\to Y \\ a &\mapsto s \\ b &\mapsto s \\ c & \mapsto m \end{align}\]

Functions are columns in databases. For example, $f$ above is the column born_in if we replace $a$ with Astrid, $b$ with Bernard, and so on. For such a function, we may define an $|X| \times |Y|$ matrix, $M_f$, with entries given by:

\[(M_f)_{xy} = \begin{cases} 1 \mbox{ if } f(x) = y,\\ 0 \mbox{ otherwise}. \end{cases}\]

For the function $f$ above, we would have

\[M_f = \begin{array}{c@{}c} & \begin{array}{cc} m & s \end{array} \\ \begin{array}{c} a \\ b \\ c \end{array} & \left( \begin{array}{cc} 0 & 1 \\ 0 & 1 \\ 1 & 0 \end{array} \right) \end{array}\]

The matrix $M_f$ represents the function $f$ in the following sense: let elements of $X$ be represented as vectors $e_a = (1,0,0),\, e_b = (0,1,0),\, e_c=(0,0,1)$ in $\mathbb{R}^{|X|}$, and elements of $Y$ as vectors $e_m = (1,0),\, e_s = (0,1)$ in $\mathbb{R}^{|Y|}$ (i.e. one-hot encoding), then we have

\[e_x \, M_f = e_{f(x)}.\]

For example, one can verify that $e_a\, M_f = e_s = e_{f(a)}$. Thus $M_f$ stores the values of $f$, and can be used to compute $f$.

The rank of $M_f$

Since $M_f$ is a matrix, we can compute its rank. There are a few equivalent definitions for the rank of a matrix, but we will use the one that generalizes most easily to tensors, since tensors are where we’re headed next.

In order to state this definition, we first define a rank-1 matrix to be a non-zero matrix that can be expressed as a matrix product $c \cdot r$ of a column vector $c$ and a row vector $r$. The following example shows a rank-1 matrix and its decomposition into $c$ and $r$:

\[\left( \begin{array}{cc} 0 & 0 \\ 0 & 0 \\ 1 & 0 \end{array} \right) = \left( \begin{array}{c} 0 \\ 0 \\ 1 \end{array} \right) \left( \begin{array}{cc} 1 & 0 \end{array} \right)\]

The rank of a matrix $M$ is then the smallest $k$ such that $M$ can be expressed as a sum of $k$ rank-1 matrices.

From this, we can see that an upper bound for the rank of $M$ is simply the number of non-zero entries it has, since $M$ can be expressed as a sum of matrices that contain only 1 non-zero entry. For example, the matrix $M_f$ above can be written as the sum:

\[M_f = \left( \begin{array}{cc} 0 & 1 \\ 0 & 0 \\ 0 & 0 \end{array} \right) + \left( \begin{array}{cc} 0 & 0 \\ 0 & 1 \\ 0 & 0 \end{array} \right) + \left( \begin{array}{cc} 0 & 0 \\ 0 & 0 \\ 1 & 0 \end{array} \right)\]

and thus $\mathrm{rank}(M_f) \leq 3$.

But in fact, the rank of $M_f$ is 2, because this is also a rank 1 matrix:

\[\left( \begin{array}{cc} 0 & 1 \\ 0 & 1 \\ 0 & 0 \end{array} \right) = \left( \begin{array}{c} 1 \\ 1 \\ 0 \end{array} \right) \left( \begin{array}{cc} 0 & 1 \end{array} \right)\]

and we can decompose $M_f$ as

\[M_f = \left( \begin{array}{cc} 0 & 1 \\ 0 & 1 \\ 0 & 0 \end{array} \right) + \left( \begin{array}{cc} 0 & 0 \\ 0 & 0 \\ 1 & 0 \end{array} \right).\]

In fact, for any function $f : X \to Y$ between finite sets, we can always decompose $M_f$ into a sum of $k$ rank-1 matrices, one for each element in the range of $f$, so that we have:

\[\mathrm{rank}(M_f) = |\mathrm{range}(f)|.\]

We thus see that the rank of $M_f$ gives us some notion of the size of $f$.

Why matrices? Why rank?

But why go through all that trouble? Why not just represent $f$ as a dictionary or hashmap, such as f = {'a': 's', 'b':'s', 'c':'m'}?

The reason is that linear algebra is the language of neural networks and multi-layer perceptrons (MLPs). Indeed, a rank-1 matrix $M$ can be represented as an MLP with 1 hidden neuron, where the activation functions are all just the identity function:

If $c \cdot r$ is the decomposition of $M$, the weights on the left are the entries of $c$, while the weights on the right are the entries of $r$.

A rank-$k$ matrix is then an MLP with $k$ hidden neurons:

So the rank of $M_f$ tells us how many hidden neurons are needed to represent the function $f$ when using a linear network (i.e. a network with identity activation functions).

Of course, “real” MLPs usually have non-linear activation functions (such as ReLu or various sigmoids). As this paper shows, such networks also store functions (or “key-value memories”). If we record which outputs we get when we pass inputs of $f$ into the network, we should expect to get a matrix that is similar to $M_f$. In this setting, the number of hidden neurons continues to be a useful measure of the size or capacity of a network, even though it may no longer be the same as the rank of $M_f$ due to non-linearity.

From functions to databases

So far, we have seen that functions (or key-value pairs, associative memories, dictionaries, hashmaps etc.) can be stored in matrices/MLPs. But functions or key-value pairs are not yet facts. Here’s what I mean: consider the key-value pair (Michael Jordan, Basketball), which is sometimes given as an example of a “fact”. Indeed, Michael Jordan is most strongly associated to basketball, and if you asked someone to imagine “Michael Jordan”, chances are they would imagine him playing basketball or in a basketball jersey.

But what is happening here is that we are implicitly filling in a relation between Michael Jordan and basketball, such as played (for most of his career), or is most often associated with. And the facticity of the pair depends on this relation. If we change the relation to something like played in 1994, then the correct fact should mention baseball instead of basketball.

To put it another way, a function or a set of key-value pairs is only one column in a database, and a fact consists not only of a key-value pair, but also the name of the column, which indicates the relationship between the key and the value. Going back to the example database at the top of this post, the pair (Astrid, Singapore) is an incomplete fact. The full fact is (Astrid, born_in, Singapore).

Since a database is a collection of functions (one for each column in the database), and each function can be represented as a matrix, it stands to reason that a database can be represented as a collection of matrices, or a tensor.

Tensor representation of a database

A tensor is a higher dimensional analogue of a matrix. Just as a matrix (or 2-tensor) can be thought of as a collection of vectors (or 1-tensors), a 3-tensor is a collection of matrices.

We’ve already seen what the matrix for the born_in column looks like. Here’s the matrix for the lives_in column:

\[\begin{array}{c@{}c} & \begin{array}{cc} m & s \end{array} \\ \begin{array}{c} a \\ b \\ c \end{array} & \left( \begin{array}{cc} 1 & 0 \\ 0 & 1 \\ 1 & 0 \end{array} \right) \end{array}\]

We get a 3-tensor by stacking these two matrices (think of them as lying on different pages of a 2-page book).

Meanwhile, the matrix representation of the currency column looks like this:

\[\begin{array}{c@{}c} & \begin{array}{cc} r & d \end{array} \\ \begin{array}{c} m \\ s \end{array} & \left( \begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array} \right) \end{array}\]

To stack this with the other two matrices, we need to pad them with zeros so that their shapes agree:

\[M_\beta = \begin{array}{cc} & \begin{array}{cccc} m & s & r & d \end{array} \\ \begin{array}{c} a \\ b \\ c \\ m \\ s \end{array} & \left( \begin{array}{cccc} 0 & 1 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{array} \right) \end{array}\] \[M_\lambda = \begin{array}{cc} & \begin{array}{cccc} m & s & r & d \end{array} \\ \begin{array}{c} a \\ b \\ c \\ m \\ s \end{array} & \left( \begin{array}{cccc} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{array} \right) \end{array}\] \[M_\chi = \begin{array}{cc} & \begin{array}{cccc} m & s & r & d \end{array} \\ \begin{array}{c} a \\ b \\ c \\ m \\ s \end{array} & \left( \begin{array}{cccc} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array} \right) \end{array}\]

Here $\beta$ is the column born_in, $\lambda$ is the column lives_in and $\chi$ is the column currency. Stacking these 3 matrices together gives us the $3 \times 5 \times 4$ tensor associated to this database.

For the general definition, it helps to think of a database as a collection of RDF triples. Although RDF triples are usually in (subject, predicate, object) or (key, query, value) format, for the purpose of this post, it will help to think of them in (predicate, subject, object) or (query, key, value) format instead. For example, our database above could be represented as the following set of triples:

\[\begin{array}{ccc} born\_in & Astrid & Singapore \\ born\_in & Bernard & Singapore \\ born\_in & Colin & Malaysia \\ lives\_in & Astrid & Malaysia \\ lives\_in & Bernard & Singapore \\ lives\_in & Colin & Malaysia \\ currency & Malaysia & Ringgit \\ currency & Singapore & Dollar \end{array}\]

If we let $\mathcal{D}$ be this list of triples, we may define a 3-tensor $D$ of the appropriate size whose entries are:

\[D_{qkv} = \begin{cases} 1 \mbox{ if } (q,k,v) \in \mathcal{D}, \mbox{ and}\\ 0 \mbox{ otherwise.} \end{cases}\]

Just as the matrix $M_f$ contains all the information of the function $f$, the tensor $D$ contains all the information of the database $\mathcal{D}$.

Technical note: In the paper ‘Paying attention to facts’, the triples are in (key,query,value) format and the entries of $D$ are indexed by $k,q,v$ instead of $q,k,v$.

Summary and what’s next

In this post, we have seen how to represent a function as a matrix, and a database as a 3-tensor. We have also touched upon the relationship between functions and MLPs. Note that a database cannot be represented as a single MLP, because MLPs only store key-value pairs! But as we will eventually see, we can encode databases in the attention heads of transformers. (see the ‘Paying attention to facts’ paper to skip ahead)

While we have defined and computed the rank of $M_f$, we haven’t talked about the rank of tensors. That will be the subject of the next post. We will also see that tensor rank has some unintuitive properties (at least for those who are more familiar with matrix rank).

Getting involved

If this line of work interests you, please reach out in the comments below, or email me at liangze.wong@gmail.com. I would be happy to collaborate on topics related to interpretability and theory of language models, even if it does not involve tensors or factual recall.

If you are hiring (or know of anyone who might be interested in hiring) research scientists with my background, I am currently open to work, and am open to re-location!

Tensors and Factual Recall in Language Models

Thu, 13 Mar 2025 00:00:00 +0000

This is the first in a series of posts explaining and expanding on the ideas introduced in the following papers:

Paying attention to facts: quantifying the knowledge capacity of attention layers
[arXiv:2502.05076]
‘Generalization is hallucination’ through the lens of tensor completions
[arXiv:2502.17305]

The overarching theme behind these papers is that tensors and tensor completions are a useful theoretical framework for thinking about language models, specifically generative models based on transformers. As I’ve only started doing this in 2025, everything is still quite preliminary. In particular, the results of these papers only hold for very simple toy models and synthetic datasets.

Nevertheless, this line of inquiry has already yielded some interesting findings. For example, the ‘Paying attention to the facts’ paper shows that transformers can store facts in their attention heads and not just their MLP layers. It also shows that we can increase model capacity without increasing the number of parameters by have larger output-value weights and smaller key-query weights. Meanwhile, the ‘Generalization is hallucination’ paper identifies a common mechanism that shows how training data gives rise to certain types of generalizations and hallucinations.

In this series of blog posts, I plan to explain and expand on some of the ideas in these papers, in the hopes of encouraging more research in this direction. In the next few posts, I will introduce tensors and their rank, relate them to language datasets and language models, talk about databases and ways of generating `random databases’, and highlight the non-linear effects of argmax and softmax on rank.

But this post will be less technical. These papers may broadly be described as using tensors to understand factual recall in language models, and so in this post I hope to explain some of the motivations behind these papers by answering two questions, “Why facts?” and “Why tensors?”

Why facts?

Factual recall in language models has some nice properties as a research topic. Firstly, facts are simple. At their core, facts can be represented as subject-predicate-object (or subject-relation-object) triples, such as (France, capital_is, Paris), although they may be represented in other ways in natural language.

Not only are facts easy to represent, it is usually the case that there is one right answer: given the pair (France, capital_is), you would expect a model to output Paris. This turns factual recall into a supervised task, and removes the complication of having multiple acceptable outputs, which are common in many other next-token prediction tasks. On a related note, databases of facts (be it in traditional tabular databases, or graph databases, or RDF triples) are readily available as supervised datasets, and are also easy to synthetically generate.

In addition to the nature of facts themselves, it seems to be generally accepted that LLMs can store facts, and that this is a desirable behaviour. It is thus important to study factual recall, because a better understanding will enable us to better control this behaviour (to increase the storage capacity, or reduce factual hallucinations, for example).

I think it is in part because of these reasons that the question of how LLMs memorize facts has received a fair amount of attention, as these examples demonstrate:

Despite the simplicity of facts, however, and despite the research that has gone into it, we still do not have a full picture of how facts are stored in language models. For example, papers about scaling laws of memorization capacity tend to quantify the size of a database of facts in terms of the number of facts contained. However, this ignores the structure of facts and patterns among facts that might make one database easier or harder to memorize than another database with the same number of facts.

My sense is that the number of facts in a database is just a proxy for another measure of database size that better reflects the underlying properties of the database. While I have not uncovered this measure yet, I think my approach (in terms of tensors) has brought us a step closer to it.

Which leads us to the next question…

Why tensors?

When I first started this line of investigation, I did not have tensors in mind at all. I was simply trying to find a way to think about facts represented as RDF triples. I wanted a setting that allowed me to talk about an RDF triple being “in the span” of other triples in a precise sense. Since each RDF triple has 3 parts (subject, predicate and object), it seemed natural to use a 3-tensor rather than a matrix to encode this information. From there, it was a natural extension to $n$-tensors for more general $n$-grams.

I do not think that tensors are the way to understand language models, merely one way. Each framework (be it embeddings, or neurons, or dictionary learning, or …) sheds light on language models from a different angle, and all I hope to do is to introduce another angle to the story. I think a lot can be gained by also trying to understand the interplay between different frameworks. For example, can we relate linear-algebraic properties of a tensor to geometric properties of embeddings that result from decomposing the tensor? Can we relate linear-algebraic properties of tensors to neuron activations?

Up next

I’ve talked a lot about tensors without really defining them. In the next post, I will define tensors and relate them to facts represented as RDF triples.

Getting involved

If you are hiring (or know of anyone who might be interested in hiring) research scientists with my background, I am currently open to work, and am open to re-location!

Distributive Laws

Sat, 18 Feb 2017 00:00:00 +0000

I’ve been participating in the Kan Extension Seminar II, and this week it’s my turn to post about Jon Beck’s “Distributive Laws” at the n-Category Cafe!

The post uses lots of string diagrams for monads, resulting in pictures like the following:

See you there!

Noncommutative Algebras in Sage

Thu, 03 Mar 2016 00:00:00 +0000

In this post, I’ll demonstrate 3 ways to define non-commutative rings in Sage. They’re essentially different ways of expressing the non-commutative relations in the ring:

Via g_algebra: define the relations directly
Via NCPolynomialRing_plural: define a pair of structural matrices
Via a quotient of a letterplace ring: define the ideal generated by the relations (only works for homogeneous relations)

As far as I know, all 3 methods rely on Sage’s interface with Singular and its non-commutative extension Plural.

In addition to all the documentation linked above, I also relied heavily on Greuel and Pfister’s A Singular Introduction to Commutative Algebra. Despite the title, it does have a pretty substantial section (1.9) devoted to non-commutative $G$-algebras.

$U(\mathfrak{sl}_2)$ and its homogenization

The running example throughout this post will be the universal enveloping algebra $U(\mathfrak{sl}_2)$ over $\mathbb{Q}$.

We’ll define this to be the (non-commutative) $\mathbb{Q}$-algebra $U$ with generators $e,f,h$ subject to the relations

\[[e,f] = h, \qquad [h,e] = 2e, \qquad [h,f] = -2f.\]

If we set $e,f,h$ to have degree 1, these relations are not homogeneous. Their left-hand sides only have degree 2 terms, while their right-hand sides have degree 1 terms as well. This is fine with the first two methods, but won’t work for method 3 (which requires homogeneous relations).

To demonstrate the third method, we’ll define the $\mathbb{Q}$-algebra $H$ with generators $e,f,h,t$ subject to the homogeneous relations

\[[t,e] = [t,f] = [t,h] = 0,\] \[[e,f] = ht, \qquad [h,e] = 2et, \qquad [h,f] = -2ft.\]

We can obtain $U$ both as a quotient and a localization of $H$:

\[H/(1-t) \;\;\cong\;\; U \;\;\cong\;\; H[t^{-1}]_0.\]

$G$-algebras

Using the g_algebra method of Sage’s FreeAlgebra class, we can simply plug our noncommutative relations in, and get our non-commutative ring. This is about as easy as it gets:

Let’s unravel what’s going on here.

Monomial orderings and PBW basis

Most algorithms for commutative and non-commutative rings require an ordering on the generators. In our case, let’s use the ordering

\[e \leq f \leq h.\]

This is implicitly stated in our code: we wrote F.<e,f,h> instead of F.<h,e,f>, for example.

A standard word is a monomial of the form

\[e^if^jh^k, \qquad i,j,k \in \mathbb{N}\]

In the polynomial ring $\mathbb{Q}[e,f,h]$, every monomial can be expressed in this form, so the set of standard words forms a $\mathbb{Q}$-basis for $\mathbb{Q}[e,f,h]$.

In a non-commutative ring, whether or not the standard words form a basis depends on what relations we have. Such a basis, if it exists, is called a PBW basis.

The free algebra $F = \mathbb{Q}\langle e,f,h\rangle$ has no relations, so does not have a PBW basis. Fortunately, our algebra $U$ does have a PBW basis.

This means that we can always express a non-standard monomial (e.g. $fe$) as a sum of standard monomials (e.g. $ef - h$). The non-commutative relations that define $U$ can thus be thought of as an algorithm for turning non-standard words into sums of standard words.

To do this in Sage, we define a dictionary whose keys are non-standard words and values are the standard words they become.

In the above example, our dictionary was short enough to fit into one line, but we could also define a dictionary separately and pass it into g_algebra:

It’s very important that the keys are non-standard words and the values are sums of standard words. Mathematically, the relation $fe = ef - h$ is the same as $ef = fe + h$, but if we replace f*e : e*f - h with e*f : f*e + h in the code, we’ll get an error (try it!).

What are $G$-algebras?

The reason why $U$ has a PBW basis is because it is a $G$-algebra. Briefly, $G$-algebras are algebras whose relations satisfy certain non-degeneracy conditions that make the algebra nice to work with.

For a full definition of $G$-algebras, refer to A Singular Introduction to Commutative Algebra or the Plural manual.

If $A$ is a $G$-algebra, then it has a PBW basis, is left and right Noetherian, and is an integral domain. More importantly (for this site at least!), it means that we can define $A$ in Singular/Plural, and hence in Sage.

Structural matrices for a $G$-algebra

Another way of writing our non-commutative relations is

\[\begin{pmatrix} 0 & fe & he \\ 0 & 0 & hf \\ 0 & 0 & 0 \end{pmatrix} = \begin{pmatrix} 0 & 1 & 1 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix} * \begin{pmatrix} 0 & ef & eh \\ 0 & 0 & fh \\ 0 & 0 & 0 \end{pmatrix} + \begin{pmatrix} 0 & -h & 2e \\ 0 & 0 & -2f \\ 0 & 0 & 0 \end{pmatrix},\]

where $ * $ denotes element-wise multiplication (so there isn’t any linear algebra going on here; we’re just using matrices to organize the information). Let $N,C,S,D$ be the matrices above, in that order, so that $N = C*S + D$.

If we let $x_1 = e, x_2 = f, x_3 = h$ (so that $x_i \leq x_j$ if $i \leq j$) then for $i < j$

\[n_{ij} = x_j x_i, \qquad s_{ij} = x_i x_j.\]

In other words, $N$ contains the non-standard words that we’re trying to express in terms of the standard words in $S$.

The matrices $C$ and $D$ are called the structural matrices of the $G$-algebra, and their entries are such that our relations may be written

\[x_jx_i = c_{ij} x_i x_j + d_{ij}, \qquad i < j\]

with zeros everywhere else ($i \geq j$). If $C = D = 0$, the resulting algebra will be commutative.

We can use the structural matrices $C$ and $D$ to define our algebra via Sage’s NCPolynomialRing_plural function (note that Python uses zero-indexing for matrices):

Note that R is a commutative polynomial ring. In fact, up till the point where we call NCPolynomialRing_plural, even the variables e,f,h are treated as commutative variables.

This method of defining $U$ is considerably longer and more prone to mistakes than using g_algebra. As stated in the documentation, this is not intended for use! I’m including it here because this is essentially how one would go about defining a $G$-algebra in Singular. In fact, the Sage method g_algebra calls NCPolynomialRing_plural, which in turn calls Singular.

Quotients of letterplace rings

Our final method for defining non-commutative rings makes use of Sage’s implementation of Singular’s letterplace rings.

As mentioned at the start of this post, this method requires the relations to be homogeneous, so we’ll work with $H$ instead of $U$.

Let $\mathbb{Q}\langle e,f,h,t \rangle$ be the free algebra on 4 variables. Consider the two-sided ideal $I$ generated by the relations for $H$:

\[I = (te - et, tf - ft, th - ht, ef - fe - ht, he - eh - 2et, hf - fh + 2ft)\]

Then

\[H = \mathbb{Q}\langle e,f,h,t \rangle/I.\]

This can be expressed Sage-ly:

The expression F*I*F is the two-sided ideal generated by elements in the list I.

Although $U$ cannot be defined using this method, $H$ can be defined using all three methods. As a (fun?) exercise, try defining $H$ using the other two methods.

Difficulties

These methods can be used to define many non-commutative algebras such as the Weyl algebra and various enveloping algebras of Lie algebras. One can also define these algebras over fields other than $\mathbb{Q}$, such as $\mathbb{F}_p$ (edit: but unfortunately not $\mathbb{C}$ or $\mathbb{R}$).

However, we cannot define algebras over $\mathbb{Q}(q)$, the fraction field of $\mathbb{Q}[q]$:

This is a problem if we want to define rings with relations such as

\[yx = qxy.\]

Such relations occur frequently when studying quantum groups, for example.

This is suprising, because one can easily define $\mathbb{Q}(q)$ and non-commutative $\mathbb{Q}(q)$-algebras in Singular/Plural, which is what Sage is using. It seems that the problem is in Sage’s wrapper for Singular/Plural, because Sage can’t even pass the ring $\mathbb{Q}(q)$ to Singular.

There’s a trac ticket for this problem, but until it gets resolved, we’ll just have to define such rings directly in Singular/Plural. Thanks to the amazing capabilities of the Sage Cell Server, ~~we’ll do this in the next post~~!

The Weyl Algebra and $\mathfrak{sl}_2$

Wed, 17 Feb 2016 00:00:00 +0000

I’ve been away from this blog for quite a while - almost a year, in fact! My excuses are my wedding and the prelims (a.k.a. quals), as well as all the preparation that had to go into them (although, to be honest, those things only occupied me till September last year!).

Looking back at my previous posts, I’ve realized that in attempting to teach both math and code, I probably ended up doing neither. This is really not the best place to learn representation theory (for example) - there are better books and blogs out there. Also, most of the code that I wrote to illustrate those posts feels contrived, and neither highlights Sage’s strengths nor reflects how I normally use Sage for my assignments and projects.

I’ve thus decided to write shorter posts with code that I actually use (on SageMathCloud), along with some explanations of the code. Lately, I’ve been writing code for non-commutative algebra and combinatorics, so today I’ll start with a simple example of a non-commutative algebra.

The Weyl Algebra

The $1$-dim. Weyl algebra is the (non-commutative) algebra generated by $x, \partial_x$ subject to the relations

\[x \partial_x = \partial_x x - 1.\]

If we treat $x$ as “multiplication by $x$” and $\partial_x$ as “differentiation w.r.t. $x$”, this relation is really just an application of the chain rule:

\[\partial_x (x (f(x)) = f(x) + x \partial_x f(x)\]

We can generalize to higher dimensions: the $n$-dim. Weyl algebra is the algebra generated by $x_1,\dots,x_n,\partial_{x_1},\dots,\partial_{x_n}$ quotiented by the relations that arise from treating them as the obvious operators on $\mathbb{F}[x_1,\dots,x_n]$.

Weyl algebras in Sage

It’s easy to define the Weyl algebra in Sage:

Calling inject_variables allows us to use the operators x,y,z,dx,dy,dz in subsequent code (where dx denotes $\partial_x$, etc).

One can do rather complicated computations:

By default, Sage chooses to represent monomials with x,y,z in front of dx,dy,dz:

Keep in mind that x does not refer to the polynomial $x \in \mathbb{F}[x]$, so one should not expect dx*x to be 1.

(For some reason show does not give the right output. Try show(x) or show(x*dx), for example.)

Representations of $\mathfrak{sl}_2$

It turns out that the $1$-dim. Weyl algebra gives a representation of $\mathfrak{sl}_2(\mathbb{F})$.

The Lie algebra $\mathfrak{sl}_2(\mathbb{F})$ is generated by $E,F,H$ subject to the relations

\[[H,E] = 2E, \qquad [H,F] = -2F, \qquad [E,F] = H.\]

Define the following elements of the $1$-dim. Weyl algebra:

\[E = x \partial_x^2,\qquad F = -x,\qquad H = -2x\partial_x.\]

We can use Sage to quickly verify that these elements indeed satisfy the relations for $\mathfrak{sl}_2$ (using the commutator as the Lie bracket i.e. $[A,B] = AB - BA$):

Working over $\mathbb{C}$, this action of $\mathfrak{sl}_2(\mathbb{C})$ makes $\mathbb{C}[x]$ a Verma module of highest weight $0$.

In fact, we can make $\mathbb{C}[x]$ a Verma module of highest weight $c$ for any $c \in \mathbb{C}$ by using:

\[E = (x \partial_x - c)\partial_x,\qquad F = -x,\qquad H = -2x\partial_x + c.\]

We verify this again in Sage:

In subsequent posts, I’ll talk more about defining other non-commutative algebras in Sage and Singular.

Character Theory Basics

Fri, 20 Mar 2015 00:00:00 +0000

This post illustrates some of SageMath’s character theory functionality, as well as some basic results about characters of finite groups.

Basic Definitions and Properties

Given a representation $(V,\rho)$ of a group $G$, its character is a map $ \chi: G \to \mathbb{C}$ that returns the trace of the matrices given by $\rho$:

\[\chi(g) = \text{trace}(\rho(g)).\]

A character $\chi$ is irreducible if the corresponding $(V,\rho)$ is irreducible.

Despite the simplicity of the definition, the (irreducible) characters of a group contain a surprising amount of information about the group. Some big theorems in group theory depend heavily on character theory.

Let’s calculate the character of the permutation representation of $D_4$. For each $g \in G$, we’ll display the pairs:

\[[\rho(g),\chi(g)]\]

(The Sage cells in this post are linked, so things may not work if you don’t execute them in order.)

Many of the following properties of characters can be deduced from properties of the trace:

The dimension of a character is the dimension of $V$ in $(V,\rho)$. Since $\rho(\text{Id})$ is always the identity matrix, the dimension of $\chi$ is $\chi(\text{Id})$.
Because the trace is invariant under similarity transformations, $\chi(hgh^{-1}) = \chi(g)$ for all $g,h \in G$. So characters are constant on conjugacy classes, and are thus class functions.
Let $\chi_V$ denote the character of $(V,\rho)$. Recalling the definitions of direct sums and tensor products, we see that

\[\begin{align*} \chi_{V_1 \oplus V_2} &= \chi_{V_1} + \chi_{V_2} \\ \chi_{V_1 \otimes V_2} &= \chi_{V_1} \times \chi_{V_2} \end{align*}\]

The Character Table

Let’s ignore the representation $\rho$ for now, and just look at the character $\chi$:

This is succinct, but we can make it even shorter. From point 2 above, $\chi$ is constant on conjugacy classes of $G$, so we don’t lose any information by just looking at the values of $\chi$ on each conjugacy class:

Even shorter, let’s just display the values of $\chi$:

This single row of numbers represents the character of one representation of $G$. If we knew all the irreducible representations of $G$ and their corresponding characters, we could form a table with one row for each character. This is called the character table of $G$.

Remember how we had to define our representations by hand, one by one? We don’t have to do that for characters, because SageMath has the character tables of small groups built-in:

This just goes to show how important the character of a group is. We can also access individual characters as a functions. Let’s say we want the last one:

Notice that the character we were playing with, $[4,2,0,0,0]$, is not in the table. This is because its representation $\rho$ is not irreducible. At the end of the post on decomposing representations, we saw that $\rho$ splits into two $1$-dimensional irreducible representations and one $2$-dimensional one. It’s not hard to see that the character of $\rho$ is the sum of rows 1,4 and 5 in our character table:

Just as we could decompose every representation of $G$ into a sum of irreducible representations, we can express any character as a sum of irreducible characters.

The next post discusses how to do this easily, by making use of the Schur orthogonality relations. These are really cool relations among the rows and columns of the character table. Apart from decomposing representations into irreducibles, we’ll also be able to prove that the character table is always square!

Edit: The promised “next post” about these topics never happened. Maybe sometime in the far future, I might come back to these topics, but no promises for now!

Animated GIFs

Thu, 12 Mar 2015 00:00:00 +0000

I really should be posting about character theory, but I got distracted making some aesthetic changes to this blog (new icon and favicon!) and creating animations like this:

I’m not putting this in a SageCell because this could take quite a while, especially if you increase the number of frames (by changing the parameters in srange), but feel free to try it out on your own copy of Sage. It saves an animated GIF that loops forever (iterations = 0) at the location specified by savefile.

For more information, checkout the Sage reference for animated plots.

The Group Ring and the Regular Representation

Sun, 15 Feb 2015 00:00:00 +0000

In the previous post, we saw how to decompose a given group representation into irreducibles. But we still don’t know much about the irreducible representations of a (finite) group. What do they look like? How many are there? Infinitely many?

In this post, we’ll construct the group ring of a group. Treating this as a vector space, we get the regular representation, which turns out to contain all the irreducible representations of $G$!

The group ring $FG$

Given a (finite) group $G$ and a field $F$, we can treat each element of $G$ as a basis element of a vector space over $F$. The resulting vector space generated by $g \in G$ is

\[FG := \left\{\sum_{g\in G} \alpha_g g: \alpha_g \in F \right\}.\]

Let’s do this is Sage with the group $G = D_4$ and the field $F = \mathbb{Q}$:

(The Sage cells in this post are linked, so things may not work if you don’t execute them in order.)

We can view $v \in FG$ as vector in $F^n$, where $n$ is the size of $G$ :

Here, we’re treating each $g \in G$ as a basis element of $FG$

Vectors in $FG$ are added component-wise:

\[\left(\sum_{g \in G} \alpha_g g\right) + \left(\sum_{g\in G} \beta_g g\right) = \sum_{g \in G} (\alpha_g+\beta_g) g.\]

Multiplication as a linear transformation

In fact $FG$ is also a ring (called the group ring), because we can multiply vectors using the multiplication rule of the group $G$:

\[\left(\sum_{h \in G} \alpha_h h\right) \left(\sum_{g\in G} \beta_g g\right) = \sum_{h,g \in G} (\alpha_h \beta_g) hg.\]

That wasn’t very illuminating. However, treating multiplication by $v \in FG$ as a function

\[\begin{align*} T_v: FG &\to FG \\ w &\mapsto vw, \end{align*}\]

one can check that each $T_v$ is a linear transformation! We can thus represent $T_v$ as a matrix whose columns are $T_v(g), g \in G$:

The regular representation

We’re especially interested in $T_g, g \in G$. These are invertible, with inverse $T_{g^{-1}}$, and their matrices are all permutation matrices, because multiplying by $g \in G$ simply permutes elements of $G$:

Define a function $\rho_{FG}$ which assigns to each $g\in G$ the corresponding $T_g$:

\[\begin{align*} \rho_{FG}: G &\to \mathrm{GL}(FG) \\ g &\mapsto T_g \end{align*}\]

Then $(FG,\rho_{FG})$ is the regular representation of $G$ over $F$.

The regular representation of any non-trivial group is not irreducible. In fact, it is a direct sum of all the irreducible representations of $G$! What’s more, if $(V,\rho)$ is an irreducible representation of $G$ and $\dim V = k$, then $V$ occurs $k$ times in the direct-sum decomposition of $FG$!

Let’s apply the decomposition algorithm in the previous post to $(FG,\rho_{FG})$ (this might take a while to run):

So the regular representation of $D_4$ decomposes into four (distinct) $1$-dim representations and two (isomorphic) $2$-dim ones.

Building character

We’ve spent a lot of time working directly with representations of a group. While more concrete, the actual matrix representations themselves tend to be a little clumsy, especially when the groups in question get large.

In the next few posts, I’ll switch gears to character theory, which is a simpler but more powerful way of working with group representations.

Decomposing Representations

Mon, 02 Feb 2015 00:00:00 +0000

In this post, we’ll implement an algorithm for decomposing representations that Dixon published in 1970.

As a motivating example, I’ll use the permutation matrix representation of $D_4$ that we saw in an earlier post. To make the code more generally applicable, let’s call the group $G$ and the representation $\rho$:

(The Sage cells in this post are linked, so things may not work if you don’t execute them in order.)

We’ll see that this is decomposable, and find out what its irreducible components are.

Unitary representations

A short remark before we begin: The algorithm assumes that $\rho$ is a unitary representation

i.e. for all $g \in G$,

\[\rho(g)^* \rho(g) = \rho(g) \rho(g)^* = I,\]

where $A*$ is the conjugate transpose of a matrix $A$. For $G$ a finite group, all representations can be made unitary under an appropriate change of basis, so we need not be too concerned about this. In any case, permutation representations are always unitary, so we can proceed with our example.

Finding non-scalar, commuting matrices

At the end of the previous post we saw that in order to decompose a representation $(V,\rho)$, it is enough to find a non-scalar matrix $T$ that commutes with $\rho(g)$ for every $g \in G$. This first step finds a Hermitian non-scalar $H$ that commutes with $\rho(G)$ (if there is one to be found).

Let $E_{rs}$ denote the $n \times n$ matrix with a $1$ in the $(r,s)$th entry and zeros everywhere else. Here $n$ is the dimension of $V$ in the representation $(V,\rho)$. Define

\[H_{rs} = \begin{cases} E_{rr} &\text{if } r = s \\ E_{rs} + E_{sr} &\text{if } r > s \\ i(E_{rs} - E_{sr}) &\text{if } r < s, \end{cases}\]

then the set of matrices $H_{rs}$ forms a Hermitian basis for the $n \times n$ matrices over $\mathbb{C}$.

Now for each $r,s$, compute the sum

\[H = \frac{1}{|G|} \sum_{g \in G} \,\, \rho(g)^* \, H_{rs} \, \rho(g).\]

Observe that $H$ has the following properties:

it is hermitian
it commutes with $\rho(g)$ for all $g \in G$

If $\rho$ is irreducible, then $H$ is a scalar matrix for all $r,s$. Otherwise, it turns out that there will be some $r,s$ such that $H$ is non-scalar (this is due to the fact that the $H_{rs}$ matrices form a basis of the $n \times n$ matrices$).

Let’s test this algorithm on our permutation representation of $D_4$:

We get a non-scalar $H$! So the permutation representation of $D_4$ is reducible!

Using $H$ to decompose $\rho$

Our next step is to use the eigenspaces of $H$ to decompose $\rho$. At the end of the previous post, we saw that $\rho(g)$ preserves the eigenspaces of $H$, so we need only find the eigenspaces of $H$ to decompose $\rho$.

Since $H$ is hermitian, it is diagonalizable, so its eigenvectors form a basis of $V$. In fact, the eigenbasis can be chosen to be orthonormal.

We can find this basis by computing the Jordan decomposition of $H$, and then orthonormalizing it:

It’s important that we orthonormalize $P$ so that $P$ becomes unitary. This will ensure that the subrepresentations remain unitary.

Observe that $P^{-1} \rho(g) P$ has the same block-diagonal form for each $g \in G$:

We have thus decomposed $\rho$ into two 1-dimensional representations and one 2-dimensional one!

Decomposing into irreducibles

Finally, to get a decomposition into irreducibles, we can apply the algorithm recursively on each of the subrepresentations to see if they further decompose.

Here’s a stand-alone script that decomposes a representation into its irreducible components:

Getting all irreducible representations

Now we know how to test for irreducibility and decompose reducible representations. But we still don’t know how many irreducible representations a group has.

It turns out that finite groups have finitely many irreducible representations! In the next post, we’ll construct a representation for any finite group $G$ that contains all the irreducible representations of $G$.