The correct finite field does wonders
So the computation I was running since thanksgiving was getting nowhere (finding degree 19 polynomials which are equal to 1 on x+y=1, which have fewest possible terms (11) and which have only positive coefficients, we know that there are finitely many, the trick is to find them). My code didn’t really have good status updates. Well it does tell you exactly where it is, but it takes a bit of computation to really see how it’s progressing, so I didn’t do that until a few days ago since I thought it was going way too slow.
And low and behold, it was going way too slow. I computed that it should take another 3-7 years or so (different parts of the search space take different times so it’s a bit hard to estimate). That was a bummer. That was about 10 times longer than I thought it should be. At first I was a bit resigned to this reality, but the next day I started to look into it, and one thing I figured out after running a bunch of tests was that one shortcut I was using was never triggered. The idea is that we need to find when a certain matrix with integer coefficients has 1 dimensional nullspace. Doing the integer row reduction is done with gmp, and is reasonably fast. But since we do this many times over and most of these matrices are simply full rank (no null space), we don’t really need to do the whole computation. So what we can do (and this is a useful trick to know), is to first do all computation in some small finite field, e.g. do everything mod p for some small prime p. If a matrix is full rank mod p, it is full rank. The computation can be done rather quickly this way and you don’t even have to involve modulo computation, since all the possible computations you can just precompute first and just build up a big table, so instead of two multiplications, an addition, and finding the remainder, you just look up in a table. Anyway, that gets us quite a bit of a speed up.
Now the thing is that I was using mod 19, since that worked for lower degrees. One thing I forgot when I started the run (remember this was a few years since I looked at this code and ran it last time), is that the modulus cannot be the same as the degree. The matrices we need to work with have most terms divisible by the degree. So moding out by 19 essentially always made the matrix all 0 (except for a few 1s scattered around). So these matrices were essentially always singular and the shortcut never triggered. So after doing a useless mod 19 calculation we had to do the actual integer arithmetic. That’s why it was slow. Damnit!
Well the calculations were not wrong, I just did a lot more computation than needed. After a small amount of testing it seemed that mod 23 was a good finite field to proceed in, so I restarted the code. Suddenly 3-7 years turned into first estimating 90 days and after running things for a day, that turned into an estimate of 30 days.
Then I noticed one more thing (and Danny pointed this out too), that his code used symmetry and just threw out half of the nonsymmetric polynomials, since the computations are the same. I remembered that my code didn’t do it. It didn’t make much sense if the longest run we did was 5 days on one core (for code that is only ever run once or twice, small speedups are somewhat pointless). I implemented this idea and it seems to achieve 33% reduction in time (there’s still the checking for symmetry, and there are of course symemtric polynomials, so that’s probably close to where we can get). So anyway, I guess within 20 days we should have the answer.
After it finsihes, I still have one more speedup up my sleeve. It could be that I can do the row reduction really fast mod 2 by using binary operations (each row would be an unsigned int). Not sure what speedup I can achieve though, at best 90%, since that’s how many cases mod 2 catches. While mod 23 or so catches essentially everything. So the idea is to do mod 2, then mod 23, and only then if the matrix is still singular do the integer arithmetic. If the speedup is another 50%, and my most optimistic estimates hold, that would put degree 21 within the realm of possibility, though at least half a year on 4 cores. That’s is, within something I’d be willing to run.
So, the mood went from “I’ll probably give up n d=19 soon” to “maybe d=21 is possible”. All this just by using a different prime 
Syndicated 2013-01-21 17:26:57 from The Spectre of Math
, showing that high frequency on the boundary means fast decay as you go closer to the center of your domain.
