performance tracking issue

[kleinschmidt]

This is a meta-issue to keep track of major performance problems (broadly construed) that may require API changes and hence are blocking a 1.0 release.

• type parameters on all term types and tuple as multi-term container make the compiler work harder than necessary (Speed creating formula #164, Performances 0.5 vs 0.6 #141)
• relatedly, a new anonymous function is created for every function term (Speed creating formula #164), fixed by FunctionTerm is dead, long live FunctionTerm #183
• also related: fit is very slow for new formulas #220 (probably due to specializing on data column names in NamedTuples)
• First-class support for sparse model matrix (dense always created first)
• Lots of allocations in modelcols which aren't really necessary (can be done in-place)
• First-class support for streaming data, for schema extraction and model matrix row-wise methods (would be mostly fixed by in-place modelcols)
• Lack of systematic performance benchmarking (https://github.com/matthewwardrop/formulaic suite might be a good place to start)

[palday]

I'll need to actually look at the current sparse code, but it's not too hard to a more proper sparse constructor that allocates a bunch of intermediate sparse matrices but doesn't do a dense allocation. So not ideal, but better than now and straightforward. (I have a hack for a current project that does exactly this.) This could be optimized in different ways for column vs. row tables -- column tables are especially easy for constructing CSC matrices.

[kleinschmidt]

I also suspect that switching everything under the hood to use in-place modelcols would also go a long way towards making it easy to support sparse model matrix