zlacker

>>mafaa+(OP)
As a lone data wrangler, I am dreaming of an "APL like R", i.e. geared specifically towards data manipulation and stats with an integrated columnar store (I am spoilt by J).

People always say that such a thing will never fly in teams due to syntactic issues, but APL really is a productivity secret weapon for loners and small teams!

>>rscho+sO
Is it a matter of writing a comprehensive stats library?

>>chrisp+mT
A comprehensive stat library for J would be close (there is an RServe lib), but the most important thing is the integrated data store. For big companies, it makes perfect sense to want a separate data store. But for small shops or loners, tight language integration is light-years better! With J, I am allowed a real database (no effing .csv) where I can use the same (terse) language as for the analysis. This is the killer feature. This is where you see that say, for example Julia, is made for serious industrial coding with teams of tens of people and not lone guys.

Following the same principle, R dplyr allows you to seamlessly interact with the DB by using a translation layer. However, every time I open R I find myself having to write tens if not hundreds of loc to shape the data where I'd do that in a few lines of J. For single researchers, it's actually much easier to read your one-page of J code 6 months later than it is for your 500-loc R script (again IMO).

Although I imagine it could be possible to really make a specialized APL geared towards data analysis as a strict DSL (APL is not a DSL). Meaning for example, making it more static and therefore statically compileable at the expense of losing things such as first-class environments (namespaces) or the "execute" primitive. One could also specialize the notation further towards statistics. There really is a whole realm of possibilities, here!

In a word, there is a market for lone scientists. It would be nice to have tools for that market ;-)

>>rscho+eV
You didn't mention it, so for avoidance of doubt: have you heard of k?

Commercial k variants come with a columnar data store (see Kx's q/kdb+, Shakti's k9).