zlacker

Google's new pipe syntax in SQL

submitted by heyden+(OP) on 2024-08-25 13:33:23 | 328 points 182 comments
[view article] [source] [go to bottom]

NOTE: showing posts with links only show all posts
2. verdve+5O9[view] [source] 2024-08-28 21:58:53
>>heyden+(OP)
The research paper: https://storage.googleapis.com/gweb-research2023-media/pubto...
3. summer+hO9[view] [source] 2024-08-28 22:00:30
>>heyden+(OP)
Previous submissions on the paper itself:

>>41321876 (first) >>41338877 (plenty of discussions)

I tried this new syntax and this seems a reasonable proposal for complex analytical queries. This new syntax probably does not change most simple transactional queries though. The syntax matches the execution semantic more closely, which means you less likely need to formulate query in a weird form to make query planner work as expected; usually users only need to move some pipe operators to more appropriate places.

4. samwil+XP9[view] [source] 2024-08-28 22:13:13
>>heyden+(OP)
Richard Hipp, creator of SQLite, has implemented this in an experimental branch: https://sqlite.org/forum/forumpost/5f218012b6e1a9db

Worth reading the thread, there are some good insights. It looks like he will be waiting on Postgres to take the initiative on implementing this before it makes it into a release.

◧◩
5. scrlk+2Q9[view] [source] [discussion] 2024-08-28 22:13:51
>>summer+hO9
There was a second submission of the paper, which attracted more comments: >>41338877
◧◩
7. simonw+bS9[view] [source] [discussion] 2024-08-28 22:29:29
>>urband+ZL9
A surprising problem I'm seeing with maintaining a link blog is that articles from it occasionally get submitted to Hacker News, where people inevitably call them out as not being as appropriate as the source they are linking to - which is fair enough! That's why I don't tend to submit them myself.

This particular post quickly turned into a very thinly veiled excuse for me to complain about PDFs, then demonstrate a Gemini Pro trick.

In this case I converted to HTML - I've since tried converting a paper to Markdown and sharing in a Gist, which I think worked even better: https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdab... - notes here https://simonwillison.net/2024/Aug/27/distro/

10. yarg+lT9[view] [source] 2024-08-28 22:40:15
>>heyden+(OP)
This reminds me .NET's short lived Linq to SQL;

There was a talk at the time, but I can't find the video: http://jaoo.dk/aarhus2007/presentation/Using+LINQ+to+SQL+to+....

Basically, it was a way to cleanly plug SQL queries into C# code.

It used this sort of ordering (where the constraints come after the thing being constrained); it needed to do so for IntelliSense to work.

13. tehlik+BU9[view] [source] 2024-08-28 22:50:46
>>heyden+(OP)
LINQ, PRQL, Kusto has all preceeded this.

While LINQ is mostly restricted to .NET, PRQL is not. https://prql-lang.org/

It's a welcome change in the industry.

I made this prediction a couple years back: https://x.com/tehlike/status/1517533067497201666

◧◩◪
21. yarg+YW9[view] [source] [discussion] 2024-08-28 23:14:47
>>dragon+eU9
It was in 3.5 only.

If they've replaced it with something else in the last decade and a half that does not mean that they didn't get rid of it, or that it wasn't short lived.

https://learn.microsoft.com/en-us/dotnet/framework/data/adon...

24. AdieuT+wZ9[view] [source] 2024-08-28 23:39:43
>>heyden+(OP)
If anyone is interested in the theoretical background to the thrush combinator, a.k.a. "|>", here is one using Ruby as the implementation language:

https://leanpub.com/combinators/read#leanpub-auto-the-thrush

Being a concept which transcends programming languages, a search for "thrush combinator" will yield examples in several languages.

31. chubot+y2a[view] [source] 2024-08-29 00:09:39
>>heyden+(OP)
The next thing I would like is to define a function / macro that has a bunch of |> terms.

I pointed out that you can do this with shell:

Pipelines Support Vectorized, Point-Free, and Imperative Style https://www.oilshell.org/blog/2017/01/15.html

e.g.

    hist() {
      sort | uniq -c | sort -n -r
    }

    $ { echo a; echo bb; echo a; } | hist
      1 bb
      2 a

    $ foo | hist
    ...
   
Something like that should be possible in SQL!
36. theneg+h9a[view] [source] 2024-08-29 01:08:28
>>heyden+(OP)
Now wondering if there is any relation to "Structural versus Pipeline Composition of Higher-Order Functions (Experience Report)":

https://cs.brown.edu/~sk/Publications/Papers/Published/rk-st...

◧◩
39. zX41Zd+Jca[view] [source] [discussion] 2024-08-29 01:43:21
>>themer+u6a
In ClickHouse you can do

    INSERT INTO table FORMAT JSONEachRow {"key": 123}
It works with all other formats as well.

Plus, it is designed in a way so you can make an INSERT query and stream the data, e.g.:

    clickhouse-client --query "INSERT INTO table FORMAT Protobuf" < data.protobuf

    curl 'https://example.com/?query=INSERT...' --data-binary @- < data.bson
◧◩
41. wslh+fda[view] [source] [discussion] 2024-08-29 01:50:38
>>AdieuT+wZ9
I find this [1] from this [2]. Seems like a good explanation. It doesn't exist on Wikipedia though.

[1] https://github.com/raganwald-deprecated/homoiconic/blob/mast...

[2] https://stackoverflow.com/a/285973/88231

◧◩◪
42. llimll+nda[view] [source] [discussion] 2024-08-29 01:53:01
>>simonw+bS9
Have you seen gist.io?

If you replace `gist.github.com/<user>/<id>` -> `https://gist.io/@<user>/<id>`, you get a gist with nice typography.

https://gist.io/@simonw/46a33d66e069efe5c10b63625fdabb4e is the same gist you linked, but nicer to read

◧◩
47. neonsu+7ha[view] [source] [discussion] 2024-08-29 02:33:31
>>yarg+lT9
There is https://github.com/linq2db/linq2db which is LINQ to SQL reincarnated.

Of course there's EF Core too.

◧◩◪
56. AdieuT+Ala[view] [source] [discussion] 2024-08-29 03:23:21
>>wslh+fda
A key thing to keep in mind is that the thrush combinator is a fancy name for a simple construct. The semantics it provides is a declarative form of traditional function composition.

For example, given the expression:

  f (g (h (x)))
The same can be expressed in languages which support the "|>" infix operator as:

  h (x) |> g |> f
There are other, equivalent, constructs such as the Cats Arrow[0] type class available in Scala, the same Arrow[1] concept available in Haskell, and the `andThen` method commonly available in many modern programming languages.

0 - https://typelevel.org/cats/typeclasses/arrow.html

1 - https://wiki.haskell.org/Arrow_tutorial

◧◩
64. simonw+vqa[view] [source] [discussion] 2024-08-29 04:18:02
>>make3+1pa
See my comment here: >>41385143
◧◩
69. mixedC+bua[view] [source] [discussion] 2024-08-29 05:00:24
>>BeefWe+oqa
https://prql-lang.org/ has a bunch of good examples on its home page.

If you engage the syntax with your System 2 thinking (prefrontal cortex, slow, the part of thinking we're naturally lazy to engage) rather than System 1 (automated, instinctual, optimized brain path to things we're used to) you'll most likely find that it is simpler, makes more logical sense so that you're filtering down things naturally like a sieve and composes far better than SQL as complexity grows.

After you've internalized that, imagine the kind of developer tooling we can build on top of that logical structure.

73. dang+tva[view] [source] 2024-08-29 05:14:48
>>heyden+(OP)
Recent and related:

Pipe Syntax in SQL - >>41338877 - Aug 2024 (219 comments)

◧◩◪
79. pradee+Nxa[view] [source] [discussion] 2024-08-29 05:43:43
>>Blackt+FS9
This syntax looks a lot like PRQL. ClickHouse supports writing queries in PRQL dialect. Moreover, ClickHouse also supports Kusto dialect too.

https://clickhouse.com/docs/en/guides/developer/alternative-...

84. mav3ri+Yza[view] [source] 2024-08-29 06:13:08
>>heyden+(OP)
The first piped query language I used was Nushell's implementation of wide-column tables. PRQL offers almost similar approach which I have loved dearly. It also maps to different SQL dialects. There is also proposal to work on type system: https://github.com/PRQL/prql/issues/381.

Google has now proposed a syntax inspired by these approaches. However, I am afraid how well it would be adopted. As someone new to SQL, nearly every DB seem to provide its own SQL dialect which becomes cumbersome very quickly.

Whereas PRQL feels something like Apache Arrow which can map to other dialects.

94. alouki+pCa[view] [source] 2024-08-29 06:44:44
>>heyden+(OP)
This like Elixir's pipe operator [1]! I use it on the daily (with Ecto) and it's epic!

[1] https://elixirschool.com/en/lessons/basics/pipe_operator

98. victor+1Da[view] [source] 2024-08-29 06:51:56
>>heyden+(OP)
Looks just like writing sql using Ecto in Elixir:

"users" |> where([u], u.age > 18) |> select([u], u.name)

https://hexdocs.pm/ecto/Ecto.Query.html

104. gopian+EEa[view] [source] 2024-08-29 07:09:16
>>heyden+(OP)
I find this particular choice of syntax somewhat amusing because the pipe notation based query construction was something I ended up using a year ago when making an SQL library in OCaml:

https://github.com/kiranandcode/petrol

An example query being:

```

let insert_person ~name:n ~age:a db = Query.insert ~table:example_table ~values:Expr.[ name := s n; age := i a ] |> Request.make_zero |> Petrol.exec db

```

105. julien+9Fa[view] [source] 2024-08-29 07:15:33
>>heyden+(OP)
I haven't seen it mentioned yet, but it reminds me of PQL (not PRQL): https://pql.dev

It's inspired by Kusto and available as an open-source CLI. I've made it compatible with SQLite in one of my tools, and it's refreshing to use.

An example:

  StormEvents
  | where State startswith "W"
  | summarize Count=count() by State
◧◩◪◨⬒⬓⬔
131. foldr+PUa[view] [source] [discussion] 2024-08-29 10:09:29
>>maxbon+vSa
Yes, it's an important point that SQLite is not a project with an open contribution model. However, they do presumably accept external contributions in the form of bug reports, suggested patches, etc. etc.

You didn't have to dig through the repository to find the CoC. It was right there on the website at /codeofconduct.html: https://web.archive.org/web/20180315125217/https://www.sqlit...

◧◩
153. theodp+0ib[view] [source] [discussion] 2024-08-29 13:27:21
>>ahmed_+SPa
Yes, it's very convenient to be able to use SQL with your massively parallel commercial database (Oracle, Snowflake, etc.) and then again with the results sets (Pandas, etc.). Interestingly, it's a concept that was implemented 35 years ago in SAS (link below) but is just now gaining traction in today's "modern" software (e.g., via DuckDB).

USING THE NEW SQL PROCEDURE IN SAS PROGRAMS (1989) https://support.sas.com/resources/papers/proceedings-archive... The Sql procedure uses SQL to create, modify, and retrieve data from SAS data sets and views derived from those data sets. You can also use the SOL procedure to join data sets and views with those from other database management systems through the SAS/ACCESS software interfaces.

◧◩◪
157. WorldM+3qb[view] [source] [discussion] 2024-08-29 14:18:40
>>neonsu+7ha
And NHibernate.Linq and Dapper.Extensions.Linq… Most ORMs in the ecosystem have at least one Linq support library, even if just a third-party extension.

Also, there are fun things that support Linq syntax for non-ORM uses, too, such as System.Reactive.Linq and LanguageExt: https://github.com/louthy/language-ext/wiki/How-to-deal-with...

◧◩
159. WorldM+Drb[view] [source] [discussion] 2024-08-29 14:29:37
>>isopro+FBa
It's like other "arrow" digraphs in common programming languages today, such as =>. You can picture it as a triangle pointing to the right.

Many Programming Ligature fonts even often draw it that way. For instance it is shown under F# in the Fira Code README: https://github.com/tonsky/FiraCode

◧◩
172. yencab+80d[view] [source] [discussion] 2024-08-30 02:08:31
>>Ericso+5ia
You might enjoy https://substrait.io/
◧◩
174. jshute+Kre[view] [source] [discussion] 2024-08-30 17:19:48
>>BeefWe+oqa
For examples of larger queries, see here for all TPC-H queries in standard syntax and converted to pipe syntax: https://github.com/google/zetasql/blob/master/zetasql/exampl...

And several more examples with pipe syntax here: https://github.com/google/zetasql/blob/master/zetasql/exampl...

◧◩
175. jshute+qse[view] [source] [discussion] 2024-08-30 17:23:53
>>chubot+y2a
It is, using table-valued functions (TVFs).

There's an example at the bottom of this file:

https://github.com/google/zetasql/blob/master/zetasql/exampl...

[go to top]