Google's new pipe syntax in SQL
328 points by heydenberk 10 months ago | 182 comments- samwillis 10 months agoRichard Hipp, creator of SQLite, has implemented this in an experimental branch: https://sqlite.org/forum/forumpost/5f218012b6e1a9db
Worth reading the thread, there are some good insights. It looks like he will be waiting on Postgres to take the initiative on implementing this before it makes it into a release.
- simonw 10 months agoThat comment where he explains why he's not rushing to add new unproven SQL syntax to SQLite is fascinating:
> My goal is to keep SQLite relevant and viable through the year 2050. That's a long time from now. If I knew that standard SQL was not going to change any between now and then, I'd go ahead and make non-standard extensions that allowed for FROM-clause-first queries, as that seems like a useful extension. The problem is that standard SQL will not remain static. Probably some future version of "standard SQL" will support some kind of FROM-clause-first query format. I need to ensure that whatever SQLite supports will be compatible with the standard, whenever it drops. And the only way to do that is to support nothing until after the standard appears.
- anitil 10 months agoIt's so ambitious in an almost boring way, exactly the right steward for a project like this
- maxbond 10 months agoDr. Hipp is one of my heroes. He seems to labor quietly in semi obscurity for decades, and at the end of it he's produced some amazing software. I was tickled by the curfuffle over his use of a set of guidelines for living in a Christian monastery as SQLite's code of ethics for the purpose of checking a box on an RFQ (part of the fallout of the libsql fork), because he does seem like a sort of programmer monk. (For what it's worth, as an agnostic, I've read them several times and found them unobjectionable. While I think the drama was unnecessary, the libsql people are doing interesting work.)
I choose never to meet this man and be disabused of this notion. Shine on, doctor.
- maxbond 10 months ago
- anitil 10 months ago
- Blackthorn 10 months agoFROM first would be nothing short of incredible. I can only hope that Postgres and others can find it within themselves to get together and standardize on such an extension!
- pradeepchhetri 10 months agoThis syntax looks a lot like PRQL. ClickHouse supports writing queries in PRQL dialect. Moreover, ClickHouse also supports Kusto dialect too.
https://clickhouse.com/docs/en/guides/developer/alternative-...
- willvarfar 10 months agoYeap I didn't know DuckDB supported it already!
Being able to do SELECT FROM WHERE in any order and allowing multiple WHEREs and AGGREGATE etc, combined with supporting trailing commas, makes copy pasting templating and reusing and code-generating SQL so much easier.
...FROM table <-- at this point there is an implicit SELECT * SELECT whatever WHERE some_filter WHERE another_filter <-- this is like AND AGGREGATE something WHERE a_filter_that_is_after_grouping <-- is like HAVING ORDER BY ALL <-- group-by-all is great in engines that support it; want it for ordering too
- quartesixte 10 months agoWhat exactly is the history of having FROM be the second item, and not the first? Because FROM first seems more intuitive and actually the way you write out queries.
Really hope this takes off and gets more widespread adoption because I really want to stop doing:
intoSELECT * FROM all_the_joins
SELECT {my statements here} FROM all_the_joins
- pradeepchhetri 10 months ago
- bvrmn 10 months agoIt's funny how he addresses the new syntax as "from-clause-first". Like a very minor change with a low value.
- Cthulhu_ 10 months agoI think that's important, because a lot of concepts are presented as prohibitively complicated; for example, functional programming makes sense in my head, but if you present it as lambda calculus and write it in concise form with new operators, you lost me.
- Cthulhu_ 10 months ago
- simonw 10 months ago
- tehlike 10 months agoLINQ, PRQL, Kusto has all preceeded this.
While LINQ is mostly restricted to .NET, PRQL is not. https://prql-lang.org/
It's a welcome change in the industry.
I made this prediction a couple years back: https://x.com/tehlike/status/1517533067497201666
- numbsafari 10 months agoThe paper directly references PRQL and Kusto. The main goal here is to take lessons learned from earlier efforts and try and find a syntax that works inside and alongside the existing SQL grammar, rather than as a wholly separate language.
- hn_throwaway_99 10 months agoI've been following PRQL for some time now since it first got good traction on HN and I like it a lot, but I'm really hoping this pipe syntax from Google takes off for a couple of reasons:
1. Similar to what you mention, while I think PRQL is pretty easy to learn if you know SQL, it still "feels" like a brand new language. This piped SQL syntax immediately felt awesome to me - it mapped how my brain likes to think about queries (essentially putting data through a chain of sieves and transforms), but all my knowledge of SQL felt like it just transferred over as-is.
2. I feel like I'm old enough now to know that the most critical thing for adoption of new technologies that are incremental improvements over existing technologies is to make the upgrade path as easy as possible. I shouldn't have to overhaul everything at once, but I just want to be able to take in small pieces a chunk at a time. While not 100% the same thing, if you look at the famously abysmal uptake of things like IPv6 and the pain it takes to use ES module-only distributions from NPM, the biggest pain point was these technologies made you do "all or nothing" migrations - they didn't have an easy, simple way to get from point A to point B. The thing I like about this piped SQL syntax is that in a large, existing code base I could easily just start adding this in new queries, but I wouldn't really feel the need to overhaul everything at once. With PRQL I'd feel a lot less enthusiastic about using that in existing projects where I'd have a mix of SQL and PRQL.
- lupire 10 months agoIt's wild that the enterprise and connected world has moved on from forcing COBOL compatibility for modern projects, but still insists on SQL compatibility.
- hn_throwaway_99 10 months ago
- andrewguy9 10 months agoI’m a big kusto user, and it’s wonderful to have pipes in a query language.
If you haven’t tried it, it’s great!
- tehlike 10 months agoI have not tried it, but I used to be a .net developer and worked a lot with LINQ (and contributed a bit to NHibernate and its Linq provider) and I am a big fan of the approach.
Kusto does seem interesting too, and i think some of the stuff i want to build will find a use for it!
- Salgat 10 months agoLINQ is so incredibly intuitive. I wonder if this will make creating C# LINQ providers for databases that support this syntax easier.
- Salgat 10 months ago
- kbouck 10 months agoIndeed. Elastic has also recently released a piped query language called ES|QL. Feels similar to Kusto.
I find piped queries both easier to write, and read.
- tehlike 10 months ago
- anonzzzies 10 months agoNot having LINQ is a terrible inconvenience everywhere. Most languages have libs that try to hack something similar, but it usually simply isn't.
- mrits 10 months agoIt's a lot easier to design a good DSL when it doesn't have to be compatible with anything
- anonzzzies 10 months agoWell, .NET was already used a lot when it was built in a few decades ago.
- anonzzzies 10 months ago
- mrits 10 months ago
- oaiey 10 months agoIs "from" keyword originating from .NET (Framework 3.5 in 2007) or is this pre-existing somewhere in research?
- numbsafari 10 months ago
- aragonite 10 months ago> This remains a long-standing pet peeve of mine. PDFs like this are horrible to read on mobile phones, hard to copy-and-paste from ...
I've never understood why copying text from digitally native PDFs (created directly from digital source files, rather than by OCR-ing scanned images) is so often such a poor experience. Even PDFs produced from LaTex often contain undesirable ligatures in the copied text like fi and fl. Text copied from some Springer journals sometimes lacks space between words or introduces unwanted space between letters in a word ... Is it due to something inherent in PDF technology?
- crazygringo 10 months ago> Is it due to something inherent in PDF technology?
Exactly. PDF doesn't have instructions to say "render this paragraph of text in this box", it has instructions to say "render each of these glyphs at each of these x,y coordinates".
It was never designed to have text extracted from it. So trying to turn it back into text involves a lot of heuristics and guesswork, like where enough separation between characters should be considered a space.
A lot also depends on what software produced the PDF, which can make it easier or harder to extract the text.
- vips7L 10 months agoMy favorite is when they do bold by duplicating and slightly shifting the letters. Bboolldd. PDFs are hell.
- lupire 10 months agoThat's inherited from the original Portable Document Format for machines - the typewriter instructions.
- lupire 10 months ago
- spatulon 10 months agoI've never looked into the PDF format, but, does it not allow for annotations that say, "the glyphs in the rectangle ((x0, y0), (x1, y1)) represent the text 'foobar'")? That's been my mental model for how they are text-searchable.
- kccqzy 10 months agoThey do but such annotations are optional.
- kccqzy 10 months ago
- 10 months ago
- vips7L 10 months ago
- jonathanyc 10 months agoPDF natively supports selectable/extractable text. Section 9.10 of ISO 32000 is literally “Extraction of Text Content.” I’ve implemented it myself in production software.
There are many good reasons why PDF has a “render glyph” instruction instead of a “render string”. In particular your printer and your PDF viewer should not need to have the same text shaping and layout algorithms in order for the PDF to render the same. Oops, your printer runs a different version of Harfbuzz!
The sibling comment is right that a lot depends on the software that produced the PDF. It’s important to be accurate about where the blame lies. I don’t blame the x86 ISA or the C++ standards committee when an Electron app uses too much memory.
- jahewson 10 months agoIt’s due to poor choices made in the implementation of pdfTeX. For example the TeX engine does not associate the original space characters with the inter-word “glue” that replaces them, so pdfTeX happily omits them. This was fixed a few years back, finally. But there’s millions(?) of papers out there with no spaces.
- mjevans 10 months agoligatures like fi fl ffi ffl etc are for changes in fonts specific to rendering correctly on a screen or printer. It's intended to be a _rendered_ format, rather than a parse-able format.
Well formatted epub and HTML generally are usually intended to update to end user needs and better fit available layout space.
- WorldMaker 10 months agoThough it's also a stuck legacy throwback. Modern advice would be to not send ligatures directly to the renderer and instead let the renderer poll OpenType features (and Unicode/ICU algorithms) to build them itself. PDF's baking of some ligatures in its files seems something of a backwards compatibility legacy mistake to still support ancient "dumb" PostScript fonts and pre-Unicode font encodings (or least pre-Unicode Normalization Forms). It's also a bit of the fact that PDF has always been confused about if it is the final renderer in a stack or not.
- jahewson 10 months agoThat wouldn’t work for PDF’s use case of being an arbitrary paper-like format because the various Unicode and OpenType algorithms don’t provide sufficient functionality for rendering arbitrary text: there are no one-size-fits all rules! The standards are a set of generic “best effort” guidelines for lowest-common-denominator text layout that are constantly being extended.
Even for English the exact tweaking of line breaking and hyphenation is a problem that requires manual intervention from time to time. In mathematics research papers it’s not uncommon to see symbols that haven’t yet made it into Unicode. Look at the state of text on the web and you’ll encounter all these problems; even Google Docs gave in and now renders to a canvas.
PDF’s Unicode handling is indeed a big mess but it does have the ability to associate any glyph with an arbitrary Unicode string, for text extraction purposes, so there’s nothing to stop the program that generates the PDF from mapping the fi ligature glyph to the to-character string “fi”.
- jahewson 10 months ago
- lupire 10 months agoThat's fine, but a good compiled format should also include a source map for accessibility.
- WorldMaker 10 months ago
- 0cf8612b2e1e 10 months agoIt is a shame that CSS pagination is still a mess. Not that I like CSS, but it would go a long way towards unlocking some layouts from PDF.
- jamesfinlayson 10 months agoAgreed - I used CSS to lay out a book a couple of years ago and it wasn't too bad, but the things that have poor support/don't work at all (like page numbers) are a pain to hack around.
- jamesfinlayson 10 months ago
- ericjmorey 10 months agoXPS solved a lot of the problems with PDF, but Microsoft couldn't reach a critical level of adoption to let network effects take hold.
However, I don't know if XPS handles the copying of text better.
- meindnoch 10 months agoIf a PDF doesn't support text extraction, it's the fault of the software that created it. Most likely the software didn't include the glyph → Unicode character mapping in the PDF.
- crazygringo 10 months ago
- summerlight 10 months agoPrevious submissions on the paper itself:
https://news.ycombinator.com/item?id=41321876 (first) https://news.ycombinator.com/item?id=41338877 (plenty of discussions)
I tried this new syntax and this seems a reasonable proposal for complex analytical queries. This new syntax probably does not change most simple transactional queries though. The syntax matches the execution semantic more closely, which means you less likely need to formulate query in a weird form to make query planner work as expected; usually users only need to move some pipe operators to more appropriate places.
- FridgeSeal 10 months agoKinda looks like a half-assed version of what PRQL does. Like, if we’re going to have nonstandard sql, let’s just fix a whole bunch of things, not just one or two?
- summerlight 10 months ago> Like, if we’re going to have nonstandard sql, let’s just fix a whole bunch of things, not just one or two?
I think they intentionally kept themselves away from massive redesign of the languages, which has a good chance of becoming multi decades of frustrating death march. I know a number of such cases from C++ standard proposals and probably the team wanted to avoid it.
- chubot 10 months agoThis is addressed in the paper -- it's nice to have something deployable in existing SQL languages, and it also doesn't rule out using PRQL
- hn_throwaway_99 10 months ago> Kinda looks like a half-assed version of what PRQL does. Like, if we’re going to have nonstandard sql, let’s just fix a whole bunch of things, not just one or two?
To be honest, this feels exactly like the kind of mistake that IPv6 made. It wasn't just "let's extend the IPv4 address space and provide an upgrade path that's as incremental as possible", it was "IPv4 has all these problems, lets solve the address space issue with a completely new address space, and while we're at it lets fix 20 other things!" Meanwhile, over a quarter century later, IPv4 shows no signs of going away any time soon.
I'd much rather have an incremental improvement that solves 90% of my pain points than to reach for some "Let's throw all the old stuff away for this new nirvana!" And I say this as someone that really likes PRQL.
- andrewshadura 10 months agoYou can't "just" extend the IPv4 address space while keeping the compatibility.
- andrewshadura 10 months ago
- summerlight 10 months ago
- scrlk 10 months agoThere was a second submission of the paper, which attracted more comments: https://news.ycombinator.com/item?id=41338877
- summerlight 10 months agoThank you, added it to my comment. I missed all the discussions!
- summerlight 10 months ago
- FridgeSeal 10 months ago
- BeefWellington 10 months agoEvery time this FROM-first syntax style crops up it's always the most basic simple query (one table, no projections / subselects / consideration to SP/Views).
Just for once I want to see complete examples of the syntax on an actual advanced query of any kind right away. Sure, toss out one simple case, but then show me how it looks when I have to join 4-5 reference tables to a fact table and then filter based on those things.
Once you do that, it becomes clear why SELECT first won out originally: legibility and troubleshooting.
As long as DBs continue to support standard SQL they can add whatever additional syntax support they want but based on history this'll wind up being a whole new generation of emacs vs vi style holy war.
- dietr1ch 10 months agoSounds a bit like "new thing scary" unless you show why having select in front actually avoids problems, and I don't think there's a clear problem they avoid, but it does make it really hard to autocomplete (can you even do it properly?) while something along the lines of just swap select for from is well defined.
- garrettgarcia 10 months ago> Sounds a bit like "new thing scary" unless you show why having select in front actually avoids problems
This isn't really fair. BeefWellington gave a reason why SQL is how it is (and how it has been for ~50 years). It's reasonable to ask for a compelling reason to change the clause order. Simon's post says it "has always been confusing", but doesn't really explain why except by linking to a blog post that says that the SQL engine (sort of but not really) executes the clauses in a different order.
I think the onus of proof that SQL clauses are in the wrong order is on the people who claim they're in the wrong order.
- Sankozi 10 months agoBut it has been explained many times from many angles.
* SELECT first makes autocomplete hard
* SELECT first is the only out of order clause in the SQL statement when you look at it from execution perspective
* you cannot use aliases defined in SELECT in following clauses
* in some places SELECT is pointless but it is still required (to keep things consistent?)
Probably many more.
- Sankozi 10 months ago
- mnsc 10 months agoThis is a case where stating your opinion and credentials will make you sound really old and conservative so it will be easy to take cheap shots like "you are just afraid of change".
At my previous gig I worked for a decade with an application that meant creating and maintaining large hairy sql that was created to offload application logic to the database (_very_ original) And we used to talk about this "wrong order" often but I never once actually missed it. It was at the most a bit annoying when you jumped in a server to troubleshoot and you knew the two columns you were interested in and you could have saved two seconds. But when working with maintaining those massive queries it always felt good to have the projection up top because that is the end result and what the query is all about. I would not have liked if the method signature in eg Java was just the parameters and the return type was after the final brace. This analogy falls apart of course since params are all over the place but swapping things around wouldn't help.
So just go 'SELECT *...' and go back and expand later, I want my sql syntax "simple". /old developer
- BeefWellington 10 months agoIt really isn't. I've been working in this field for ages and did a lot of those years as a DBA and data modeler. I've worked with other syntaxes too, mostly MDX but some others specific to Hadoop/Spark. I'm not afraid of new things. I just want them to improve on what we have. I want them to be honest about situations where their solution isn't great.
SQL has lots of warts, e.g.: the fact that you can write SQL that joins tables without including those tables in a JOIN, which leads to confusion. It's fragmented too -- the other example I posted shows two different syntaxes for TOP N / LIMIT N because different vendors went different ways. The fact that some RDBMSes provide locking hint mechanics and some don't (at least not reliably). The fact that there's no standard set of "library" functions defined anywhere, so porting between databases requires a lot of validation work. It makes portability hard, and some of those features are missing from standards.
You'll note I also mentioned that if they want to add it that's fine but it's gonna wind up being a point of contention in a lot of places. That's because I've seen the same thing happen with the "Big Data" vs "what we have works" crowd.
Having select up front avoids problems in a couple key ways:
1. App devs who are working on their application can immediately see what fields they should expect in their resultset. For CRUD, it's probably usually just whatever fields they selected or `*` because everyone's in the habit of asking for every field they'll never use.
2. Troubleshooting problems is far easier because they almost always stem from a field in the projection. Seeing the projected field list (and thus, table aliases that field comes from) are literally the first pieces of information you need (what field is it and where does that field come from) to start troubleshooting. This is why SELECT ... FROM makes the most sense -- it's literally the two most crucial pieces of information right up front.
3. Query planners already optimize and essentially compile the entire thing anyways, so legibility trumps other options IME.
Another point I'd make to you and everyone else bringing up autocomplete: If you need it, nothing is stopping you from writing your FROM clause first and then moving a line up to write your SELECT. Kinda like how you might stub out a function definition and later add arguments. This doesn't affect the final form for legibility.
- garrettgarcia 10 months ago
- nsonha 10 months ago> becomes clear why SELECT first won out originally: legibility and troubleshooting
nothing "becomes clear" just by you claiming so, better elaborate
- jshute4444 10 months agoFor examples of larger queries, see here for all TPC-H queries in standard syntax and converted to pipe syntax: https://github.com/google/zetasql/blob/master/zetasql/exampl...
And several more examples with pipe syntax here: https://github.com/google/zetasql/blob/master/zetasql/exampl...
- WorldMaker 10 months ago> Once you do that, it becomes clear why SELECT first won out originally: legibility and troubleshooting.
Select first was as much an accident of "it sounded better as an English sentence" to the early SQL designers. Plus also they were working with early era parsers with very limited look ahead and putting the primary "verb" up front was important at the time.
But English is very flexible, especially in "command syntax" and From first is surprisingly common: "From the middle cupboard, grab a plate". SQL trying to sound like English here only shows how inflexible it still is in comparison to actual English.
I've been using C#'s LINQ since it was added to the language in 2007 and the from/where/join/group by/select order feels great, is very legible especially because it gives you great autocomplete support, and troubleshooting is easier than people think.
- mixedCase 10 months agohttps://prql-lang.org/ has a bunch of good examples on its home page.
If you engage the syntax with your System 2 thinking (prefrontal cortex, slow, the part of thinking we're naturally lazy to engage) rather than System 1 (automated, instinctual, optimized brain path to things we're used to) you'll most likely find that it is simpler, makes more logical sense so that you're filtering down things naturally like a sieve and composes far better than SQL as complexity grows.
After you've internalized that, imagine the kind of developer tooling we can build on top of that logical structure.
- meepmorp 10 months ago> If you engage the syntax with your System 2 thinking (prefrontal cortex, slow, the part of thinking we're naturally lazy to engage) rather than System 1 (automated, instinctual, optimized brain path to things we're used to)
You might not have intended it this way, but your choice of phrasing is very condescending.
- mixedCase 10 months agoRe-reading it I can see how it could be perceived by some people as such, thanks for pointing it out. There's probably better phrasing or adding more context could make it more amicable:
The goal was to explicitly tell people not to bother "just reading it" as one (and by one I mean myself and most people I know, surely there are exceptions) is naturally inclined to do unless something is particularly piquing our interest.
Without engaging in active, conscious effort, syntax that is different than what we're used to (specially something as established as SQL) where the changes aren't groundbreaking at first glance can easily make us dismissive without realizing the benefits. And after seeing it too many times with all kinds of technologies that stray away from the familiar, I just want to prepare the reader so that their judgment can be formed with full use of their faculties rather than a reflex response.
- mixedCase 10 months ago
- BeefWellington 10 months agoEdit: In my pre-coffee rush this morning I completely missed the grouping by role (which is not that much harder FWIW). This unfortunately invalidates my entire post as it was posted and I don't want to spread misinfo.
- fader 10 months agoI don't think your alternatives actually solve the same problem. Your alternatives would give you the single most recently joined employee. The actual problem being solved is to find the most recently joined employee in each role.
You'd need to do some grouping in there to be able to get one employee per role instead of a single employee out of the whole data set.
- fader 10 months ago
- meepmorp 10 months ago
- summerlight 10 months agoAs a test, I refactored a 500 line-ish analytical query that joins more than 20 tables with tens of complex CTE and I can say that this FROM-first syntax is superior than the legacy syntax on almost every single aspect.
- bvrmn 10 months ago> SELECT first won out originally: legibility and troubleshooting.
It quite interesting to dive into history of SQL alternatives in 70x/80x.
- WesolyKubeczek 10 months ago> Once you do that, it becomes clear why SELECT first won out originally: legibility and troubleshooting.
Also, tools can trivially tell DQL from DML by the first word they encounter, barring data-modifying functions (o great heavens, no!).
- otabdeveloper4 10 months agoFROM order is, like, the least offensive and least wrong thing about SQL.
Bikeshedding par excellence.
- dietr1ch 10 months ago
- urbandw311er 10 months agoTitle should probably be changed, since the article is about using AI to convert a PDF to semantic HTML.
- simonw 10 months agoA surprising problem I'm seeing with maintaining a link blog is that articles from it occasionally get submitted to Hacker News, where people inevitably call them out as not being as appropriate as the source they are linking to - which is fair enough! That's why I don't tend to submit them myself.
This particular post quickly turned into a very thinly veiled excuse for me to complain about PDFs, then demonstrate a Gemini Pro trick.
In this case I converted to HTML - I've since tried converting a paper to Markdown and sharing in a Gist, which I think worked even better: https://gist.github.com/simonw/46a33d66e069efe5c10b63625fdab... - notes here https://simonwillison.net/2024/Aug/27/distro/
- llimllib 10 months agoHave you seen gist.io?
If you replace `gist.github.com/<user>/<id>` -> `https://gist.io/@<user>/<id>`, you get a gist with nice typography.
https://gist.io/@simonw/46a33d66e069efe5c10b63625fdabb4e is the same gist you linked, but nicer to read
- simonw 10 months agoThat's pretty neat! I like that it's run by a GitHub employee too (presumably as a side-project, but still) - makes me less nervous about the domain name blinking out of existence one day.
- simonw 10 months ago
- llimllib 10 months ago
- 10 months ago
- simonw 10 months ago
- yarg 10 months agoThis reminds me .NET's short lived Linq to SQL;
There was a talk at the time, but I can't find the video: http://jaoo.dk/aarhus2007/presentation/Using+LINQ+to+SQL+to+....
Basically, it was a way to cleanly plug SQL queries into C# code.
It used this sort of ordering (where the constraints come after the thing being constrained); it needed to do so for IntelliSense to work.
- cyberax 10 months ago"Short-lived"? LINQ is very much alive in the C# ecosystem.
And FROM-first syntax absolutely makes more sense, regardless of autocomplete. You should put the "what I need to select" after the "what I'm selecting from", in general.
- yarg 10 months agoLINQ yes, but they killed off the component not long after introducing it.
- 10 months ago
- jiggawatts 10 months agoIt was replaced by Entity Framework.
- BartjeD 10 months agoLinq to sql still lives
- 10 months ago
- yarg 10 months ago
- dragonwriter 10 months ago> This reminds me .NET's short lived Linq to SQL;
"Short lived"? Its still alive, AFAIK, and the more popular newer thing for the same use case, Linq to Enntities, has the same salient features but (because it is tied to Entity Framework and not SQL Server specific) is more broadly usable.
- yarg 10 months agoIt was in 3.5 only.
If they've replaced it with something else in the last decade and a half that does not mean that they didn't get rid of it, or that it wasn't short lived.
https://learn.microsoft.com/en-us/dotnet/framework/data/adon...
- LeonB 10 months agoYeh. Linq to sql was a much more lightweight extension than EF, and was killed due to internal warring at MS.
Database people were investing a lot of time and energy on doing things “properly” with EF, and this scrappy little useful tool, linq to sql, was seen as a competitor.
- plusplusungood 10 months agoLINQ is not the same as LINQ-to-SQL. The former is a language feature, the latter a library (one of many) that uses that feature.
- LeonB 10 months ago
- yarg 10 months ago
- neonsunset 10 months agoThere is https://github.com/linq2db/linq2db which is LINQ to SQL reincarnated.
Of course there's EF Core too.
- WorldMaker 10 months agoAnd NHibernate.Linq and Dapper.Extensions.Linq… Most ORMs in the ecosystem have at least one Linq support library, even if just a third-party extension.
Also, there are fun things that support Linq syntax for non-ORM uses, too, such as System.Reactive.Linq and LanguageExt: https://github.com/louthy/language-ext/wiki/How-to-deal-with...
- WorldMaker 10 months ago
- cyberax 10 months ago
- mav3ri3k 10 months agoThe first piped query language I used was Nushell's implementation of wide-column tables. PRQL offers almost similar approach which I have loved dearly. It also maps to different SQL dialects. There is also proposal to work on type system: https://github.com/PRQL/prql/issues/381.
Google has now proposed a syntax inspired by these approaches. However, I am afraid how well it would be adopted. As someone new to SQL, nearly every DB seem to provide its own SQL dialect which becomes cumbersome very quickly.
Whereas PRQL feels something like Apache Arrow which can map to other dialects.
- 0xbadcafebee 10 months agoAs to the writer's problem with PDFs on the web: they aren't for reactive web app viewing on mobile phones. Not everything has to be. If you reeeeeeeally need to read that research paper, find a screen that's bigger than 3" wide.
- jillesvangurp 10 months agoI think his point is that Google is a web company. And a mobile phone company. And they publish a lot of stuff in a format that's basically optimized for print and kind of useless for anything else.
I did my PhD more than 20 years ago and it was annoying then to be working with all these postscript and pdf documents. It's still annoying. These days people publish content in PDF form on websites and mostly not in printed media. People might print these or not. Twenty years ago, I definitely did. But it's weird how we stick with this. And PDFs are of course very unstructured and hard to make sense of programmatically as well.
I bet a lot of modern day scientists don't actually print the articles they read anymore and instead read them on screen or maybe on some ipad or e-reader. Print has become an edge case. Reading a pdf on a small e-reader is not ideal. Anything with columns is kind of awkward to deal with. There's a reason why most websites don't use columns: it kind of sucks as a UX. The optimal form to deliver text is in a responsive form that can adapt to any screen size where you can change the font size as well. A lot of scientific paper layouts are optimized to conserve a resource that is no longer relevant: paper real estate. Tiny fonts, multiple columns, etc.
Anyway, I like Simon's solution and how it kind of works. It's kind of funny how some of these LLMs can be so lazy. The thing with the references being omitted is hilarious. I see the same with chat gpt where it goes out of its way to never do exactly as you asked and instead just give you bits and pieces of what you ask for until you beg it to just please FFing do as you're told?! I guess they are trying to save some tokens or GPU time.
- simonw 10 months agoWhy shouldn’t I read research papers on my phone? That’s where I read almost everything else.
- adrian_b 10 months agoEven when reading on the phone, I do not understand the complaint against the two-column format.
The one-column format is fine on a large monitor, but on a small phone I prefer narrower columns, because a wide column would either make the text too small or it would require horizontal panning while reading.
So I consider the two-column format as better for phones, not worse.
- 9dev 10 months agoOne of the most complex and battle-tested open source projects is essentially a rendering engine for semantic text that has supported reflowing text to fit the screen for decades. And now you’re seriously considering having to zoom in on a column, then scrolling all the way back up and right to the next column, then down to the footnotes at the bottom, then to a random figure, to be a solution?
- 9dev 10 months ago
- adrian_b 10 months ago
- jillesvangurp 10 months ago
- slaymaker1907 10 months agoI actually work on SQL Server, but I also write a lot of KQL queries which also work this way and I totally agree that the sequential pipe stuff is easier to write. I haven't read through the whole paper, but one aspect that I really like is that I think it's easier to guide the query optimization in this sequential style.
- beart 10 months agoIs there any internal inertia for such changes to SQL server?
- WorldMaker 10 months agoGiven how Entity Framework is quite ubiquitous as "the ORM of choice" for SQL Server and its usage of C# Linq, there's certainly external momentum, whether or not SQL Server devs themselves are paying attention to how the majority of their users are writing queries today.
- WorldMaker 10 months ago
- beart 10 months ago
- donatj 10 months agoI've been writing SQL for something like 25 years and always thought the columns being SELECTed should have come last, not first. Naming your sources before what you're trying to get from them to me at least makes much more logical sense. Calling aliased table names before I have done the aliasing is weird.
Also it would make autocomplete in intelligent IDEs much more helpful when typing a query out from nothing.
- victorbjorklund 10 months agoLooks just like writing sql using Ecto in Elixir:
"users" |> where([u], u.age > 18) |> select([u], u.name)
- h0l0cube 10 months agoThought this too. The example queries look very much like Ecto statements. I miss the ergonomics and flexibility of Ecto when I use database wrappers on other platforms.
- h0l0cube 10 months ago
- chubot 10 months agoThe next thing I would like is to define a function / macro that has a bunch of |> terms.
I pointed out that you can do this with shell:
Pipelines Support Vectorized, Point-Free, and Imperative Stylehttps://www.oilshell.org/blog/2017/01/15.html
e.g.
Something like that should be possible in SQL!hist() { sort | uniq -c | sort -n -r } $ { echo a; echo bb; echo a; } | hist 1 bb 2 a $ foo | hist ...
- jshute4444 10 months agoIt is, using table-valued functions (TVFs).
There's an example at the bottom of this file:
https://github.com/google/zetasql/blob/master/zetasql/exampl...
- chubot 10 months agoThat's cool, thanks!
What about scalar valued functions? :) So I can reuse an expression in a WHERE and so forth
(and I appreciate that HAVING can be generalized/removed)
- chubot 10 months ago
- jshute4444 10 months ago
- wvenable 10 months agoI didn't see this the first time:
Is this kind of syntax combining grouping and ordering really necessary in addition the pipe operator? My advice would be to add the pipe operator and not get fancy adding other syntax to SQL as well.GROUP AND ORDER BY component_id DESC;
- bvrmn 10 months agoIt could be a custom zetasql extension leaked into the paper.
- bvrmn 10 months ago
- minkles 10 months agoThat is basically R with tidyverse.
If you haven't used R, it has some serious data manipulation legs built into it.flights |> filter( carrier == "UA", dest %in% c("IAH", "HOU"), sched_dep_time > 0900, sched_arr_time < 2000 ) |> group_by(flight) |> summarize( delay = mean(arr_delay, na.rm = TRUE), cancelled = sum(is.na(arr_delay)), n = n() ) |> filter(n > 10)
- dan-robertson 10 months agoAn interesting thing to me about all these dplyr-style syntaxes is that Wickham thinks the group_by operator was a design mistake. In modern dplyr you can often specify a .by on an operation instead. I found switching to this style a pretty easy adjustment, and I think it’s a bit better. Example:
I think PRQL were thinking a bit about ways to avoid a group_by operation and I think what they have is a kind of ‘scoped’ or ‘higher order’ group_by operation which takes your grouping keys and a pipeline and outputs a pipeline step that applies the inner pipeline to each group.d |> filter(id==max(id),.by=orderId)
- _Wintermute 10 months agoGiven 10 more years dplyr syntax might resemble data.table's
- _Wintermute 10 months ago
- countrymile 10 months agoMy thoughts exactly, it even uses the same pipe syntax, though I do prefer `%>%`. I've been avoiding SQL for a while now as it feels so clunky next to the tidyverse
- dan-robertson 10 months ago
- AdieuToLogic 10 months agoIf anyone is interested in the theoretical background to the thrush combinator, a.k.a. "|>", here is one using Ruby as the implementation language:
https://leanpub.com/combinators/read#leanpub-auto-the-thrush
Being a concept which transcends programming languages, a search for "thrush combinator" will yield examples in several languages.
- wslh 10 months agoI find this [1] from this [2]. Seems like a good explanation. It doesn't exist on Wikipedia though.
[1] https://github.com/raganwald-deprecated/homoiconic/blob/mast...
- AdieuToLogic 10 months agoA key thing to keep in mind is that the thrush combinator is a fancy name for a simple construct. The semantics it provides is a declarative form of traditional function composition.
For example, given the expression:
The same can be expressed in languages which support the "|>" infix operator as:f (g (h (x)))
There are other, equivalent, constructs such as the Cats Arrow[0] type class available in Scala, the same Arrow[1] concept available in Haskell, and the `andThen` method commonly available in many modern programming languages.h (x) |> g |> f
- AdieuToLogic 10 months ago
- wslh 10 months ago
- Ericson2314 10 months agoWe should really standardize a core language for SQL. Rust has MIR, Clang is making a CIR for C/C++. Once we have that, we'll be able to to communicate much better.
Right now, it's everyone faffing around with different mental models and ugly single pass compilers (my understanding is that parsing-->query planning is not nearly as well-separated in most DBs as parsing-->optomize-->codegen in most compilers).
- anothername12 10 months ago> We should really standardize a core language for SQL
Do you mean something other than ISO/IEC 9075:2023 (the 9th edition of the SQL standard)?
- roenxi 10 months agoIt costs 194 CHF to read. There is room for improvement.
- Ericson2314 10 months agoA core language is a minimal AST without surface syntax (and thus no bikeshedding of that) that distills the surface language to its essence.
- Ericson2314 10 months agoSQL is basically the list monad, with various quotients / refinements:
- Sometimes the order doesn't matter - Sometimes there are functional dependencies - Sometimes one knows the length of the list in question is 1 (foreign key constraints)
- Ericson2314 10 months ago
- roenxi 10 months ago
- rrrrrrrrrrrryan 10 months agoANSI SQL is very much a thing, and you should strive to keep your queries as close as possible to standard SQL as your database engine allows, if you want those queries to be portable to other database technology in the future.
- yencabulator 10 months agoYou might enjoy https://substrait.io/
- anothername12 10 months ago
- verdverm 10 months agoThe research paper: https://storage.googleapis.com/gweb-research2023-media/pubto...
- Zopieux 10 months agoI just want trailing commas allowed everywhere. I can't believe this 2024 and we still have to deal with this crap. Humanity deserves better.
Syntax/DSL designers: if your language uses a separator for anything, please kindly allow trailing versions of that separator anywhere possible.
- themerone 10 months agoMy big wish for SQL is for single row inserts to have a {key: value} syntax.
- zX41ZdbW 10 months agoIn ClickHouse you can do
It works with all other formats as well.INSERT INTO table FORMAT JSONEachRow {"key": 123}
Plus, it is designed in a way so you can make an INSERT query and stream the data, e.g.:
clickhouse-client --query "INSERT INTO table FORMAT Protobuf" < data.protobuf curl 'https://example.com/?query=INSERT...' --data-binary @- < data.bson
- BostonFern 10 months agoMySQL has it without the braces.
- nickpeterson 10 months agoThis would condense lines of code by a lot and prevent a lot of dumb bugs.
- zX41ZdbW 10 months ago
- gopiandcode 10 months agoI find this particular choice of syntax somewhat amusing because the pipe notation based query construction was something I ended up using a year ago when making an SQL library in OCaml:
https://github.com/kiranandcode/petrol
An example query being:
```
let insert_person ~name:n ~age:a db = Query.insert ~table:example_table ~values:Expr.[ name := s n; age := i a ] |> Request.make_zero |> Petrol.exec db
```
- KronisLV 10 months agoThis feels like this should be in the official SQL standard and supported across a bunch of RDBMSes and understood by IDEs, libraries and frameworks.
- riku_iki 10 months agoYeah, and we will have two standards given popularity of existing syntax
- riku_iki 10 months ago
- middayc 10 months agoLooking at the first example from PDF:
You could do something similar with Ryelang's spreadsheet datatype:FROM customer |> LEFT OUTER JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%unusual%packages%' |> AGGREGATE COUNT(o_orderkey) c_count GROUP BY c_custkey |> AGGREGATE COUNT(*) AS custdist GROUP BY c_count |> ORDER BY custdist DESC, c_count DESC;
Looking at this, maybe we should add an option to name the new aggregate column (now they get named automatically) in group-by function because c_custkey_count_count is not that elegant for example.customers: load\csv %customers.csv orders: load\csv %orders.csv orders .where-not-contains 'o_comment "unusual packages" |left-join customers 'o_custkey 'c_custkey |group-by 'c_custkey { 'c_custkey count } |group-by 'c_custkey_count { 'c_custkey_count count } |order-by 'c_custkey_count_count 'descending
- rileymat2 10 months agoIs there research on what is easier to read when you are sifting through many queries?
I like the syntax for reading what the statement expects to output first, even though I agree that I don’t write them select first. I feel like this might be optimizing the wrong thing.
Although the example is nice, it does not show 20 tables joined first, which will really muddle it.
- beart 10 months agoThe select list is meaningless without everything that follows. Knowing that a query selects "id, "date" tells you nothing without knowing the table, the search criteria, etc.
- antonvs 10 months agoThat's one benefit of the SQL naming convention which would use names like e.g. customer_id, invoice_date, etc. Also, when joining tables (depending on the SQL dialect) that can allow a shortcut synax, JOIN ON field_name, if the field name in the two tables is the same.
- aragonite 10 months agoI really wish SQL used "RETURN" instead of "SELECT" (like in XQuery):
1. Calling it "RETURN" makes the fact of its later order of execution (relative to FROM etc) less surprising.
2. "RETURN RAND()" just reads more naturally than "SELECT RAND()". After all, we're not really "selecting" anything here, are we?
3. Would also eliminate any confusion with the selection operation in relational algebra.
- rileymat2 10 months agoIf you name fields that way, but accountId, createDate may not be meaningless in the context you are looking at.
- antonvs 10 months ago
- beart 10 months ago
- delegate 10 months agoThere's honeysql library in Clojure, where you define queries as maps, which are then rendered to SQL strings:
Since maps are unordered, this is equivalent to{:select [:name :age] :from {:people :p} :where [:> :age 10]}
and also{:from {:people :p} :select [:name :age] :where [:> :age 10]}
These can all be rendered to 'SELECT... FROM' or 'FROM .. SELECT'.{:where [:> :age 10] :select [:name :age] :from {:people :p}}
Queries as data structures are very versatile, since you can use the language constructs to compose them.
Queries as strings (FROM-first or not) are still strings which are hard to compose without breaking the syntax.
- OptionOfT 10 months ago> GROUP AND ORDER BY component_id DESC;
This feels like too much. GROUP BY and ORDER BY are separate clauses, and creating a way to group (heh) them in one clause complicates cognitive load, especially when there is an effort to reduce the overall effort to parse the query in your mind (and to provide a way for an intellisense-like system a way to make better suggestions).
vsGROUP AND ORDER BY x DESC;
This long form is 1 word longer, but, it easier to parse in your mind, and doesn't introduce unneeded diffs when changing either the GROUP or the ORDER BY column reference.GROUP BY x; ORDER BY x DESC;
- isoprophlex 10 months agoI love the idea but something in my brain starts to itch when I see that pipe operator
What IS that thing? A unix pipe that got confused with a redirect? A weird smiley of a bird wearing sunglasses?|>
It'll take some getting used to, for me...
- WorldMaker 10 months agoIt's like other "arrow" digraphs in common programming languages today, such as =>. You can picture it as a triangle pointing to the right.
Many Programming Ligature fonts even often draw it that way. For instance it is shown under F# in the Fira Code README: https://github.com/tonsky/FiraCode
- summerlight 10 months agoThey considered ditching `|>` or using `|` but unfortunately there's a bunch of syntactic ambiguity.
- WorldMaker 10 months ago
- OscarCunningham 10 months ago> Rationale: We used the same operator name for full-table and grouped aggregation to minimize edit distance between these operations. Unfortunately, this puts the grouping and aggregate columns in different orders in the syntax and output. Putting GROUP BY first would require adding a required keyword before the AGGREGATE list.
I think this is bad rationale. Having the columns in order is much more important than having neat syntax for full-table aggregation.
- philippta 10 months agoWhy even add the pipe operator?
If the DB engine is executing the statement out of order, why not allow the statement to be written in any order and let itself figure it out?
- julien040 10 months agoI haven't seen it mentioned yet, but it reminds me of PQL (not PRQL): https://pql.dev
It's inspired by Kusto and available as an open-source CLI. I've made it compatible with SQLite in one of my tools, and it's refreshing to use.
An example:
StormEvents | where State startswith "W" | summarize Count=count() by State
- eezing 10 months agoFor autocomplete, FROM first makes a lot of sense. For readability, SELECT first makes more sense because the output is always at the top.
- nagisa 10 months agoPeople here are describing many projects that already have something resembling this syntax and concept, so I'll add another query language to the pile too: Influx's now-mostly-abandoned Flux. Uses the same |> token and structures the query descriptions starting with an equivalent of "FROM".
- ahmed_ds 10 months agoThis is why I like tools like datastation and hex.tech. You write the initial query using SQL than process the results as a dataframe using Python/pandas. Surely, mixing Pandas and SQL like that is not good for data pipelines but for exploration and analytics, I have found this approach to be enjoyable.
- theodpHN 10 months agoYes, it's very convenient to be able to use SQL with your massively parallel commercial database (Oracle, Snowflake, etc.) and then again with the results sets (Pandas, etc.). Interestingly, it's a concept that was implemented 35 years ago in SAS (link below) but is just now gaining traction in today's "modern" software (e.g., via DuckDB).
USING THE NEW SQL PROCEDURE IN SAS PROGRAMS (1989) https://support.sas.com/resources/papers/proceedings-archive... The Sql procedure uses SQL to create, modify, and retrieve data from SAS data sets and views derived from those data sets. You can also use the SOL procedure to join data sets and views with those from other database management systems through the SAS/ACCESS software interfaces.
- ahmed_ds 10 months agoWow, that is really cool. One of my theses is that DuckDB will be bought by GCP (BigQuery), and polars will be bought by Databricks (or AWS). The thesis is based on the idea that Snowflake bought the Modin platform. The movement in DE seems to be towards data warehouse platforms streaming data (views/results) down to dataframe (Modin, Polars, DuckDB) platforms, which then stream down to their BI platforms. Because these database platforms are designed as OLAP platforms so, this approach makes sense.
- ahmed_ds 10 months ago
- theodpHN 10 months ago
- aloukissas 10 months agoThis like Elixir's pipe operator [1]! I use it on the daily (with Ecto) and it's epic!
[1] https://elixirschool.com/en/lessons/basics/pipe_operator
- stevefan1999 10 months agoThat's just Linq from C# except Google want to make it a SQL standard...
- dang 10 months agoRecent and related:
Pipe Syntax in SQL - https://news.ycombinator.com/item?id=41338877 - Aug 2024 (219 comments)
- datadeft 10 months ago> It's been 50 years. It's time to clean up SQL. This
Is it though?
Are we trying to solve the human SQL parser and generator problem or there is some underlying implementation detail that benefits from pipes?
- eternauta3k 10 months agoDo manually-generated SQL strings have a place outside of interactive use? I use them in my small projects but I wonder if a query builder isn't better for larger systems.
- otabdeveloper4 10 months agoQuery building for an analytics database is impossible.
These queries are always hand-rolled because you pay the analysts to optimize them.
- otabdeveloper4 10 months ago
- oznog 10 months agoSQL replacements is like not understanding the magnitude of the success of something so old.
SQL is fine.
SQL has been the state of the art for db queries for 40 years.
And it will continue to be when we all retire.
- jiggawatts 10 months agoThey’re a bit late to the game, there’s are least a dozen such popular query languages. LINQ and KQL come to mind, but there are many others…
- metadat 10 months agoSimon: Please keep pushing, and mute nothing.
- carabiner 10 months agoI like this. Reminds me of pandas.
- 10 months ago
- fridental 10 months agoFor the sake of God, please fucking stop inventing new pipe languages.
LINQ: exists
Splunk query language: exists
KQL: exists
MongoDB query language: exists
PRQL: exists
- pxc 10 months agoLINQ, Splunk, and KQL are all proprietary. For the purposes of setting new standards, they might as well not exist.
PRQL is the only real entrant in your list when it comes to adding a pipelining syntax to a language for relational queries in a way that others can freely build on.
- bvrmn 10 months agoSQL parsers: exists.
The paper clearly describes the goal: add a pipe syntax into existing systems with minor changes and be compatible with existing SQL queries.
BTW: LINQ is an AST transformer not a language per se tied to a particular platform. None of existing DBs allows to use it directly.
- pxc 10 months ago
- 10 months ago
- 1024core 10 months agoIsn't this the same syntax (or very similar to) Apache Beam?
- notfed 10 months agoIs it just me, or does this seem anachronistic? Like, this is a conversation I expected to blow up 20 years ago. Better late than never.
- make3 10 months agothis reads like an article written by someone with adhd who started writing about a scientific paper but got distracted by some random thing instead of reading it
- simonw 10 months agoSee my comment here: https://news.ycombinator.com/item?id=41385143
- simonw 10 months ago
- rosencrantz 10 months agoint *ptr;
// but let's change it to *int ptr;
// because the pointer symbol is more logical to write first
Please can we solve a real problem instead?
- jappgar 10 months agoWait, is this post about SQL or PDF...
- thenegation 10 months agoNow wondering if there is any relation to "Structural versus Pipeline Composition of Higher-Order Functions (Experience Report)":
https://cs.brown.edu/~sk/Publications/Papers/Published/rk-st...
- sharpshadow 10 months agoI have to honestly say that I like PDFs they always work and don’t fail without JS.
- 10 months ago
- Liona234 10 months ago[dead]