Silkenweb Example: Hackernews Clone

How the Windows Subsystem for Linux Redirects Syscalls

359 points by jackhammons 9 years ago | 266 comments

ataylor284_ 9 years ago
> The real NtQueryDirectoryFile API takes 11 parameters
Curiosity got the best of me here: I had to look this up in the docs to see how a linux syscall that takes 3 parameters could possibly take 11 parameters. Spoiler alert: they are used for async callbacks, filtering by name, allowing only partial results, and the ability to progressively scan with repeated calls.
- bitwize 9 years ago
  This is a recurring pattern in Windows development. Unix devs look at the Windows API and go "This syscall takes 11 parameters? GROAN." But the NT kernel is much more sophisticated and powerful than Linux, so its system calls are going to be necessarily more complicated.
  - trentnelson 9 years ago
    Curiosity got the better of me recently when I re-read Russinovich's [NT and VMS - The Rest Of The Story](http://windowsitpro.com/windows-client/windows-nt-and-vms-re...), and I bought a copy of [VMS Internals and Data Structures](http://www.amazon.com/VAX-VMS-Internals-Data-Structures/dp/1...).
    Side-by-side, comparing VMS to UNIX, and VMS's approach to a few key areas like I/O, ASTs and tiered interrupt levels are simply just more sophisticated. NT inherited all of that. It was fundamentally superior, as a kernel, to UNIX, from day 1.
    I haven't met a single person that has understood NT and Linux/UNIX, and still thinks UNIX is superior as far as the kernels go. I have definitely alienated myself the more I've discovered that though, as it's such a wildly unpopular sentiment in open source land.
    Cutler got a call from Gates in 89, and from 89-93, NT was built. He was 47 at the time, and was one of the lead developers of VMS, which was a rock-solid operating system.
    In 93, Linus was 22, and starting "implementing enough syscalls until bash ran" as a fun project to work on.
    Cutler despised the UNIX I/O model. "Getta byte getta byte getta byte byte byte." The I/O request packet approach to I/O (and tiered interrupts) is one of the key reasons behind NT's superiority. And once you've grok'd things like APCs and structured exception handling, signals just seem absolutely ghastly in comparison.
    - filereaper 9 years ago
      Since we're going into the history of Windows NT, VMS and Dave Cutler. I'd like to highlight this classic book on the history of all three of the above[1]
      It follows the same line of narrative as The Soul of a New Machine
      [1] https://www.amazon.com/Showstopper-Breakneck-Windows-Generat...
    - jen20 9 years ago
      I've never met a single person who understood what they were talking about and referred to a "UNIX kernels". It may be true that Linux was once less advanced than NT - this is no longer the case, despite egregious design flaws in things like epoll. It has simply never been true (for example) for the Illumos (nee Solaris) kernel.
    - 1024core 9 years ago
      The joke used to be: VMS++ --> WNT
    - adamnemecek 9 years ago
      Can I ask what else is on your reading list? I ended up buying the VMS Internals book.
      Also do you have an opinion on BeOS?
  - tremon 9 years ago
    But the NT kernel is much more sophisticated and powerful than Linux
    That does not follow from the example. All it shows is that Microsoft prefers to put a lot of functionality in one interface, while Linux probably prefers low-level functions to be as small as possible, and probably offers things like filtering on a higher level (in glibc, for example).
    Neither explanation has anything to do with sophistication. I personally believe that small interfaces are a better design.
    - bitwize 9 years ago
      Actually it does, as it mentioned that the extra parameters are for things like async callbacks and partial results.
      The I/O model that Windows supports is a strict superset of the Unix I/O model. Windows supports true async I/O, allowing process to start I/O operations and wait on an object like an I/O completion port for them to complete. Multiple threads can share a completion port, allowing for useful allocation of thread pools instead of thread-per-request.
      In Unix all I/O is synchronous; asynchronicity must be faked by setting O_NONBLOCK and buzzing in a select loop, interleaving bits of I/O with other processing. It adds complexity to code to simulate what Windows gives you for real, for free. And sometimes it breaks down; if I/O is hung on a device the kernel considers "fast" like a disk, that process is hosed until the operation completes or errors out.
  - ckaygusu 9 years ago
    I think the problem here is not a syscall taking 11 parameters, it's a syscall that merely lists what is inside a directory taking 11 parameters. ataylor_284 explained the reasons (how convincingly, I'd argue) but on the first sight that surely smells bloat.
    I'd also object NT kernel being more "powerful". Sure unixy kernels and NT has their differences but I don't think either one is superior.
    - bigger_cheese 9 years ago
      11 parameters may seem bloated but in some cases Unix syscalls weren't designed with enough parameters whcih caused a bunch of pain necessitating things like
      dup->dup2->dup3 pipe->piep2 rename->renameat->renameat2
      Best practice nowadays in linux is to allow overloading syscalls via a flags parameter.
      see https://lwn.net/Articles/585415/
      So modern linux syscalls may be bloated too.
    - rbanffy 9 years ago
      I remember the struct I had to populate to start a new process in 1997 or 1998...
  - darkengine 9 years ago
    It may be more "sophisticated" (sounds like a more positive synonym of "complex" to me), but I certainly don't think it's more powerful.
  - deprave 9 years ago
    Since when is kernel complexity a measure of quality...? :)
    - jasonm23 9 years ago
      hmmm...
      Usage of the adjective "sophisticated" always precedes an outpouring either ignorance or straight bs.
  - pjmlp 9 years ago
    Also UNIX devs seem to forget how cumbersome the X11, Xlib and Motif APIs are.
  - pbarnes_1 9 years ago
    This was maybe the case at Linux 2.0, but is not the case now.
    Also, Windows development is infinitely more painful than Unix/Linux.
  - uudecode 9 years ago
    "... so its system calls are going to be necessarily more complicated."
    Are you implying that an increase in "power" can never be achieved through increasing simplicity?
    - bitwize 9 years ago
      That's the thing. Just by glancing at the API docs, Windows looks more complicated but where the rubber meets the road in terms of real high-performance application development, Windows is way simpler. In Windows you can do in one syscall what would take several in Linux. You can schedule I/O calls across multiple threads in a completely thread-safe manner without having to manage the synchronization yourself -- and since threads go to sleep entirely while waiting for I/O operations to complete, there is no chewing up CPU cycles in a select/epoll loop. So yes, writing "hello world" or simple filters is simpler in Unix -- but writing multithreaded server applications that maximize throughput is simpler in Windows.
      Unix is bristling with features designed to "allow you to save me some time". It was designed to make it easy to write quick, "one-off" programs in C. VMS -- the predecessor to Windows NT -- was designed to run long-lasting, high-performance, high-reliability business applications for real users with money on the line (i.e., not just hackers) and Windows NT inherits this legacy.
  - zxcvcxz 9 years ago
    >the NT kernel is much more sophisticated and powerful than Linux
    Source?
    It's not sophisticated enough or powerful enough to be the most used kernel on super computers (and in the world). Windows pretty much only dominates the desktop market. Servers, super computers, mainframes, etc, mostly use Linux.
    A few years ago there was even a bug in Windows that caused degradation in network performance during multimedia playback that was directly connected with mechanisms employed by the Multimedia Class Scheduler Service (MMCSS), this is used on a lot of audio setups. If they can't even get audio setups right how can people consider anything Windows releases "sophisticated"?
    It's made to do anything you throw at it I guess, it's definitely complicated, but powerful and sophisticated aren't words I would use to describe NT.
    - recursive 9 years ago
      If you're arguing in favor of linux, you probably shouldn't use any arguments that deal with getting audio setups right.
    - Sanddancer 9 years ago
      They got audio setups right. The reason the network degradation happened is that video and audio playback were given realtime priority so background processes couldn't cause pops, stutters, etc. At the time Vista was released, most home users didn't have a gigabit network, so the performance degradation would only happen on a small number of users, and most would rather prefer good audio and video performance to a slowdown in network performance in a small percentage of users. With today's massively multicore systems, it's even less of an issue, while linux still has a problem with latency on applications like pro audio.
    - pbarnes_1 9 years ago
      I don't know why you're getting downvoted since the parent is basically stating random opinions about "power" and "sophistication" without anything to actually back it up.
      11 param functions don't say "power" to me. They say "poorly thought out API design". Much can be said for most Windows APIs in general.
- tptacek 9 years ago
  Overloaded system call entrypoints are a fact of life on all mainstream platforms. Consider for instance "ioctl".
  - marvy 9 years ago
    I've heard that Plan 9 doesn't have ioctl. But I guess that doesn't count as mainstream.
    - wahern 9 years ago
      Plan 9 replaces ioctl with special files that require writing magic incantations.
      It's similar to the various knobs in Linux /proc which require reading and writing specially formatted data. ioctl is simpler in that you don't need to worry as much about formatting the data (the struct declarations take care of that for you), but a file-oriented interface is nicer in that it's a higher-level abstraction--for example, it maps better to different languages, similar to how ioctl requires C or C-like shims whereas /proc can be used from any language that understands open/read/write/close, including the shell.
- deprave 9 years ago
  A lot of Microsoft APIs and subsystems are similarly bloated. There are probably tons of factors at play, but I believe being closed-source and having to support many individual use cases is one fundamental reason. (See for example CreateProcess vs. fork...)
  - bitwize 9 years ago
    When it comes to system call interfaces, it's because Dave Cutler has forgotten more than many modern "kernel hackers" will ever know about how to design an OS.
    - deprave 9 years ago
      I appreciate the name-dropping.
      Dave Cutler's skills aside, Unix predates Windows by decades, and to anyone remotely familiar with kernel development it is clear that the sheer quantity and complexity of subsystems stem from the fact that nobody but Microsoft can actually see, modify, and redistribute Windows' source code.
      Unless you can actually say "here's why Windows is qualitatively better" and point out specific tasks Windows does better, I'll just point you to the fact that the internet infrastructure and most of the servers on it, along with every Apple desktop and pretty much every mobile device, run Unix.
    - xorblurb 9 years ago
      Well, that's an (unbacked) opinion, and I don't share it. NT design is not too bad (obviously especially in contrast with Consumer Windows), and especially given what was achieved on the first few releases (that was made easier by Cutler serious experience in the area), but now it is far from brillant, and it has it (huge) share of problems every serious users of both Windows and Unix based OS knows.
      Now at one point, way in the past, NT was far above Linux, and some Linux fanboys existed that did not even knew what they were talking about, yet had strong opinions of superiority about the kernel they used. Now we are ironically in the opposite situation: Linux has basically caught up on all the things that matters (preemptive kernel, stability, versatility, scalability) and then quickly overtook NT, yet some people like to talk endlessly about the supposed architectural superiority of NT, that did not provide anything concrete in the real world in the long term and widely used, and that MS had to work around and/or redo with an other approach (while keeping vestigial of all the old ones) to do all its modern stuff.
      What kernel hackers know to do, is to detect problem in architecture that look neat on paper. Brillant ones are able to anticipate. I don't even have to: history has shown were NT has been hold back by its original design.
    - trentnelson 9 years ago
      You, I like you. You get it.
    - 9 years ago
luchs 9 years ago
>As of this article, lxss.sys has ~235 of the Linux syscalls implemented with varying level of support.
Is there a list of these syscalls somewhere? It would be cool to check it against the recent Linux API compatibility paper [0, 1].
[0]: http://oscar.cs.stonybrook.edu/api-compat-study/ [1]: http://www.oscar.cs.stonybrook.edu/papers/files/syspop16.pdf
- besselheim 9 years ago
  You piqued my curiosity - just made one by extracting the syscall dispatch table from lxcore.sys and placing it alongside the Linux syscall list: https://goo.gl/QHGe1U
  A lot of coverage there, but interesting to see which ones aren't yet implemented, at least in the recent build 14342.
  (I used Filippo Valsorda's work from https://filippo.io/linux-syscall-table as the Linux syscall data source.)
- xorblurb 9 years ago
  A list of (at least partly) supported syscalls is here: https://msdn.microsoft.com/en-us/commandline/wsl/release_not...
  Not details on which one are fully or partly supported, though.
Maarten88 9 years ago
I have installed the current fast ring build and have tried installing several packages on Windows. Some do install and work (compilers, build environment, node, redis server), but packages that use more advanced socket options (such as Ethereum) or that configure a deamon (most databases), still end with an error. Compatibility is improving with every new build, and you can ditch/reset the whole Linux environment on Windows with a single command, which is nice for testing.
- _khhm 9 years ago
  They've said the initial intent is for developers to use it, not for running servers / etc (which is why they only target Windows 10 client and not Windows Server OSs).
  - ygjb-dupe 9 years ago
    There is "running servers" in production and there is "running servers" in dev.
    If I can't run the entire stack I use for dev under the subsystem then I will go the other route, which is to continue using VMs. I am excited about the initial release, and the prospect of being able to use Windows for all of the regular things I do, but it's clear that this isn't ready for primetime even as a dev tool.
  - stuaxo 9 years ago
    Yup, when I'm developing I need to run pretty much most stuff. I guess, I can install say postgres using the windows native version, but then we are back at square zero.
    - Maarten88 9 years ago
      Installing postgres on lxss still ends in a 'syscall not implemented' error.
caf 9 years ago
Since NT syscalls follow the x64 calling convention, the kernel does not need to save off volatile registers since that was handled by the compiler emitting instructions before the syscall to save off any volatile registers that needed to be preserved.
Say what? The NT kernel doesn't restore caller-saved registers at syscall exit? This seems extraordinary, because unless it either restores them or zaps them then it will be in danger of leaking internal kernel values to userspace - and if it zaps them then it might as well save and restore them, so userspace won't need to.
- trentnelson 9 years ago
  I think that's referring to the prolog/epilog convention and "homing" of parameter registers, e.g.
  Frame struct ReturnAddress dq ? HomeRcx dq ? HomeRdx dq ? HomeR8 dq ? HomeR9 dq ? Frame ends
```
    NESTED_ENTRY Foo, _TEXT$00

    mov Frame.HomeRcx[rsp], rcx
    mov Frame.HomeRdx[rsp], rcd
    mov Frame.HomeR8[rsp], r8
    mov Frame.HomeR9[rsp], r9

    alloc_stack 64

    END_PROLOG
    
    ; *do stuff*

    BEGIN_EPILOG

    add rsp, 64

    NESTED_END Foo, _TEXT$00
```
  https://msdn.microsoft.com/en-us/library/tawsa7cb.aspx
emcrazyone 9 years ago
I can't think of much that would benefit from this except for, perhaps, headless command line type applications. The one that comes to mind is rsync. Being able to compile the latest version/protocol of rsync on a Linux machine and then running the same binary on a Windows host would be nice but fun seems to end there plus with Cygwin, this is largely a no-brainer without M$ help.
What about applications that hook to X Windows or do things like opening the frame buffer device. I've got a messaging application that can be compiled for both Windows and Linux and depending on the OS, I compile a different transport layer. Under Linux heavy use of epoll is used which is very different than how NT handles Async I/O - especially with sockets. So my application's "transport driver" is either compiling an NT code base using WinSock & OVERLAPPED IO or a Linux code base using EPOLL and pthreads.
Over all it seems like a nice to have but I'm struggling to extract any real benefit.
Can anyone offer up some real good use cases I may be overlooking?
- quux 9 years ago
  There are both free and commercial X servers for Windows, and you can get a linux app running under WSL to work with one of those X servers very easily. I played with it a little bit and it worked fine.
coverband 9 years ago
With this feature, if you're a Linux developer, you're automatically a Windows developer as well. Almost like being able to run all Android or iOS apps on Windows phones.[1][2]
[1] http://www.pcworld.com/article/3038652/windows/microsoft-kil... [2] https://developer.microsoft.com/en-us/windows/bridges/ios
Edit: Now I am puzzled as to why this got downvoted?
- besselheim 9 years ago
  If you disassemble lxcore.sys you can still see hints of the Android subsystem project that it grew from: the \Device\adss and /dev/adss devices, the application name Microsoft.Windows.Subsystem.Adss, various function names containing "Adss", and some other textual references to Android.
Animats 9 years ago
It's too bad that x86 hardware doesn't do virtualization as well as IBM hardware. You can't stack VMs. That's exactly what's needed here - a non-kernel VM that runs above NT but below the application.
- pmalynin 9 years ago
  You can have nested VMs.
  https://www.kernel.org/doc/Documentation/virtual/kvm/nested-...
  - overgryphon 9 years ago
    Windows also now supports nested virtualization.
    https://msdn.microsoft.com/en-us/virtualization/hyperv_on_wi...
- geofft 9 years ago
  I thought that a) the conclusion of VMware's "Comparison of techniques" paper [1] was that x86 and possibly everything is Popek-and-Goldberg-virtualizable [2] via binary translation, and b) the last several years of Intel and AMD chips all have hardware virtualization support, including nested virtualization, that made their architectures Popek-and-Goldberg-virtualizable in the obvious way?
  [1] https://www.vmware.com/pdf/asplos235_adams.pdf
  [2] https://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualizat...
- pjmlp 9 years ago
  Looking at the way mainframes work, with their higher level languages, JIT compilers at kernel level, object databases, type 1 hypervisors, ....
  It is quite interesting to see mainstream OSes increasingly get adopting all those features.
  - Animats 9 years ago
    Very, very slowly. Microprocessors still have DMA instead of mainframe-like "channels", although we're starting to see MMUs on the I/O side. With channels, devices can't blither all over memory and neither driver nor device need be trusted.
kevincox 9 years ago
> the Linux fork syscall has no documented equivalent for Windows
Emphasis is mine. I wonder if this is something that cygwin could (ab)use. Also I wonder why they would need this undocumented call.
- wfunction 9 years ago
  It's been tried and failed. See [1].
  [1] https://cygwin.com/ml/cygwin-developers/2011-04/msg00036.htm...
- bboreham 9 years ago
  > Also I wonder why they would need this undocumented call.
  To implement the first NT Posix subsystem, which was a FIPS requirement.
- xorblurb 9 years ago
  Cygwin is layered above Win32. Win32 has no provision to nicely handle forks. So even if there was an NT API fork syscall (I'm don't think there is on Windows 10, WSL does not use the NT API, there is not any more Posix/SFU/{Whatever Unix NT classic subsys of the day} as far as I know), this would not go anywhere.
  - pcwalton 9 years ago
    > So even if there was an NT API fork syscall
    You can do it with NtCreateProcess: https://groups.google.com/d/msg/microsoft.public.win32.progr...
    (The Win32 userland won't understand what you did, but you can still do it.)
    - xorblurb 9 years ago
      Well, you can do it on some versions of Windows. On Windows 10, and even future version of Windows 10, not so sure...
  - rossy 9 years ago
    Cygwin programs technically run under the Win32 subsystem, but they're not that cleanly layered. The runtime calls into a lot of Nt* functions, including undocumented ones. midipix (which another commenter mentioned) is another Unix-like environment for Windows that also runs under the Win32 subsystem, and apparently it has successfully implemented a real copy-on-write fork() on top of undocumented NT syscalls, so it's definitely possible.
- CUViper 9 years ago
  midipix is trying to use it, and advertises copy-on-write fork as an advantage over Cygwin, but I don't know how well it works yet. http://midipix.org/#sec-midipix http://midipix.org/git/cgit.cgi/ntapi/tree/src/process
bla2 9 years ago
Does anybody know how fork() is implemented? This blog post kind of sounds like fork() would do the slow emulation of it through CreateProcess().
- xorblurb 9 years ago
  fork() is properly implemented by the NT kernel. WSL is not layered above Win32.
obnauticus 9 years ago
Excellent post, Jack.
quux 9 years ago
Interesting, I wonder how much overhead is added to syscalls to look up the process type. Does NT still do this check when no WSL processes are running?
- stuaxo 9 years ago
  Pretty sure these are different entry points, so you wouldn't need to do anything different for normal Windows processes whether WSL is running or not.
  - quux 9 years ago
    I don't think so... both linux and windows binaries are using the same SYSCALL cpu instruction, and thus must be going to the same handler in the NT kernel.
_RPM 9 years ago
Does Microsoft document all system calls?
- detaro 9 years ago
  They document the WinAPI, but how that talks to the kernel is not documented. You can talk to it directly if you want, but there is nothing from Microsoft on how to do that. So if you see those as the true system calls, they are not documented at all.
  - xorblurb 9 years ago
    Well, tiny parts of the NT API (callable from userspace) are documented, but then often with the caveat that they are not stable (in practice, even some undocumented ones can be considered stable if used by enough programs in the wild, especially if they are simple and standalone and have no Win32 equivalent)
    The very precise mechanism, though, is extremely unstable. For example virtually every release of Windows (even sometimes SP) changes the syscall numbers. You have to go through the ntdll, which is kind of a more heavyweight version of the Linux VDSO. (The NTDLL approach was invented way before the VDSO, though)
    - therein 9 years ago
      Ntdll is similar to VDSO in the sense that it is loaded into the memory space of every userspace process. Even that I think might have exceptions on the Linux side. Either way, unlike VDSO, Ntdll actually does export functions potentially useful when called from the program. Here is an interesting read. http://undocumented.ntinternals.net/
9 years ago
davidgerard 9 years ago
Yes, yes, but can we run Wine on it?
negus 9 years ago
wtf is "pico process" and "pico driver"?
- wereHamster 9 years ago
  https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process...
prirun 9 years ago
Step 1: embrace
smegel 9 years ago
Funny they don't mention ioctl.
vegabook 9 years ago
Next step is Microsoft basically needs to turn Windows into a flavour of Linux. If they don't, they're under massive pincer threat from Android and Chrome, which are rapidly becoming the consumer endpoints of the future. Windows is about to "do an IBM" and throw away a market that it created. See PS/2 and OS/2.
They should probably just buy Canonical. That would put the shivers into Google, properly.
- mxuribe 9 years ago
  Funny years ago i would have reflexively flabbergasted at the thought of microsoft buying canonical (or any linux distro producer)...but actually thinking on that concept, and seeing recent (perhaps less-than-hostile) approach that microsoft has taken towards open source and linux, that wouldn't be a bad idea. I mean if microsoft could have both offerings - for windows servers and ubuntu-installed servers - i suppose that would be a very smart business move. Assuming they don't actually butcher or deny resources to whatever linux company they would buy, i could see several benefits - not only to microsoft but to developers, system integrators, etc. worldwide. Hey if a side benefit is that it would spur the market (a la google, apple, etc.) a little - to the benefit of us civilians - that's cool too.
  - orionblastar 9 years ago
    I think Microsoft should do what Apple did with BSD Unix aka Nextstep and merge it with their old OS.
    Microsoft should take the Windows GUI and put it over Linux as a desktop manager. Microsoft could sell the Windows GUI for Linux users that want to run Windows apps.
    - vegabook 9 years ago
      Could not agree more. Windows WM as an option on Linux is a clear and logical strategy.
  - vegabook 9 years ago
    I've been heavily downvoted for the view, but the facts are, there are hundreds of billions of dollars being spent in the Linux ecosystem, by corporations. Microsoft cannot afford not to be present in it. It's as simple as that. Canonical is starting to look like hitting Red Hat a bit on support contracts for corpos ets, so that's why I suggested that, but as you say, it could be another big and credible Linux distro (though Ubuntu all over the cloud must surely be tempting). Generally the idea that Microsoft wants to/must go big into Linux is uncontroversial, for me.
- 9 years ago
zxcvcxz 9 years ago
You should also mention that you're the author of Windows only software that directly exploits certain features specific to Windows. An implementation of python that according to http://pyparallel.org/ out-performs pretty much every other technology. Is that correct? I'm also wondering if anyone has replicated these results? From what I can find not a single person has replicated your tests.
According to your own website pretty much every other technology runs better on Linux than it does on windows (http://pyparallel.org/wrk-rps-comparison2.svg).
It could be said you're a little biased, seeing as your software only targets Windows.
- dang 9 years ago
  We detached this subthread from https://news.ycombinator.com/item?id=11866402 and marked it off-topic.
- trentnelson 9 years ago
  I'm incredibly biased. But that bias has come from assessing the technical details and concluding that NT really is superior, if that's any consolation.
  PyParallel flogs the Windows versions of things like Go, Node and Tornado because none of those were implemented in a way that allows the NT completion-oriented I/O facilities to be optimally exploited.
  It's depressing, honestly. In the sense that open source software never really comes close to taking advantage of NT because there are no such paradigms on UNIX. It's also complex as hell... I came from a UNIX background and completion ports were just a bizarre black box of overcomplicated engineering -- but after taking the time to understand the why, that was just a blub paradox reaction. And it's been a couple of years now of concerted study to really start appreciating the little details.
  - justinmk 9 years ago
    > things like Go, Node and Tornado [does not allow] the NT completion-oriented I/O facilities to be optimally exploited.
    nodejs is built on libuv[1], which uses IOCP on Windows (and epoll, kqueue, etc. on other platforms). What's non-optimal about it?
    [1] https://github.com/libuv/libuv
    - trentnelson 9 years ago
      "The things that bothers me about all the 'async I/O' libraries out there... is that the implementation -- single-threaded, non-blocking sockets, event loop, I/O multiplex via kqueue/epoll, is well suited to Linux/BSD/OSX, but there's nothing asynchronous about it, it's technically synchronous non-blocking I/O, and it's inherently single-threaded."
      https://speakerdeck.com/trent/pyparallel-how-we-removed-the-...
  - e12e 9 years ago
    Now I'm curious; have you played with reactOS at all? Do they implement the same VMS paradigm? That is, if given decent hw drivers for io on ReactOS, could one expect performance on the same order of magnitude as with a new nt-derived kernel?
    - trentnelson 9 years ago
      I haven't actually. Although now I'm kind of curious.
  - zxcvcxz 9 years ago
    So no one has replicated your findings and you're better than everyone else? K.
    edit:
    TBH it looks like you're making multiple accounts to comment on your own posts to make the comments that disagree with you appear lower.
    You and the child above mine had this same conversation 383 days ago:
    https://news.ycombinator.com/item?id=9584269
    This is pathetic.
    for posterity:
    https://web.archive.org/web/20160609010955/https://news.ycom...
    https://web.archive.org/web/20160609014033/https://news.ycom...
    - dang 9 years ago
      We've banned this account for repeatedly violating the HN guidelines.
    - trentnelson 9 years ago
      Well now I have no idea what you're talking about.
      I can barely remember the password to this hacker news account let alone managing multiple identities.
- Koromix 9 years ago
  I only have superficial experience with it, but from what I can tell there is a huge mismatch between IOCP and the UNIX readiness/poll model, and from my experience most server programs are written primarily for the latter.
  You need to design your server code somewhat differently to take advantage of IOCP. What many UNIX-like softwares do instead is bend IOCP or WaitForMultipleObjects() to behave like Linux. It works, but the performance is not there.
  Note that I haven't checked the code for any of the softwares in your chart so I could be wrong.
  - trentnelson 9 years ago
    Oh man, seeing UNIX people use WaitForMultipleObjects() as if it were select() (or, gasp, trying to use it for socket server stuff) just kills me.
    It's actually a really sophisticated multi-semaphore behind the scenes; but that sophistication comes at an inevitable performance penalty, because the wait blocks are in a linked list and it's generally expensive (in comparison to say, a completion packet being queued to a port) to satisfy that wait (the kernel needs to check lots of things and do locking and unlocking and linked list manipulation).
  - zxcvcxz 9 years ago
    Here's what I don't understand: If IOCP gives us such a great performance boost why don't I see people using it, even on Windows systems? The first thing that comes to mind is that IOCP can likely only maintain high performance under certain edge-cases that aren't pertinent to the real world, second is that the other I/O models are needed not for unix, but for other aspects of the language its self where IOCP is not appropriate. So IOCP likely causes overhead. I don't know much about it so maybe someone can explain. It sounds revolutionary if it can be applied to the real world and not just edge cases.
    - Koromix 9 years ago
      People are used to the POSIX readiness model, that's all. And you can kind of make it work on Windows, even though the performance is not great. OTOH, porting an IOCP-oriented application to Linux will give you catastrophic performance because you need to use many threads to replicate the async model on top of the synchronous Linux I/O calls.
      And for some reason, people get very defensive when you point out some of the advantages the NT kernel has over Linux. I mean, I use Linux, I don't like to use Windows but I have no problem admitting the IOCP model is better and async I/O on Linux is a sad broken mess. Denying it only serves to keep it that way.
    - politician 9 years ago
      IOCP is the top item on my (short) list of things Windows simply does better. The performance boost you see from designing a server for IOCP from the ground up is jaw-dropping. However, it's hard to grok because it's a very different model, most servers are proprietary code, and the MSDN documentation is barely sufficient (at least, back when I was writing IOCP-based servers in '09).
    - bitwize 9 years ago
      Inertia and the fact that select/epoll/kqueue loops are "good enough".
    - clevernickname 9 years ago
      The simplest explanation is that the performance benefit of using Windows and implementing just about any Windows-specific design is outweighed by the cost of the Windows licensing fees, when compared to a measurably worse-performing Linux or FreeBSD solution that costs nothing. So very few bother to treat Windows versions of "backend" software as anything but an afterthought.
- WayneBro 9 years ago
  I guess you should also mention on each of your comments that you are extremely biased against and paranoid about Microsoft and that you actively target any popular Microsoft article posted here in order to inject something negative, whether it's relevant or not.
  - trentnelson 9 years ago
    Heh.
zxcvcxz 9 years ago
I use to run Linux in a VM on windows and use Chocolatey for package management and cygwin and powershell etc, then I realized I was just trying to make Windows into Linux. Seems to be the way things are going and with the addition of the linux subsystem it kind of proves that Windows really isn't a good OS on it's own, especially not for developers.
I wish Windows/MS would abandon NT and just create a Linux distro. I don't know anyone who particularly likes NT and jamming multiple systems together seems like an awful idea.
Windows services and Linux services likely won't play nice together (think long file paths created by Linux services and other incompatibilities), for them to be 100% backward compatible they need to not only make Windows compatible with the things Linux outputs, but Linux compatible with the things windows services output, and to keep the Linux people from figuring out how to use Windows on Linux systems they'd need to make a lot of what they do closed source.
So I don't see a Linux+Windows setup being deployed for production. It's cool for developers, but even then you can't do much real world stuff that utilizes both windows and Linux. If you're only taking advantage of one system then whats the point of having two?
I went ahead and made the switch to Linux since I was trying to make Windows behave just like Linux.
- pcwalton 9 years ago
  > I wish Windows/MS would abandon NT and just create a Linux distro. I don't know anyone who particularly likes NT and jamming multiple systems together seems like an awful idea.
  I do. The NT kernel is pretty clean and well architected. (Yes, there are mistakes and cruft in it, but Unix has that in spades.) It's not "jamming multiple systems together"; an explicit design goal of the NT kernel was to support multiple userland APIs in a unified manner. Darwin is a much better example of a messy kernel, with Mach and FreeBSD mashed together in a way that neither was designed for.
  It's the Win32 API that is the real mess. Having a better officially supported API to talk to the NT kernel can only be a good thing, from my point of view.
  - xorblurb 9 years ago
    Well, large parts of the NT API are very close from Win32 API for obvious reasons, and so are often in the realm of dozen of params and even more crazy Ex functions. Internally there are redundancies that do not make much sense (like multiple versions of mutex or spinlock depending on which parts of kernel space use them, IIRC), and some whole picture aspects of Windows makes no sense at all given the architectural cost it induces (Winsock split in half between userspace and obviously needed kernel support is just completely utterly crazy, beyond repair, it makes so little sense you want to go back in past and explain the designer of that mess how stupid this is). The initial approach of NT subsystems was absolutely insane (hard dep on a NT API core, so can't do emulation with classic NT subsystems - so either limited to OS having some technical similarities like OS/2, or very small communities when doing a new target like the Posix or SFU was) -- WSL makes complete sense, though, but it is maybe a little late to the party. Classic NT subsystems are of so little use that MS did not even use them for their own Metro and then UWP things, even though they would like very hard to distinguish that more from Win32 and make the world consider Win32 as legacy. I've read the original paper motivating to put Posix in an NT subsystem, and it contained no real strong point, only repeated incantations that this will be better in an NT subsystem and worse if done otherwise (well for fork this is obvious, but the paper was not even focused on that), with none of the limitations I've explained above ever considered.
    Still considering the whole system, an instable user kernel interface has few advantages and tons of drawbacks. MS is extremely late to the chroot and then container party because of that (and let's remember that the core technology behind WSL emerged because they wanted to solve the chroot aside userspace system on their OS in the first place, NOT because they wanted to run Linux binaries) -- so yet another point why classic NT subsystems are useless.
    Back to core kernel stuff, IRQL model is shit. Does not make any sense when you consider what really happens, and you can't really use arbitrary multiple levels. It seems cute and clean and all of that, but Linux approach of top and bottom halves and kernel and user threads might seem messy but is actually far more usable. Another point: now everybody uses multiprocessor computers, but back in the day the multiple HAL were also a false good idea. MS recognize it now and only want to handle ACPI computers, even on ARM. Other OSes do all kind of computers... Cutler pretended to not like the "everything is a file" approach, but NT does basically the same thing with "everything is a handle". And soon enough, you hit exactly the same conceptual limitations (except not in the same places) that not everything is actually the same, so that cute abstraction leaks soon enough (well, it does in any OS).
    On a more result oriented approach, one of the things WSL makes clear is that file operations are very slow (just compare an exactly identical file heavy workload under WSL and then under a real Linux)
    So of course there are (probably) some good parts, like in any mainstream kernel, but there are also some quite dark corners, and I am not an expert about all architectural design of NT but I'm not a fan of the parts I know, and I strongly prefer the Linux way to do equivalent things.
    - wfunction 9 years ago
      > Cutler pretended to not like the "everything is a file" approach, but NT does basically the same thing with "everything is a handle". And soon enough, you hit exactly the same conceptual limitations (except not in the same places) that not everything is actually the same, so that cute abstraction leaks soon enough (well, it does in any OS).
      Explain? Pretty much the only thing you can do with a handle is to release it. That's very different from a file, which you can read, write, delete, modify, add metadata to, etc... handles aren't even an abstraction over anything, they're just a resource management mechanism.
    - JdeBP 9 years ago
      > IRQL model is shit. Does not make any sense when you consider what really happens,
      On the contrary. It's only when one considers what happens, especially in the local APIC world as opposed to the old 8259 world, that what the model is actually does finally make sense.
      * http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/ir...
  - tremon 9 years ago
    Having a better officially supported API to talk to the NT kernel can only be a good thing, from my point of view.
    That's particular interesting now that SQL Server has been ported to Linux. Would be funny if they're going to use the Linux subsystem on Windows too.
    Although I suspect SQL Server already talks to the kernel directly.
    - amaks 9 years ago
      No, they do have sophisticated user mode library but use only public kernel APIs. That user model library also helped them relatively painlessly migrate SQL Server to Linux.
  - pjmlp 9 years ago
    > Having a better officially supported API to talk to the NT kernel can only be a good thing, from my point of view.
    This is what I am looking forward to with WinRT, hence why Rust should make it as easy as C++/CX and C# to use those APIs. :)
  - zxcvcxz 9 years ago
    Well I've personally seen Microsoft employees themselves complaining about the state of NT while saying it's "fallen behind Linux".
    An old HN commenter once wrote (mrb)
    > There is not much discussion about Windows internals, not only because they are not shared, but also because quite frankly the Windows kernel evolves slower than the Linux kernel in terms of new algorithms implemented. For example it is almost certain that Microsoft never tested I/O schedulers, process schedulers, filesystem optimizations, TCP/IP stack tweaks for wireless networks, etc, as much as the Linux community did. One can tell just by seeing the sheer amount of intense competition and interest amongst Linux kernel developers to research all these areas.
    >The net result of that is a generally acknowledged fact that Windows is slower than Linux when running complex workloads that push network/disk/cpu scheduling to its limit: https://news.ycombinator.com/item?id=3368771 A really concrete and technical example is the network throughput in Windows Vista which is degraded when playing audio! https://blogs.technet.microsoft.com/markrussinovich/2007/08/...
    >Note: my post may sound I am freely bashing Windows, but I am not. This is the cold hard truth. Countless of multi-platform developers will attest to this, me included. I can't even remember the number of times I have written a multi-platform program in C or Java that always runs slower on Windows than on Linux, across dozens of different versions of Windows and Linux. The last time I troubleshooted a Windows performance issue, I found out it was the MFT of an NTFS filesystem was being fragmented; this to say I am generally regarded as the one guy in the company who can troubleshoot any issue, yet I acknowledge I can almost never get Windows to perform as good as, or better than Linux, when there is a performance discrepancy in the first place.
- 9 years ago
dragonbonheur 9 years ago
.
- geofft 9 years ago
  "As Brother Francis readily admitted, his mastery of pre-Deluge English was far from masterful yet. The way nouns could sometimes modify other nouns in that tongue had always been one of his weak points. In Latin, as in most simple dialects of the region, a construction like servus puer meant about the same thing as puer servus, and even in English slave boy meant boy slave. But there the similarity ended. He had finally learned that house cat did not mean cat house, and that a dative of purpose or possession, as in mihi amicus, was somehow conveyed by dog food or sentry box even without inflection. But what of a triple appositive like fallout survival shelter? Brother Francis shook his head."
  - stuaxo 9 years ago
    What is this from ?
    - geofft 9 years ago
      Walter Miller's A Canticle for Leibowitz.
- pionar 9 years ago
  It's a subsystem of Windows for running part of Linux, so Windows Subsystem for Linux. :)
- jcoffland 9 years ago
  This title would be clearer: Windows subsystem for Linux apps.
- coverband 9 years ago
  That's the output though, they're not changing anything in Linux. The "Windows subsystem" they're developing will act as the translator to get there.
- TazeTSchnitzel 9 years ago
  Linux Subsystem for Windows could also be misread the same way. Both are ambiguous.
  - kristopolous 9 years ago
    I initially thought this one was less ambiguous but I have to admit, I think Microsoft's phrasing is right. Let's try some substitution:
    "Russia Factory for England" most likely exists inside of Russia and is for the English.
    "John's mail for Sally [try: who is out of town]" even with the addition, I presume that John has authored mail for Sally and is not collecting the parcels to give to her.
    Here's a trickier one:
    "Sampsons' Dinner for Two". This could be the following:
    1. A product named "Sampsons' Dinner for Two" bought from a retail store
    2. An item "Dinner for Two" on a menu from a restaurant named "Sampsons"
    3. A place named "Sampsons' Dinner for Two" with only two-person tables.
    4. A product "Sampsons' Dinner" which comes in multiple sizes, one of them being designed for two people. (which is the ambiguous form - presuming there's also say Annie's Dinner for One/Two and Martha's Dinner for One/Two - each with a brand specific cuisine). Even here though, the ownership of which "Dinner for Two" product is still clear - it's the "Sampsons'" or "Martha's" brand.
    Regardless of what kind of substitution, we go back to "Windows Subsystem for Linux" for the most part parsing as
    "Windows [Subsystem for Linux]" like "Windows [Media Player]". I don't assume that it's "[Windows Media] Player" - as in some multi-platform software that is tasked with playing the proprietary windows media formats.
    It seems weird, but I think it's unarguably the right choice.
- jcoffland 9 years ago
  Maybe because the way they stated it it would be a much more attractive technology. Seems like an attempt to regaining ground in the server market.
  This would be really useful for distributing Windows apps as Linux binaries. It would make it easier to develop from Linux and target Windows. Need the same for OSX.
l3m0ndr0p 9 years ago
Pretty neat stuff. I think that MS should just create their own Linux Distribution & port all MS products. Get rid of the Windows NT Kernel. I believe it's outdated & doesn't have the same update cycle that the Linux Kernel has.
Why run a Linux Application/binary on a windows server OS? When you can just run it on Linux OS and get better performance & stability.
- jjtheblunt 9 years ago
  What makes you believe it's outdated?
  - zxcvcxz 9 years ago
    Can you show me the source so I can check?
    - UK-AL 9 years ago
      Actually there was a leak for 2000, most critics said it was surprisingly good.
- serge2k 9 years ago
  > Get rid of the Windows NT Kernel. I believe it's outdated & doesn't have the same update cycle that the Linux Kernel has.
  Curious why you claim this? What's outdated about the NT Kernel?
  - l3m0ndr0p 9 years ago
    Here are some, or maybe this is not part of the NT Kernel... 1. The use of drive letters A-Z for file system access. 2. Creating symbolic links to files and folders, like you can in Unix/Linux. You have to set a setting somewhere to enable this, but there's a security risk. 3. Standard functional/usable non-gui terminal application like Unix/Linux ssh. PowerShell doesn't come close. 4. Ability to SUDO or su Admin like Unix/Linux. Maybe these are not kernel related above, but the OS specific layer.
    - jdmichal 9 years ago
      1. The use of a multi-root hierarchy vs. a single root hierarchy is pretty arbitrary. Drive letters in turn are just an arbitrary way to define the multi-root hierarchy.
      2. `mklink` [0] has existed since Windows Vista for NTFS file system. No settings toggling required.
      3. What is your argument against PowerShell? In what ways does it fall short? I have been pretty successful with using it for various tasks.
      4. This is about the only legit claim. Windows always requires full credentials to execute as another user. Windows does provide `runas.exe`, but you must provide the target user's full credentials.
      [0] https://technet.microsoft.com/en-us/library/cc753194%28v=ws....
    - asveikau 9 years ago
      > 1. The use of drive letters A-Z for file system access.
      NT has a root directory like Unix does. Drive letters are symbolic links inside a directory called \DosDevices.
      Granted, this is not user-visible but an implementation detail. The needs of Win32 applications dictate a lot of user-visible behavior.
      > 2. Creating symbolic links to files and folders,
      NT supports symbolic links. Open cmd and type "mklink".
    - slededit 9 years ago
      You are confusing the win32 subsystem with the NT kernel. They are not the same, the win32 layer acts as a translation. Also symbolic and hard links are supported by NTFS, they are just not exposed in the UI. There are utilities to create them if you really want to.
      The shell itself and the rest of userland has very little to do with the kernel. It seems its the userland you are upset with. Swapping out the kernel won't fix that.
    - sz4kerto 9 years ago
      None of these are kernel-related.
      You can create hard and soft links. PowerShell is great, just different. UAC is not sudo, but works very well. It's a different OS.
    - bigger_cheese 9 years ago
      My biggest pet Peeve about Windows is the way it accesses files I'm not sure if this is a kernel or filesystem issue. But when a remote user has a file open as long as that file is open other users are prevented from updating or replacing the file. It happens all the time at my work and I know of no obvious way to work out who has the file open because as far as I can tell nothing like lsof exists.
      This is probably the number one cause of me banging my head against the desk and wishing Windows behaved more like Linux.
    - 9 years ago
    - wfunction 9 years ago
      > or maybe this is not part of the NT Kernel
      > 1. The use of drive letters A-Z for file system access.
      Indeed it is not. The kernel sees a single root for the object namespace, not drive letters.
    - detaro 9 years ago
      2. Filesystem feature, supported by NTFS for quite a while in differen styles. It has a security policy because a lot of userland software doesn't know about them. I use them all the time and they work, even if you move things like /Users/.
      3. Userland issue. You can compile bash and an ssh server for Windows if you want. PowerShell is quite different, yes.
      4. exists, both in GUI and commandline.
    - UK-AL 9 years ago
      A-z drive letter is just a win32 thing.
      Symbolic links is supported by NTFS, just not exposed to normal users.
      That's just your opinion about powershell...
      UAT, and runas?
    - stordoff 9 years ago
      > 1. The use of drive letters A-Z for file system access.
      Why is this a problem? As a user, I've always preferred to have drive letters - it makes it immediately clear if, for example, I'm moving files between different physical drives.
    - yread 9 years ago
      4. Shift+Right click or CPAU http://www.joeware.net/freetools/tools/cpau/ for command line