Silkenweb Example: Hackernews Clone

File System Improvements to the Windows Subsystem for Linux

221 points by jackhammons 8 years ago | 102 comments

corysama 8 years ago
I haven't dug deep into it yet. But, I'll say that I've been having fun compiling (command line) Linux apps with Clang and visually debugging them with GDB all through the UI of VStudio Community Ed and without a VM. /shill mode
Linux is not a target for the code I'm working on. But, I like to keep my code as cross-platform as possible. Between this setup and prototyping code on several compilers via goldbolt.org, staying standard-compliant and compiler-agnostic is much easier than ever before.
- jhgorrell 8 years ago
  https://godbolt.org/ ? Goldbolt.org seems to be fax sw.
  - corysama 8 years ago
    Yep. Typo.
NelsonMinar 8 years ago
Network filesystems! That's the last big feature I've been needing for regular use of WSL for light tasks. There's still enough weird behavior I wouldn't look at WSL as a complete Linux replacement, but it sure is a lot nicer than Cygwin/msys2.
They don't say anything about FUSE, I wonder if this work is at all related?
- eVeechu7 8 years ago
  I am pretty sure this update does not add FUSE support. FUSE would add the ability to use SSHFS, which is the network file system support I'm interested in.
  If FUSE would be useful to you, please vote for it at https://wpdev.uservoice.com/forums/266908-command-prompt-con...
  - laumars 8 years ago
    There is a Windows-native SSHFS client[1]. I have it installed on a Win10 VM so can confirm it "works" albeit I haven't needed to use it for anything so cannot confirm how well it works.
    [1] https://github.com/Foreveryone-cz/win-sshfs I think is the currently maintained version but you're welcome to confirm that independently.
    - noen 8 years ago
      I helped resurrect this to support Windows 10 a year or so ago. It works, but because of its reliance on Dokany (a FUSE-like usermode file system library), there are a lot of ways it still doesn't work perfectly.
      It was a lifesaver for me when I was doing development on embedded Linux devices (Pi/Edison et al). But if you're not tied to remote hardware, WSL is an infinitely better, faster, much more reliable dev process.
      The biggest issue I ran into with win-sshfs was its tendency to bodge git repo status if you maintained the repo on the mapped folder.
    - NelsonMinar 8 years ago
      Any idea if filesystems mounted with that SSHFS client are now visible under WSL? They weren't in the original WSL release and I'm guessing aren't under the recent Creators Update release. But maybe they are with this new DrvFs?
    - abhirag 8 years ago
      Yes SSHFS support would be great, till then WinSCP(https://winscp.net/eng/download.php) helps reduce my troubles a bit.
simonh 8 years ago
WSL is really interesting. I switched to the Mac in 2006 but I’ve never really used the fact it’s built on Unix much because it’s so different and frankly outdated comparede to the Linux systems I work on. When I want to do ‘Unixy’ stuff at home I run a Linux VM in VirtualBox. But WSL seems to offer the best of both worlds – a solid mature desktop environment with plenty of well supported professional level apps and an approaching first class Linux environment baked right in. I’m still very happy on the Mac and would miss a lot of Mac specific features and applications, but this is the first thing to happen in Windows Land that’s made me even think of the possibility of wanting to run Windows at home for over 10 years.
Edit: What would be the nearest thing to WSL on a Mac? VirtualBox is a bit heavy-weight. Maybe something based on Hyperkit and Docker?
https://github.com/moby/hyperkit
https://docs.docker.com/docker-for-mac/
- ghshephard 8 years ago
  Likewise - except for me it's been 15 years. Switched over in 2001, and didn't look back. Running WSL creators edition is really a delight. It's not quite there yet - still has a year or two to bake - but the progress is great. They just added serial support yesterday (which is important for a bunch of the linux binaries I run). For the very first time I'm honestly considering sitting down to a Windows System to do work - I never thought I would have said that - I've got * nix in my finger DNA, so powershell, despite being incredibly powerful, just never managed to get into my workflow.
  But now - It's best of both worlds - solid User Environment + Solid Unix CLI. Even better, it's a linux CLI, which means I get all the apt-get goodness that I've been missing on OS X (which admittedly has a pretty good brew install infrastructure).
  - chx 8 years ago
    For me at least the big problem is malware. Running Arch Linux means I do not need to worry about malware. Ransomware. Crap like that. Yes, I am a reasonably experienced computer user and it's quite unlikely I would click anything "wrong" but not having to worry about a single bad click causing hours of recovery is a big plus.
    Edit: while I understand downvotes I would be much happier if you'd educate me where I am wrong.
    - noen 8 years ago
      If you avoid the darknet/warez and use Windows defender and Windows security essentials (MS antivirus), you will largely forget this was ever an issue with Windows.
      Even my parents, love them, haven't had a single piece of malware or a virus since I forced Windows 10 upon them (from Windows 7). They were not happy at first, but the number of calls I get for "why is my computer so slow" and "what is this thing that keeps popping up" have dropped to 0.
    - danieldk 8 years ago
      Running Arch Linux means I do not need to worry about malware. Ransomware. Crap like that.
      I didn't downvote you and understand your concern. However, keep in mind that X11 has no GUI isolation. So, an attacker could snoop all your keyboard and mouse events, make screengrabs, etc. through a compromised web browser.
      Wayland provides GUI isolation, but is not widely deployed yet. Most X11 users are only safe because the X11 desktop is not a large target group, but it is probably less secure than Windows, which does GUI isolation between processes of different privilege levels, sandboxing for Store apps, etc.
    - ghshephard 8 years ago
      Beats the heck out of me - you make a good point. Windows Systems are certainly more popular targets for Malware than ArchLinux/OpenBSD. I absolutely agree with you that if just want to not deal with that (and still actually use the Internet) - getting off the major platforms is a pretty straightforward play.
      But - if you want to use all the Tier 1 Commercial software (or, in the cases of things like garageband, free software ) - kind of hard not to have a Windows or Mac.
    - oblio 8 years ago
      All experienced Windows user have a curated list of software. If you download shady apps, sure, good chance of getting infected. But Chrome, Firefox, TotalCommander, 7-zip, Sublime, etc., etc. are all free, mature and malware free.
      It may not be as conceptually easy as Arch Linux but on the other hand you get access to Windows-only stuff which has its benefits.
      And almost any Linux user should be able to create that curated list within 6 months, IMO. If they aren't, then I doubt even their Linux chops...
    - lottin 8 years ago
      Malware, spyware built into the system and the fact that it's not open source, meaning the whole thing it's basically an unmaintainable black box.
    - 8 years ago
ungzd 8 years ago
Does it suffer from similar quirks as Cygwin (.exe suffix removal, symlinks, permissions mapping)?
- https://cygwin.com/cygwin-ug-net/using-specialnames.html
- https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlin...
- hermitdev 8 years ago
  In my experience, no. They only just recently added support for executing apps between environments.
  Executing Windows apps from the bash shell can be surprising:
  $ notepad The program 'notepad' is currently not installed. You can install it by typing: sudo apt install wine1.6
  $ notepad.exe (Note Pad is launched)
  $ /mnt/c/Windows/System32/notepad.exe (Note Pad is launched)
  In the latest builds symlinks no longer require administrator permissions.
  Don't know what you're referring to in regards to permissions mappings, but know that the Windows' permissions always trump whatever your see from chmod/ls -al
- sengork 8 years ago
  Cygwin (64bit) didn't work for me in the latest creators update (actually I think it broke even before that). Now I am trying things using WSL itself, so far it has been a productive move.
  File permissions and security descriptors proved to be quite a hassle when removing Cygwin requiring multiple takeown and cacls hacks to remove some of the files.
  On the flip side I saved some disk space too.
  - barrkel 8 years ago
    OTOH when I wanted to have pretty pictures on my lock screen but without the Microsoft ads, I needed to rely on Cygwin to blow away the folder that caches the pictures deep in ProgramData/.../SystemData/*/ReadOnly - the lock screen image setting API has a lot of bugs. Without Cygwin, I was fiddling with cacls and takeown in an elevated prompt trying to blow away the folder. An elevated Cygwin prompt and it was just rm -rf.
    (Bugs: https://www.google.co.uk/search?q=lock+screen+windows+10+ima...)
    - mkup 8 years ago
      Sometimes elevated command prompt is not powerful enough and you need command prompt running from "SYSTEM" user account (Windows NT equivalent of Unix "root"). You can get that one with psexec tool from Microsoft SysInternals: psexec.exe -i -s -d %WINDIR%\System32\cmd.exe
  - kevin_thibedeau 8 years ago
    "WSL isn't done until Cygwin doesn't run"
revelation 8 years ago
I wonder if they have plans to add other emulation features, most importantly the ability to passthrough USB devices (or even graphic cards?).
I don't need or want the Linux graphical interface of my VM, it's pretty much redundant, but I do need to make devices available to the Linux software. Otherwise it's a bit like emulating a microcontroller - sure, Turing says the thing can compute everything, but in a very practical sense it can do absolutely nothing without it's outside inputs and periphery. I need the damn side-effects!
- ajross 8 years ago
  Seems like you might be better served with a linux desktop and windows in a VM then.
  The use case for WSL right now is someone like a web developer. They can develop and debug node apps locally in their native environment.
  - gravelc 8 years ago
    Works great for bioinformatics too. I've exited the Apple world (high prices etc.), and don't miss much now.
    No longer have to modify Powershell to make it more Unix-like. Jupyter notebook works fine, so I'm happy. No more bother with dual booting either; Ubuntu is gone as well.
    - Stubb 8 years ago
      It's funny how Windows 10 is rapidly becoming a better Unix than MacOS since everything that matters runs Linux.
  - revelation 8 years ago
    That only works because they provide emulation for AF_INET, AF_UNIX and AF_NETLINK sockets. It's not an automatic freebie.
    - ajross 8 years ago
      I understand. Nonetheless they did to the work to enable straightforward client networking (which works because Windows' own networking layer is basically Berkeley sockets under the hood).
      Writing an emulation of the Linux USB sysfs and device node API on top of Windows' very different USB driver layer is a rather bigger task. I'm sure they'll get to it eventually, but it's not something you can demand as an "obvious" feature.
    - josteink 8 years ago
      > That only works because they provide emulation for AF_INET, AF_UNIX and AF_NETLINK sockets. It's not an automatic freebie.
      To be pedantic, Microsoft is actually emulating the whole lot. You're not running a Linux-kernel on your Windows-machine.
  - cryptonector 8 years ago
    Might a developer need to talk to something like a Yubico smartcard? Yes, yes they might.
- hermitdev 8 years ago
  They did recently add support for standard serial devices[1]. I'm not sure if this support extends to USB or what any such timeline looks like.
  [1] https://blogs.msdn.microsoft.com/wsl/2017/04/14/serial-suppo...
  - revelation 8 years ago
    It does seem to work with USB serial devices, but only because those drivers on windows in turn emulate a serial port.
    So we have the USB driver pretending to be a serial port to Windows pretending to be a TTY to the Linux subsystem.
    It's a bit silly, but then I imagine TTY is a whole lot easier to emulate (even with all the cruft on that) than a USB chipset.
    - hermitdev 8 years ago
      Tangential, but there was a recent HN article (I think) (that for the life of me I cannot find a link to) that referenced a Microsoft DSL for describing finite state machines that was principally developed to implement MS's USB driver. Closest link I could find (which is not the correct one) is: https://www.microsoft.com/en-us/research/project/model-based...
skrebbel 8 years ago
I wish there was a way to get lxss updates without being a Windows Insider.
I turned on Insider on my brand new laptop last summer and it immediately overwrote some vendor-supplied drivers by newer-but-broken ones. Result: no sound. I didn't manage to fix it except through a full reinstall. Now that's exactly the sort of problem you need to prepared for when doing the Insider thing, but it sortof ruins the point. I want to beta test lxss, not Paint 3D or some fancy new device driver release schedule.
- ghshephard 8 years ago
  I'm pretty certain I had WSL running on my system without being a member of Windows Insider. You just need to turn that on if you want to be on the Fast Track...
  - JorgeGT 8 years ago
    Parent said WSL updates. You can get WSL but in order to get updates like this one you need Insider builds.
    - ghshephard 8 years ago
      I thought this WSL update came with creators edition? Unless you mean the fast track WSL Updates, but not the other windows updates - which yes, I agree, would be nice.
nialv7 8 years ago
Windows subsystem for Linux? Or the Linux subsystem for Windows?
These names are confusing...
- masklinn 8 years ago
  It's a Windows subsystem[0] for [running] Linux [applications].
  The terminology is "Windows subsystem for [target environment]" so "Windows subsystem for OS/2" lets you run OS/2 application on NT, "Windows subsystem for Win32" lets you run Win32 applications on NT, and "Windows subsystem for Linux" lets you run Linux applications on NT.
  [0] "The interface between user mode applications and operating system kernel functions […] There are four main environment subsystems: the Win32 subsystem, an OS/2 subsystem, the Windows Subsystem for Linux and a POSIX subsystem."
  - joombaga 8 years ago
    Yet they chose different terminology for WoW64 (Windows 32-bit on Windows 64-bit), and chose to store 32-bit binaries in the WoW64 directory.
    - masklinn 8 years ago
      I don't know that Win64/WoW64 is a separate subsystem. I think it's a subset of the 64b version of the Win32 subsystem.
    - 8 years ago
  - signal11 8 years ago
    The Architecture of Windows NT page on Wikipedia has a good diagram to help visualize NT subsystems. The first few versions of NT, iirc, shipped with the OS/2 subsystem but this was removed with Windows XP.
    https://en.wikipedia.org/wiki/Architecture_of_Windows_NT
- skrebbel 8 years ago
  It's confusing, but accurate. It's a subsystem of Windows ("Subsystem" is a well-defined term in Windows land, the win32 API is o its own subsystem as well).
  I wish they had promoted the name lxss more, though. It's short, very googlable, and it's the internal name.
  The git repo name is even more ridiculous: https://github.com/Microsoft/BashOnWindows
  (after all, I had bash on windows for years already)
- eriknstr 8 years ago
  I suppose it might be named that way due to trademark conserns.
  Naming something as "X for Y", where "Y" is a trademark owned by someone else is often done, but I haven't seen "Y for X".
  Just speculating.
- zorked 8 years ago
  I was also confused by their ancient product "Windows Services for UNIX", which does not run on Unix.
  https://en.wikipedia.org/wiki/Windows_Services_for_UNIX
- Animats 8 years ago
  Microsoft also has an Android emulator for Windows.[1] One would think that would run into licensing problems.
  [1] https://www.visualstudio.com/vs/msft-android-emulator/
  - laumars 8 years ago
    Sorry but I'm not sure I understand your point?
    Android is Apache 2.0 with the Linux / GNU parts retaining their respective GPL licence. Microsoft are distributing the OS for free, which is allowed within the terms of Android's licencing and something that is done by many other organisations including Google themselves.
Filligree 8 years ago
What would be a good X11 server to use? Are there any that support the full stack? (shm, glx, etc.)
- gegenschall 8 years ago
  Haven't used LXSS in a while but my X11 of choice for Windows was VcXsrv: https://sourceforge.net/projects/vcxsrv/
  Just export $DISPLAY to :0 (AFAIK) and you're all set.
- jmspring 8 years ago
  I work for the company, mostly use a Mac, sometimes use X11, and started laughing when I read X11 server on WSL/Linux on Windows. Things have come a long way. I don't know if there is X11 support, but when I have some time later this week, I am going to look based off this question and see. I'm having visions of Hummingbird Exceed from back in the day at the moment.
  I'm not being flip, I'm actually curious as well and will try figuring out when I have a spare few moments.
- josteink 8 years ago
  > What would be a good X11 server to use? Are there any that support the full stack? (shm, glx, etc.)
  I recall using exodus in the past (at uni), but it's commercial (pricey). Not to mention these days it seems to be Mac only.
  From similar threads I've seen recently, and quick searching, it seems Xming is the preferred Windows-option these days:
  https://sourceforge.net/projects/xming/
  - saboot 8 years ago
    The free version of Xming does not implement several things properly. Not sure which but some applications haven't run properly. There is a charityware version but I can't find my license for it.
    MobaXterm is another free version, It's a term emulator with an X11 server, but you can just turn it on and ignore the term. It runs all the applications Ive found that the free Xming cant.
rebootthesystem 8 years ago
Is it at the point where one can, for example, use PyCharm to develop Python/Django sites residing in the Windowns file system (say, z:/sites/project_099) while hosting from WSL?
Or would it be treated more like a VM that you simply deploy to?
We currently run one or more Ubuntu server VM's per machine as needed for local testing. If needs go beyond that, we have physical (virtualized) servers on our network (for example, web and database servers).
I can see WSL possibly being good for local development if it can somehow integrate seamlessly with the Windows file system and various tools (PyCharm) can be setup to run on Windows yet talk to WSL. I guess this might be equivalent to running a remote interpreter.
Need to think about this a bit. Running a server or two on VMs is pretty clean and painless. On machines with 64 GB of memory you don't even know they are running (from a performance/resource perspective).
- WorldMaker 8 years ago
  You can cd to /mnt/z/sites/project_099 and work there directly in WSL. Open the same files in a Windows-based text editor. Even use Visual Studio to drive gdb debugging of the Linux process in WSL.
  I've not done that last bit, but I definitely have moved all my ruby/Jekyll work into WSL and use VSCode to edit.
mjevans 8 years ago
I was going to make another comment, but just found out while searching that WSL is //supposed// to be based on Ubuntu (20)16.04 now (instead of 14.04) I guess the next time I work with those systems I'll have to run down some upgrade / re-install directions.
I wonder if that will fix the issue where I was unable to mv folders around the c:\ drive within the /mnt/c FS. (It more or less locked up to the point that I had to restart to clear the issue.)
- WorldMaker 8 years ago
  It's mentioned in the release notes blog post: https://blogs.msdn.microsoft.com/commandline/2017/04/11/wind...
  You can do the traditional Ubuntu do-release-upgrade to upgrade your install, or the suggestion is to replace and reinstall (from PowerShell or cmd):
```
    lxrun /uninstall /full /y
    lxrun /install
```
  I copied my few dotfiles to a safe place and did that.
- JoBrad 8 years ago
  FWIW, I haven't had any issues moving files before or after upgrading from 14.04 -> 16.04. I use Git in WSL, but interact with the files in both Windows and WSL without any special considerations.
  - mjevans 8 years ago
    With the updated WSL version out I probably won't ever run across the issue again. I can't remember the exact circumstances, I may have had the folder open in a window or something (thinking of it, Windows doesn't let you rename files out from under processes, which may have been the reason).
    However I do recall that trying to ctrl+c kill it didn't work. I don't recall what else I had/hadn't tried; I was rather busy at the time and couldn't stop to make a proper bug report.
    All that mattered to me was that it completely stalled for no transparent reason and my best option was to just not even try to do real file management from the WSL environment.
neves 8 years ago
Last time I tried to use the Windows Subsystem for Linux it was a mess. I thought I could access my files from windows and messed all the filesystem.
Now it looks like I can freely exchange my files between the WSL and Windows. Is it true?
Do I finally have a linux inside a windows machine where I can install things using apt-get (instead of cygwin setup.exe)?
When will it goes into the official Windows 10 version?
- Stubb 8 years ago
  Your Windows drives appear under /mnt/c, /mnt/d, etc. in WSL. So your Windows home directory is /mnt/c/Users/ffoobar.
  You can easily read/write Windows files from Linux, but I don't think you can do the opposite. I've used git within WSL to drive a repo on my Windows drive.
  WSL gives you full-blown Ubuntu 16.04 where you manage packages with apt-get. You have to look close to the bare metal to see differences. Until the creator's update, for example, ifconfig didn't work.
  Earlier today, I compiled and ran some OpenMP-based code in WSL and it happily detected all the cores on my dev machine and kept them busy with worker threads. Pretty cool!
  - steveklabnik 8 years ago
    Previously, if you read Linux files from Windows, it would brick the lxss entirely.
    The bug report I filed suggested that this was being fixed soon.
  - neves 8 years ago
    I thought it would give me total interoperability. That I could edit an movie in a windows app, run it through ffmepg, and go back to editing.
    Cygwin is still useful.
    - wvenable 8 years ago
      It does but you have to do that from the Windows file system not the Linux one. So from Linux you are accessing /mnt/c/Users/you/somedocument and from Windows it's C:\Users\you\somedocument.
- jug 8 years ago
  I wonder about this too.
  I know that apt-get works since earlier instead of the terrible Cygwin distribution model, but yes, IIRC there are NTFS streams being used for the Linux file system metadata IIRC and if these get overwritten or somehow modified things go bad. And NTFS streams being a somewhat uncommon NTFS feature, some applications don't handle them well?
  On the other hand, wasn't a new Windows 10 Creators Update feature that you could now indeed edit the files from Windows, and not just from the Linux subsystem? And that the Linux tools such as web servers are now even notified when the files change so they can reload them automatically? I'm not sure how they solve the metadata borking issue though, in that case.
- bmon 8 years ago
  It's already on the stable Windows 10 build a, and there was also significant improvements in the creators update. You have to enable it via "Windows Features" and reboot before first use.
p2t2p 8 years ago
Java still doesn't work properly though =(. "mvn clean package" on one of corporate projects takes about 5 to 8 minutes in native java on Windows or Mac and it takes more than one hour in WSL. I had so much hope for WSL that I would finally have only ONE device (surface), but no, we're not there yet.
- wbkang 8 years ago
  Welp I mean, with Java, you at least have an option to compile from Windows (given that you can call out an Windows executable from WSL now)
- Traubenfuchs 8 years ago
  Five to eight minutes? That sounds horrible. Have you tried concurrent compilation with "mvn -T 1C"? Gets our build time down from 3 minutes to 1 minute, but we have lots of smaller projects.
  - p2t2p 8 years ago
    Well, that's a clean build of more 500 megs of source code and includes NPM building frontend and lots of other fancy plugins for generating docs and stuff.
- kristianp 8 years ago
  Sorry, I have to ask. Why do you need to use Java with WSL? Native methods?
  - p2t2p 8 years ago
    We have a build system which is built around maven with *nix only in mind. It does a lot of handy stuff and 99.999% people in company are on macs, I just wanted to have one device instead of two so I was trying to use surface. I keep trying it like once or twice a year and then back to my Mac + iPad + Apple Pen mainly because of that. I also tried to run Linux in hyper-v and use shared directories but io perforce is quite poor in this case too.
  - hiram112 8 years ago
    I would love this because I still want a 'real' shell when dealing with Java - things like emacs, grep, find, etc.
    Cygwin and using shared VM directories has too many glitches with things like npm, links, file system endings, etc. As Docker becomes more important, this gets even worse.
    Really, developing on a Mac is just so much better than both Windows and Linux, even for Java.
    - milesrout 8 years ago
      But why not just use Linux?
- nailer 8 years ago
  Have you reported it?
  - p2t2p 8 years ago
    Nope, somebody else did it so I just upvoted. It became better though, it used to be impossible to run Java 8 at all.
- anqh4 8 years ago
  NTFS is soooo slow when dealing with lots of small files.
emptyfile 8 years ago
With the Creators update, can you now actually run a postgres service or apache in the linux subsystem?
- ghshephard 8 years ago
  See: https://blogs.msdn.microsoft.com/wsl/2017/04/11/testing-the-...
  In particular, they run Nginx 1.4.6 with 100% test coverage and postgresql-9.5.3 with 100% test coverage.
  Oddly (very), they have a few projects, Cassandra, Coffescript that show better test pass rates with WSL than Native Ubuntu.
tkubacki 8 years ago
In a way Balmer was right -"Cancer" is eating Windows :)
- tempay 8 years ago
  I assume this is a reference to something?
  - eugeneionesco 8 years ago
    https://www.theregister.co.uk/2001/06/02/ballmer_linux_is_a_...
    Unbelievable how much M$ changed for better in the last years!
    - oblio 8 years ago
      16 years is quite a few "last years" :)
egl2018 8 years ago
Does WSL work on Windows Home, or does it require Pro? Is this an accident or by design?
- hungys 8 years ago
  The screenshots in official installation guide are Windows 10 Home edition.
  https://msdn.microsoft.com/zh-tw/commandline/wsl/install_gui...
- richard_todd 8 years ago
  Definitely works on Home. Source: the laptop I'm using as I type this.
urbanxs 8 years ago
Not this crap again
tkubacki 8 years ago
In a way Ballmer was right -"Cancer" is eating Windows ;)