Linux in Windows - Implementation

[Ace Pace]

So if you haven't heard, Microsoft created a way to run native Linux binaries on Windows.

I have to say it works pretty well, but that's not why I decided to post about it. Rather lets talk details.

I've been working this the past day, total of 3 hours of work. This is a quick snippet of how this is implemented, I'll hopefully post more over time. This isn't prettied up to be a blog post.

So let's start with something simple, what's running?


Right away we can see we're really running linux binaries (this can be verified using SHA1) and not some port, most of the standard programs are unaware they're running on Windows.

But still, how does stuff work? For example, libc?

So libc is still libc (I didn't check the hashs), which means syscalls are still expecting Linux conventions to work.



Now, a quick primer. syscall is an x64 mechanism to transition to kernel mode, used by every OS. It's a hardware mechanism meaning it can't differntiate on it's own between a regular "Windows" syscall and the fact we want to reach Linux on Windows code. So what happens?

Apperently, Microsoft is using it's Project Drawbridge to implement a "container OS" inside Windows. This should be the underpinning for the Windows Containers alternative, but it's being used here to implement "Linux inside Windows".
I'll skip the init code, but the syscall mechanism in Windows now does the following branch.




What happens when we run into this branch? We reach code called LxpSysDispatch, which draws from a large syscall table (312 syscalls, not sure which linux kernel version runs precisely 312 syscalls) and runs the appropiate function, in this case.



Inside we have totally normal Windows kernel mode code.




That's all for today. I've started working out how /dev and /prov are implemented and have some things to say about that, but I'll get back to homework.

[Ace Pace]

So what I've been doing this weekend has been setting up some basic infrastructure for further research.

Some fun anecdotes.

Most Windows debugging tools, including Windbg (the serious debugger) can't handle Linux processes yet.



So Windows can recognise that python is running and suspended, go Windows. However, it's missing something, because LiW thinks lighttpd is running.



So clearly something is odd with handling daemons. What is it? Ah



Due to how Windows displays things, something with no parent PID gets thrown into the butt end of the process list. In this view we also see Process Explorer (an internals oriented tool) can't really parse this stuff.

So this has all been very fun, but lets go back to code.
What I've been doing is abusing the little debug code snippets that look like this.



I wrote up a quick IDAPython script to match string names to the function name and wham. Gave names to 634 functions out of the 1864 functions in the driver. (Go me!)

What's next? There's some more debug strings floating around, but the next challenge is figuring out the VFS.

[Purple]

I am torn between being excited simply because this is nifty as heck and simply asking why.

[Ace Pace]

I am torn between being excited simply because this is nifty as heck and simply asking why.

Why am I doing this or why Microsoft is doing this?

[Ace Pace]

I am torn between being excited simply because this is nifty as heck and simply asking why.

Why am I doing this or why Microsoft is doing this?

Edit window passed.

If you're asking why I'm doing? It's a cross between professional curiosity (there's some stuff that doesn't add up), showing off to professional friends, a tiny chance to be the first to publish (if this forum counts, I'm definitely the first worldwide) and a challenge that is far easier than my homework.

If you're asking why Microsoft is doing this? Look no further to .net core open-source, the latest VS, SQLServer on Linux and other projects to understand MS is trying very hard to regain the developer crowd. Talking with friends, they're doing a very good job of it.

[Purple]

Edit window passed.

If you're asking why I'm doing? It's a cross between professional curiosity (there's some stuff that doesn't add up), showing off to professional friends, a tiny chance to be the first to publish (if this forum counts, I'm definitely the first worldwide) and a challenge that is far easier than my homework.

If you're asking why Microsoft is doing this? Look no further to .net core open-source, the latest VS, SQLServer on Linux and other projects to understand MS is trying very hard to regain the developer crowd. Talking with friends, they're doing a very good job of it.

I was asking about both. But since you've answered #1 so completely I'll just focus on #2. And to that from my point of view there are two things to say:
1. MS already has developers on their side. I mean really, let's be honest here. If you want to sell software it has to run on windows. And if your software has to run on windows it's just so much easier to develop on windows too. I should know.

2. I was not aware that there is a significant market share of software that is linux only and yet worth using on Windows. It was always my understanding that linux code is part port, part worse knockoff, part OS specific (drivers and such). I mean, I literally can't name one program that runs on linux only that I would miss on Windows.

[Ace Pace]

I was asking about both. But since you've answered #1 so completely I'll just focus on #2. And to that from my point of view there are two things to say:
1. MS already has developers on their side. I mean really, let's be honest here. If you want to sell software it has to run on windows. And if your software has to run on windows it's just so much easier to develop on windows too. I should know.

2. I was not aware that there is a significant market share of software that is linux only and yet worth using on Windows. It was always my understanding that linux code is part port, part worse knockoff, part OS specific (drivers and such). I mean, I literally can't name one program that runs on linux only that I would miss on Windows.

Neither are correct. Today if you want to make money developing commercial software(note: not line of business software) you're doing it on mobile or on the web. Outside of niche industries (graphics, manufacturing, whatever) the bulk of commercial activity is in web or mobile. Why? Far faster to deploy, easier to make money (SaaS instead of shrinkwrap) and sexier. The downside to MS? Neither are Windows platforms and so people aren't tied to any OS. And the easiest server OS is...you guess it, Linux. Mobile? You can't develop iOS in Windows (nor on Linux).
Add to this the growing Apple mindshare both in the U.S. and the EU, and you have a pretty bad position. So much that many young developers in the states don't even work with VS (MS claims they have to teach their new employees how to use it!).

The second part is somewhat correct, but not precisely. Apache, nginx and other servers run primarily on Linux. Ruby (which is far too popular for such a crap language) runs primarily on Linux and OSX. Android Studio runs better on OSX than on Windows. This piles up.

Are the majority of developers still working on Windows machines? Yes, for now. But they're deploying their code to the web, running on Linux machines, using non-MS platforms (python, ruby, node *puke*), attracted to developing for mobile which doesn't include Microsoft.

[Ace Pace]

Today's advancement (and probably last for the weekend), I finished scripting up the system call table.

The format for a system call entry is


Another few IDAPython scripts (I hate IDApython and IDC so much) and we have a nicely parsed system call table.



As is clearly seen, not all the system calls are implemented. Out of 312 calls in the syscall table, 191 are implemented. I'm too lazy to check which aren't but one of them is sendfile64, because lighttpd doesn't work.

[Purple]

Neither are correct. Today if you want to make money developing commercial software(note: not line of business software) you're doing it on mobile or on the web. Outside of niche industries (graphics, manufacturing, whatever) the bulk of commercial activity is in web or mobile. Why? Far faster to deploy, easier to make money (SaaS instead of shrinkwrap) and sexier. The downside to MS? Neither are Windows platforms and so people aren't tied to any OS. And the easiest server OS is...you guess it, Linux. Mobile? You can't develop iOS in Windows (nor on Linux).

Add to this the growing Apple mindshare both in the U.S. and the EU, and you have a pretty bad position. So much that many young developers in the states don't even work with VS (MS claims they have to teach their new employees how to use it!).

I know all that. But I was ignoring the two markets on account of this being a discussion pertaining to linux and Windows as opposed to one about the broader market. As in, my above post can be taken to start with an unspoken "when talking about the two..."

The second part is somewhat correct, but not precisely. Apache, nginx and other servers run primarily on Linux. Ruby (which is far too popular for such a crap language) runs primarily on Linux and OSX. Android Studio runs better on OSX than on Windows. This piles up.

Yea, but like server software and such has run on linux and will run on linux ad infinitum. MS isn't going to convince people to migrate.

Are the majority of developers still working on Windows machines? Yes, for now. But they're deploying their code to the web, running on Linux machines, using non-MS platforms (python, ruby, node *puke*), attracted to developing for mobile which doesn't include Microsoft.

That makes far too much sense in hindsight. I keep getting stuck in the desktop mentality for some reason.

[Ace Pace]

No oh so sexy screencaps today.

So filesystems in this ...thing are split into two types. Drive filesystems and plugin filesystems (not my names). Each has multiple implementations, for example the Drive subsystem has an implementation of /mnt/[Windows volumes]/blah and both the entire root FS. While the plugin filesystems includes implementations of /dev, /proc and all those fun special places.

Each of these is obviously implemented in lxcore.sys (I.E entirely kernel mode). That was most of today's session.
Purple, if you have specific questions on how stuff is implemented, ask away. I'm mostly working around questions from people I know, which is why some of my next RE sprint will be dedicated to finding the UNIX sockets code.

[Spyder]

Nice research.

How does having two file systems work? Is the plugin file system on a VHD?

On the subject of formatting, is it smart enough to recognise both *nix and windows style line breaks or do you need to make sure you're using a *nix text editor?

Tangent: what makes node puke worthy? Just curious as I'm thinking of doing a little dev work on the side and it seems to be what all the cool kids are into these days.

[Ace Pace]

Nice research.

Thanks, my workstation died and is currently open for surgery so no updates. Also, I might be commercializing some of this.

How does having two file systems work? Is the plugin file system on a VHD?

Not precisely. There is no VHD. It's all running on your normal Volumes. If we stick to Drive type filesystems (their term, not mine), we have something that emulates the linux / directory and sub-directories, mapping them to a root directory in Windows (IIRC %appdata%/lxss/rootfs). In addition, there is another file system that maps /mnt/[drive letter]/your drive.
Both in the end convert Unix style paths to NT paths and access the file system using NT idioms (NtCreateFile etc.) meaning security is kept.

On the subject of formatting, is it smart enough to recognise both *nix and windows style line breaks or do you need to make sure you're using a *nix text editor?

If you're opening files from the shell, it's a debian binary. So up to the binary.

Tangent: what makes node puke worthy? Just curious as I'm thinking of doing a little dev work on the side and it seems to be what all the cool kids are into these days.

1. The infamous javascript wat video. God what a sucky language.

2. Node JS is very nice for performance, for getting something up and working. It is not a good dev environment in terms of dependency chains (the left-pad events). It is not well documented. It is not well tested. There is no clear roadmap.

If you want to write a one-off app, yeah why not. If this is something that should last a while...Well I like Python. I can accept Ruby. If you want something with tool support, then even php. Just...not a cool fad that will hopefully die with the acceptance of asm.js.

[phongn]

The second part is somewhat correct, but not precisely. Apache, nginx and other servers run primarily on Linux. Ruby (which is far too popular for such a crap language) runs primarily on Linux and OSX. Android Studio runs better on OSX than on Windows. This piles up.

Yea, but like server software and such has run on linux and will run on linux ad infinitum. MS isn't going to convince people to migrate.

No, but they might be able to convince developers to develop and do initial testing on a Windows client machine before staging to a real Linux server. A lot of people do this on OS X.

1. The infamous javascript wat video. God what a sucky language.

Well, it was a fast hack to make the monkey dance, as it were. Netscape never, ever in their lives expected that people'd be building 'real' programs around it (and certainly not on the server!)

2. Node JS is very nice for performance, for getting something up and working. It is not a good dev environment in terms of dependency chains (the left-pad events). It is not well documented. It is not well tested. There is no clear roadmap.

So long as your program works in an event-based paradigm, anyways

[Spyder]

Thanks, my workstation died and is currently open for surgery so no updates. Also, I might be commercializing some of this.

Well it's definitely adding value, so yeah, go for it.

Not precisely. There is no VHD. It's all running on your normal Volumes. If we stick to Drive type filesystems (their term, not mine), we have something that emulates the linux / directory and sub-directories, mapping them to a root directory in Windows (IIRC %appdata%/lxss/rootfs). In addition, there is another file system that maps /mnt/[drive letter]/your drive.
Both in the end convert Unix style paths to NT paths and access the file system using NT idioms (NtCreateFile etc.) meaning security is kept.

With two filesystems pointing at the same volume, does it avoid recursing? e.g C:\Users\username\Application Data\lxss\rootfs\mnt\c\Users\username\Application Data\...

If you're opening files from the shell, it's a debian binary. So up to the binary.

I was thinking, if the whole point is to attract devs to the windows platform then I'd guess Visual Studio would be part of the plan? If Visual Studio broke linebreaks because of bad text formatting then that could be a dealbreaker for some.

1. The infamous javascript wat video. God what a sucky language.

2. Node JS is very nice for performance, for getting something up and working. It is not a good dev environment in terms of dependency chains (the left-pad events). It is not well documented. It is not well tested. There is no clear roadmap.

If you want to write a one-off app, yeah why not. If this is something that should last a while...Well I like Python. I can accept Ruby. If you want something with tool support, then even php. Just...not a cool fad that will hopefully die with the acceptance of asm.js.

Thanks, will think about it.

[JLTucker]

Ace, I thought Apple open sourced Swift 2 for Linux. I dual boot Ubuntu and thought of developing on it.

[Ace Pace]

Ace, I thought Apple open sourced Swift 2 for Linux. I dual boot Ubuntu and thought of developing on it.

Yes, so?

[JLTucker]

Ace, I thought Apple open sourced Swift 2 for Linux. I dual boot Ubuntu and thought of developing on it.

Yes, so?

So is it possible to develop iOS apps on Linux? In another post you said you couldn't. If we can't, what was the point of open sourcing Swift 2?

[Ace Pace]

Ace, I thought Apple open sourced Swift 2 for Linux. I dual boot Ubuntu and thought of developing on it.

Yes, so?

So is it possible to develop iOS apps on Linux? In another post you said you couldn't. If we can't, what was the point of open sourcing Swift 2?

I'll save myself a lot of typing and reference Ars Technica. The gist is, mindshare and trying to get rid of crappy languages. Similar to the reason MS open sourced .net.

[Purple]

Out of curiosity what's with the Javascript hate? I mean sure it ain't exactly the swiftest language out there but it's far from bad.