Microsoft's Nerw "Recall" Feature Has Some Issues

[Zaune]

Charles Stross

Is Microsoft trying to commit suicide?

The breaking tech news this year has been the pervasive spread of "AI" (or rather, statistical modeling based on hidden layer neural networks) into everything. It's the latest hype bubble now that Cryptocurrencies are no longer the freshest sucker-bait in town, and the media (who these days are mostly stenographers recycling press releases) are screaming at every business in tech to add AI to their product.

Well, Apple and Intel and Microsoft were already in there, but evidently they weren't in there enough, so now we're into the silly season with Microsoft's announcement of CoPilot plus Recall, the product nobody wanted.

CoPilot+ is Microsoft's LLM-based add-on for Windows, sort of like 2000's Clippy the Talking Paperclip only with added hallucinations. Clippy was rule-based: a huge bundle of IF ... THEN statements hooked together like a 1980s Expert System to help users accomplish what Microsoft believed to be common tasks, but which turned out to be irritatingly unlike anything actual humans wanted to accomplish. Because CoPilot+ is purportedly trained on what users actually do, it looked plausible to someone in marketing at Microsoft that it could deliver on "help the users get stuff done". Unfortunately, human beings assume that LLMs are sentient and understand the questions they're asked, rather than being unthinking statistical models that cough up the highest probability answer-shaped object generated in response to any prompt, regardless of whether it's a truthful answer or not.

Anyway, CoPilot+ is also a play by Microsoft to sell Windows on ARM. Microsoft don't want to be entirely dependent on Intel, especially as Intel's share of the global microprocessor market is rapidly shrinking, so they've been trying to boost Windows on ARM to orbital velocity for a decade now. The new CoPilot+ branded PCs going on sale later this month are marketed as being suitable for AI (spot the sucker-bait there?) and have powerful new ARM processors from Qualcomm, which are pitched as "Macbook Air killers", largely because they're playing catch-up with Apple's M-series ARM-based processors in terms of processing power per watt and having an on-device coprocessor optimized for training neural networks.

Having built the hardware and the operating system Microsoft faces the inevitable question, why would a customer want this stuff? And being Microsoft, they took the first answer that bubbled up from their in-company echo chamber and pitched it at the market as a forced update to Windows 11. And the internet promptly exploded.

First, a word about Apple. Apple have been quietly adding AI features to macOS and iOS for the past several years. In fact, they got serious about AI in 2015, and every Apple Silicon processor they've released since 2016 has had a neural engine (an AI coprocessor) on board. Now that the older phones and laptops are hitting end of life, the most recent operating system releases are rolling out AI-based features. For example, there's on-device OCR for text embedded in any image. There's a language translation service for the OCR output, too. I can point my phone at a brochure or menu in a language I can't read, activate the camera, and immediately read a surprisingly good translation: this is an actually useful feature of AI. (The ability to tag all the photos in my Photos library with the names of people present in them, and to search for people, is likewise moderately useful: the jury is still out on the pet recognition, though.) So the Apple roll-out of AI has so far been uneventful and unobjectionable, with a focus on identifying things people want to do and making them easier.

Microsoft Recall is not that.

"Hey, wouldn't it be great if we could use AI in Windows to help our users see everything they've ever done on their computer?" Is a great pitch, and Recall kinda-sorta achieves this. But the implementation is soemthing rather different. Recall takes snapshots of all the windows on a Windows computer's screen (except the DRM'd media, because the MPAA must have their kilo of flesh) and saves them locally. The local part is good: the term for software that takes regular screenshots and saves them in the cloud is "part of a remote access trojan". It then OCRs any text in the images, and I believe also transcribes any speech, and saves the resulting output in an unencrypted SQLite database stored in:

C:\Users\$USER\AppData\Local\CoreAIPlatform.00\UKP{GUID}

And there are tools already out there to slurp through the database and see what's in it, such as TotalRecall.

Surprise! It turns out that the unencrypted database and the stored images may contain your user credentials and passwords. And other stuff. Got a porn habit? Congratulations, anyone with access to your user account can see what you've been seeing. Use a password manager like 1Password? Sorry, your 1Password passwords are probably visible via Recall, now.

Now, "unencrypted" is relative; the database is stored on a filesystem which should be encrypted using Microsoft's BitLocker. But anyone with credentials for your Microsoft account can decrypt it and poke around. Indeed, anyone with access to your PC, unlocked, has your entire world at their fingertips.

But this is an utter privacy shit-show. Victims of domestic abuse are at risk of their abuser trawling their PC for any signs that they're looking for help. Anyone who's fallen for a scam that gave criminals access to their PC is also completely at risk.

Worse: even if you don't use Recall, if you send an email or instant message to someone else who does then it will be OCRd and indexed via Recall: and preserved for posterity.

Now imagine the shit-show when this goes corporate.

And it turns out that Microsoft is pushing this feature into the latest update of Windows 11 for all compatible hardware and making it impossible to remove or disable, because that tactic has worked so well for them in the past at driving the uptake of new technologies that Microsoft wanted its ~~customers~~ victims to start using. Like, oh, Microsoft Internet Explorer back in 2001, and remember how well that worked out for them.

Suddenly every PC becomes a target for Discovery during legal proceedings. Lawyers can subpoena your Recall database and search it, no longer being limited to email but being able to search for terms that came up in Teams or Slack or Signal messages, and potentially verbally via Zoom or Skype if speech-to-text is included in Recall data.

It's a shit-show for any organization that handles medical records or has a duty of legal confidentiality; indeed, for any business that has to comply with GDPR (how does Recall handle the Right to be Forgotten? In a word: badly), or HIPAA in the US. This misfeature contravenes privacy law throughout the EU (and in the UK), and in healthcare organizations everywhere which has a medical right to privacy. About the only people whose privacy it doesn't infringe are the Hollywood studios and Netflix, which tells you something about the state of things.

Recall is already attracting the attention of data protection regulators; I suspect in its current form it's going to be dead on arrival, and those CoPilot+ PCs due to launch on June 18th are going to get a hurried overhaul. It's also going to be interesting to see what Apple does, or more importantly doesn't announce at WWDC next week, which is being trailed as the year when Apple goes all-in on AI.

More to the point, though, Windows Recall blows a hole under the waterline of Microsoft's trustworthiness. Microsoft "got serious" about security earlier this decade, around the time Steve Balmer stepped down as CEO, and managed to recover somwhat from having a reputation for taking a slapdash approach to its users data. But they've been going backwards since 2020, with dick moves like disabling auto-save to local files in Microsoft Word (your autosave data only autosaves to OneDrive), slurping all incoming email for accounts accessed via Microsoft Outlook into Microsoft's own cloud for AI training purposes (ask the Department of Justice how they feel about Microsoft potentially having access to the correspondence for all their investigations in progress), and now this. Recall undermines trust, and once an institution loses trust it's really hard to regain it.

Some commentators are snarking that Microsoft really really wants to make 2025 the year of Linux on the Desktop, and it's kind of hard to refute them right now.

On the plus side, unlike Twitter, at least they have the sense not to try and give their AI digital assistants Genuine People Personalities.

[bobalot]

Even Microsoft has jumped on the Enshitenication bandwagon.

[bilateralrope]

I've been following this for a while. Turns out that enough people complained and/or threatened legal action over the obvious problems that Microsoft has made it opt in:

Microsoft overhauls Recall, makes it opt-in

So I'm predicting that everyone who understands security will leave it off. But enough will turn it on that the inevitable security breach will still happen.

[Tribble]

I've been following this for a while. Turns out that enough people complained and/or threatened legal action over the obvious problems that Microsoft has made it opt in:

Microsoft overhauls Recall, makes it opt-in

So I'm predicting that everyone who understands security will leave it off. But enough will turn it on that the inevitable security breach will still happen.

“Opt in” my ass. I guarantee that it’ll find a way to send stuff out to M$ regardless of what you officially set it to.

At the very least I don’t think Windows 11 home users will be safe from intrusion, since those users are less likely to be large corporate/governments with sensitive information and big wallets to sue. Also, it’ll probably pester them to officially turn it on and/or some vague options in response to other features show up that an unaware user might click and that opts them in. Or commonly used features won’t work properly unless the user “opts in.” Oh and of course there will be updates which “somehow” accidentally set to to opt in by mistake! Really, the possibilities are endless.

I’ve been with M$ since DOS days, and I’ve seen it all before. However, this time they’ve gone to far- officially outright screenshotting and translating everything you do to send it to them for profit is a bit much even for me. I’m sure they’ve been doing it all along, but at least they used to claim it was a big, not a feature!

I’m sure we’ll hear plenty of stories of “I opted out of Recall yet it’s still storing everything and/or sending data to M$” soon enough.

[bilateralrope]

“Opt in” my ass. I guarantee that it’ll find a way to send stuff out to M$ regardless of what you officially set it to.

Microsoft has been very consistent in saying that the collected data would remain on the computer in question. If the EULA for using recall says otherwise, that's going to be a major PR problem for Microsoft. If the data started being sent to MS without them asking permission, that's going to violate a lot of privacy laws, including the GDPR.

There is no way Microsoft can have it sending data back without losing hard.

[Solauren]

There is no way Microsoft can have it sending data back without losing hard.

Especially if it starts doing that with Corporate or Government data.

[tezunegari]

Well, at least the database that stores all this information is heavily encrypted and protected, right?

As Hagenah points out:

“The database is unencrypted. It’s all plain text.”

TotalRecall can automatically find the Recall database on a person’s computer and make a copy of the file, for whatever date range you want. Pulling one day of screenshots from Recall, which stores its information in an SQLite database, took two seconds at most, according to Hagenah. Once TotalRecall has been deployed, it is possible to generate a summary about the data or search for specific terms in the database.

[bilateralrope]

It sounds like Recall was developed in secret. Skipping the normal testing process at Microsoft. Hidden from MS employees who could have pointed out the problems privately.

So I wonder what other problems will be discovered with it.

Microsoft delays Recall again, won’t debut it with new Copilot+ PCs after all

Recall will go through Windows Insider pipeline like any other Windows feature.
ANDREW CUNNINGHAM - 6/14/2024, 2:40 PM

Microsoft will be delaying its controversial Recall feature again, according to an updated blog post by Windows and Devices VP Pavan Davuluri. And when the feature does return "in the coming weeks," Davuluri writes, it will be as a preview available to PCs in the Windows Insider Program, the same public testing and validation pipeline that all other Windows features usually go through before being released to the general populace.

Recall is a new Windows 11 AI feature that will be available on PCs that meet the company's requirements for its "Copilot+ PC" program. Copilot+ PCs need at least 16GB of RAM, 256GB of storage, and a neural processing unit (NPU) capable of at least 40 trillion operations per second (TOPS). The first (and for a few months, only) PCs that will meet this requirement are all using Qualcomm's Snapdragon X Plus and X Elite Arm chips, with compatible Intel and AMD processors following later this year. Copilot+ PCs ship with other generative AI features too, but Recall's widely-publicized security problems have sucked most of the oxygen out of the room so far.

The Windows Insider preview of Recall will still require a PC that meets the Copilot+ requirements, though third-party scripts may be able to turn on Recall for PCs without the necessary hardware. We'll know more when Recall makes its reappearance.

Why Recall was recalled

Recall works by periodically capturing screenshots of your PC and saving them to disk, and scanning those screenshots with OCR to make a big searchable text database that can help you find anything you had previously viewed on your PC.

The main problem, as we confirmed with our own testing, was that all of this was saved to disk with no additional encryption or other protection, and was easily viewable and copyable by pretty much any user (or attacker) with access to the PC. Recall was also going to be enabled by default on Copilot+ PCs despite being a "preview," meaning that users who didn't touch the default settings were going to have all of this data recorded by default.

This was the version of Recall that was initially meant to ship out to reviewers this week on the first wave of Copilot+ PCs from Microsoft and other PC companies. After security researcher Kevin Beaumont publicized these security holes in that version of Recall, the company promised to add additional encryption and authentication protections and to disable Recall by default. These tweaks would have gone out as an update to the first shipments of Copilot+ PCs on June 18th (reviewers also wouldn't get systems before June 18th, a sign of how much Microsoft was rushing behind the scenes to implement these changes). Now Recall is being pushed back again.

A report from Windows Central claims that Recall was developed "in secret" and that it wasn't even distributed widely within Microsoft before it was announced. which could explain why these security issues weren't flagged and fixed before the feature showed up in a publicly-available version of Windows.

Microsoft's Recall delay follows Microsoft President Brad Smith's testimony to Congress during a House Committee on Homeland Security hearing about the company's "cascade of security failures" in recent months. Among other things, Smith said that Microsoft would commit to prioritizing security issues over new AI-powered features as part of the company's recently announced Secure Future Initiative (SFI). Microsoft has also hired additional security personnel and tied executive pay to meeting security goals.

"If you’re faced with the tradeoff between security and another priority, your answer is clear: Do security," wrote Microsoft CEO Satya Nadella in an internal memo about the SFI announcement. "In some cases, this will mean prioritizing security above other things we do, such as releasing new features or providing ongoing support for legacy systems."

Recall has managed to tie together all the big Windows and Microsoft stories from the last year or two: the company's all-consuming push to quickly release generative AI features, its security failures and subsequent promises to do better, and the general degradation of the Windows 11 user interface with unwanted apps, ads, reminders, account sign-in requirements, and other cruft.

[bobalot]

I'm struggling to understand why so many companies have decided to emulate the Boeing school of management.

[Solauren]

I'm struggling to understand why they think that companies will want this in the software at all? Network bandwidth is a real concern, especially with more people working 'remotely/at home'

[bilateralrope]

I'm struggling to understand why they think that companies will want this in the software at all? Network bandwidth is a real concern, especially with more people working 'remotely/at home'

Recall has so much wrong with it that we probably should stick to the criticism that we know are true. They are enough to make it too risky to use. The security concerns alone should kill it in the eyes of a competent IT department. Then we can start asking questions about how much power it uses or how it impacts SSD write endurance.

Microsoft is saying that it's purely local. Meaning nothing that uses network bandwidth. People noticed the unencrypted data it stores, so I think they would have noticed any network traffic it used.

[Eternal_Freedom]

Well I'm 98% sure if this Recall feature is auto-included that no government department/agency will ever adopt Windows 11, for two main reasons. 1: this is an absolute data/protection disaster, and 2: they'd have to provide new laptops/desktops that can actually run Windows 11, and there's no way they're doing that (especially in the UK).

[3-Body Problem]

Microsoft is saying that it's purely local. Meaning nothing that uses network bandwidth. People noticed the unencrypted data it stores, so I think they would have noticed any network traffic it used.

Local doesn't mean much in sn office where most computers are actually thin clients running a windows VM from a central server. This also means that recall data is being sent over the network as it's generated.

[Solauren]

Well I'm 98% sure if this Recall feature is auto-included that no government department/agency will ever adopt Windows 11, for two main reasons. 1: this is an absolute data/protection disaster, and 2: they'd have to provide new laptops/desktops that can actually run Windows 11, and there's no way they're doing that (especially in the UK).

I work for the CRA. If two pieces of information that can be used to identify an individual or legal entity are in the same file, they have to be encrypted. People have been fired for not following that.

[Eternal_Freedom]

It's the same with us in the Court Service. But being a UK government department I'm confident they'll find some way to fuck it up.

[bilateralrope]

Can you think of anyone for whom turning Recall on would be a good idea ?

[Mr Bean]

Can you think of anyone for whom turning Recall on would be a good idea ?

Intelligence agencies trying to spy on other people turning it on in a sneaky way? Cops turning it on to spy on suspects? Abusive boyfriends spying on loved ones? Controlling parents wanting to spy on their children?

About the only time I can find it being useful is if I as an IT professional wants to re-create a user error and don't have the user on hand.

[bilateralrope]

Microsoft will try the data-scraping Windows Recall feature again in October

Initial Recall preview was lambasted for obvious privacy and security failures.
Andrew Cunningham - 8/22/2024, 8:08 AM

Microsoft will begin sending a revised version of its controversial Recall feature to Windows Insider PCs beginning in October, according to an update published today to the company's original blog post about the Recall controversy. The company didn't elaborate further on specific changes it's making to Recall beyond what it already announced in June.

For those unfamiliar, Recall is a Windows service that runs in the background on compatible PCs, continuously taking screenshots of user activity, scanning those screenshots with optical character recognition (OCR), and saving the OCR text and the screenshots to a giant searchable database on your PC. The goal, according to Microsoft, is to help users retrace their steps and dig up information about things they had used their PCs to find or do in the past.

The problem was that other users on the same PC, or attackers with physical or remote access to your PC, could easily access, view, and export those screenshots and the OCR database since none of the information was encrypted at rest or protected in any substantive way.

Microsoft had planned to launch Recall as one of the flagship features of its Copilot+ PC launch in July, along with the new Qualcomm Snapdragon-powered Surface devices, but its rollout was bumped back and then paused entirely so that Recall could be reworked and then sent out to Windows Insiders for testing like most other Windows features are.

Among the changes Microsoft has said it will make: The database will be encrypted at rest and will require authentication (and periodic reauthentication) with Windows Hello before users will be allowed to access it. The feature will also be off by default, whereas the original plan was to turn it on by default and make users go into Settings to turn it off.

"Security continues to be our top priority and when Recall is available for Windows Insiders in October we will publish a blog with more details," reads today's update to Microsoft Windows and Devices Corporate Vice President Pavan Davuluri's blog post.

When the preview is released, Windows Insiders who want to test the Recall preview will need to do it on a PC that meets Microsoft's Copilot+ system requirements. Those include a processor with a neural processing unit (NPU) capable of at least 40 trillion operations per second (TOPS), 16GB of RAM, and 256GB of storage. The x86 builds of Windows for Intel and AMD processors don't currently support any Copilot+ features regardless of whether the PC meets those requirements, but that should change later this year.

That said, security researchers and reporters who found the holes in the original version of Recall could only find them because it was possible to enable them on unsupported PCs, just as it's possible to run Windows 11 on PCs that don't meet the system requirements. It's possible that users will figure out how to get Recall and other Copilot+ features running on unsupported PCs at some point, too.

Off by default. Until that gets changed in an update.

Still too worrying to use.

[bilateralrope]

Microsoft says its Recall uninstall option in Windows 11 is just a bug

[/i]Microsoft is working to fix Recall showing up as an optional Windows 11 feature.

By Tom Warren, a senior editor and author of Notepad, who has been covering all things Microsoft, PC, and tech for over 20 years.

Sep 2, 2024, 8:42 PM GMT+12[/i]

While the latest update to Windows 11 makes it look like the upcoming Recall feature can be easily removed by users, Microsoft tells us it’s just a bug and a fix is coming. Deskmodder spotted the change last week in the latest 24H2 version of Windows 11, with KB5041865 seemingly delivering the ability to uninstall Recall from the Windows Features section.

“We are aware of an issue where Recall is incorrectly listed as an option under the ‘Turn Windows features on or off’ dialog in Control Panel,” says Windows senior product manager Brandon LeBlanc in a statement to The Verge. “This will be fixed in an upcoming update.”

The controversial Recall AI feature, which creates screenshots of mostly everything you see or do on a computer, was originally supposed to debut with Copilot Plus PCs in June. Microsoft was forced to delay the feature after security researchers raised concerns. Microsoft says it remains on track to preview Recall with Windows Insiders on Copilot Plus PCs in October, after the company has had more time to make major changes to Recall.

Security researchers initially found that the Recall database that stores the snapshots of your computer every few seconds wasn’t encrypted, and malware could have potentially accessed the Recall feature. Microsoft is now making the AI-powered feature an opt-in experience instead of on by default, encrypting the database, and authenticating through Windows Hello.

We did ask Microsoft whether it will allow Windows users to fully uninstall Recall, as this appearance in the Windows features list suggests, but the company only confirmed this was just “incorrectly listed” for now. It’s possible that Microsoft may need to add a Recall uninstall option to EU copies of Windows 11 to comply with the European Commission’s Digital Markets Act. Microsoft has already had to add an uninstall option for Edge in European Economic Area (EEA) countries, alongside the ability to remove the Bing-powered web search in the Start menu.

That's an impressive bug for something that can't afford to have major bugs.

[bilateralrope]

Microsoft delays rollout of the Windows 11 Recall feature yet again

Microsoft works to make Recall "secure and trusted" after security complaints.

Andrew Cunningham – Nov 2, 2024 3:31 AM

When Microsoft launched its Copilot+ AI PC initiative over the summer, one of the flagship features was Recall, a feature that would log months' worth of your PC usage, with the stated goal of helping you remember things you did and find them again. But if you've heard of Recall, it's probably because of the problems that surfaced in preview builds of Windows before the feature could launch: It stored all of its data in plaintext, and it was relatively trivial for other users on the PC (or for malicious software) to access the database and screenshots, potentially exposing huge amounts of user data.

Microsoft was supposed to launch Recall over the summer but delayed the feature to rework it. The company went into detail on the new version of Recall's security protections in late September, declaring that a preview would be ready in time for Windows Insider Program testers in October. Now that we're past October, Microsoft has officially announced that the Recall preview is being delayed yet again and that it will begin rolling out to testers in December.

“We are committed to delivering a secure and trusted experience with Recall. To ensure we deliver on these important updates, we’re taking additional time to refine the experience before previewing it with Windows Insiders,” said Microsoft Windows Insider Senior Program Manager Brandon LeBlanc in a statement provided to The Verge.

LeBlanc didn't offer additional details on the latest Recall delay or make any new announcements about other security precautions Microsoft is taking with the feature. The company's September blog post detailed how data was being protected using Windows' Virtualization-Based Security (VBS) features and Windows Hello authentication and reiterated that Recall will be opt-in by default and that it will be fully removable for Windows users who aren't interested in using it.

When it does start to roll out, Recall will still require a Copilot+ PC, which gets some AI-related features not available to typical Windows 11 PCs. To meet the Copilot+ requirements, PCs must have at least 16GB of RAM and 256GB of storage, plus a neural processing unit (NPU) that can perform at least 40 trillion operations per second (TOPS). Users will also need their PCs to be enrolled in the Windows Insider Program; we have no idea when non-Windows Insider PCs will start getting Recall, though at this point, it seems likely it won't be until sometime in 2025.

It sounds like Microsoft are aware that it's still not ready for release.

It also occurs to me that, even if Recall works exactly as Microsoft have promised, it's not worth taking up that much storage space.

[Broomstick]

More software bloat.

Do I even want such a feature? No, I actually don't. I've had various PC's since the 1990's and in over 30 years I've never felt a need for this.

[bilateralrope]

The only people I've run into who want this on their computers are those who didn't understand the privacy problems that come with it.

Once Windows 10 reaches end of life next year, you're choices are going to be:
- Use an OS that isn't receiving security updates
- Windows 11. With Recall installed but not enabled. If you have hardware that can run it.
- Abandon windows.

Do you really expect sane decisions from a company where the CEO [url=https://www.straitstimes.com/business/c ... 63-to-104m]got a pay rise after asking for a pay cut[/ur] ?

[Mr Bean]

The only people I've run into who want this on their computers are those who didn't understand the privacy problems that come with it.

Once Windows 10 reaches end of life next year, you're choices are going to be:
- Use an OS that isn't receiving security updates
- Windows 11. With Recall installed but not enabled. If you have hardware that can run it.
- Abandon windows.

Do you really expect sane decisions from a company where the CEO got a pay rise after asking for a pay cut[/ur] ?

There is a third option Pay 31$ to get ... Microsoft

Yes it kicks the can down the road but that is an entire other year for Vista part 2 to die.

[bilateralrope]

There is a third option
Pay 31$ to get another year of support per Microsoft

Yes it kicks the can down the road but that is an entire other year for Vista part 2 to die.

I'm trying to think of people would actually benefit from taking that option. Maybe people who need that extra year to afford a replacement computer.

I suspect that a lot of people who take it will be thinking that Microsoft will extend support again after that year. I wouldn't want to be trying to buy a computer around the time they realize they are wrong.