Google Chrome Keystone Updater is removing the /var symlink causing account and boot issues.

UPDATE: 10/03/19

Google has posted an update that they found the issue in the Keystone Update and fixed it. They are are pushing the updated version now. The push started last night and the new version should be 1.2.13.79 or 1.2.13.80.

Google has been able to reproduce the bug. Chromium bug – bugs.chromium.org/p/chromium/issues/detail?id=1007358

Google now is recommending that you reinstall macOS to fix the issue. support.google.com/chrome/thread/15235262

Statement From AVID 9/25- avid.com/92419issue

See Index #7 for the Updated Fix from Google Support

#Varsectomy FAQ- Index

1. What is going on here? – #Varsectomy

2. Which macOS Versions are affected ?

3. Specific Mac Hardware ?

4. SIP – System Integrity Protection

5. AVID or Google Chrome Keystone Update causing this issue ?

6. How can I check my /var symlink?

7. How can I fix the issue = 2 ways to fix.

8. What about Hackintosh users? A fix for you!

9. Should I disable Chrome Auto Updates

10. Jamf Pro Extension Atribute

11 . Links

. 12. Credit – Investigation by #MacAdmins & Apple

1. What is going on here ? – #Varsectomy

The purpose of this article is to gather all known information and data around the current issues some AVID and NON SIP protected Macs are experiencing. The issue first started on Monday 9/23/19.

Late Yesterday some #MacAdmins started to report that a few of their systems would not boot properly. They would have the following issues.

1. After rebooting the affected system it would Kernel Panic. The system will reboot only to KP again

2. User Logs out and the system shows the Setup Assistant.

3. The System Kernel Panics into a boot Loop.

AVID users were some of the first to report the issue. AVID.com says

Avid empowers media creators with innovative technology and collaborative tools to entertain, inform, educate and enlighten the world.

Sometimes AVID Media Creators use 3rd Party Graphics cards connected to their Mac Pro. When the issue hit yesterday, it was thought that AVID was the main cause of the problems since all the users experiencing the issue had AVID software.

Only later after a MacAdmins deep dive investigation was it found that AVID was NOT the cause of the problem. It was Google Chrome Keystone Updater!

Join the #Varsectomy channel in MacAdmins Chat for the latest info!

2. Which macOS Versions are affected ?

MacOS 10.9 – 10.14 Mojave

NOTE: macOS 10.9 & 10.10 so not have SIP. System Integrity Protection was introduced in macOS 10.11 El Capitan.

3. Specific Mac Hardware ?

At first it was first reported that the 2013 Mac Pro was affected, this is not true.

Now it seems the issue affects all Macs that have SIP (System Integrity Protection) Disabled or turned OFF

Check if SIP is enabled by running csrutil status

System Integrity Protection status: enabled.

System Integrity Protection status: disabled

4. SIP – System Integrity Protection

All Reports so far look to be from Macs that have SIP Disabled!

Please do not disable SIP, it was created to protect macOS from this very issue.

5. AVID or Google Chrome Keystone Update causing this issue ?

At first the issue seemed to be caused by Avid Media Composer. After further investigation it was found that AVID was not the problem! Some AVID users may need to DISABLE SIP for 3rd party Video Card support.

After investigation from some of the top minds in the MacAmins Slack Chat #varsectomy channel it was found that the Google Keystone Updater was at the heart of the issue.

The Google Chrome Keystone updater lives here (if set to only update for the user who installed Chrome) – ~/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/

If set to install for ALL users. (This also prompts for admin access for install) Thanks Jeff Johnson /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/

You can run the Google Chrome Keystone Updater Manually to kick off an update check. (NOTE: Google has disabled the updater for now!) see bugs.chromium.org/p/chromium/issues/detail?id=1007358

~/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/GoogleSoftwareUpdateAgent.app/Contents/MacOS/GoogleSoftwareUpdateAgent -runmode oneshot

Thank you eholtam for the VM investigation and Screenshots!

Google Chrome Keystone Update Policy Running.

After kicking off the update you can see the problem below.

“Found and deleted symlink at path /var”

Found and deleted symlink at path /var

6. How do I check my /var symlink?

Check to see if your /var symlink was modified by running the following command.

ls -ldO /var

You should get one of the following outputs. The first one below means that your /var volder is SIP protected (notice the restricted flag) and the proper sym link /var -> private/var

lrwxr-xr-x@ 1 root wheel restricted,hidden 11 Apr 1 2018 /var -> private/var

The next one means that your symlink is broken and the folder is NOT SIP Protected.

drwxr-xr-x 5 503 wheel - 170 Sep 24 14:37 /var

If you find /var in this condition you are affected! If you LOGOUT, SHUTDOWN OR RESTART your Mac will NOT Boot! You will need to boot into recovery, repair the /var symlink and reset the restricted flags. Fix below.

7. How can I fix the issue? 2 ways to fix.

UPDATE: 9/25 – Rich Trouton posted a great article showing you step by step how to fix both 10.9-10.10 & 10.11-10.14 systems – derflounder.wordpress.com/2019/09/25/google-keystone-update-breaks-macs-ability-to-boot-if-system-integrity-protection-is-disabled/

1st Fix from MacAdmins User Juest

First you have to fix the /var symlink so it shows /var -> private/var

Then you can remove the affected LaunchAgents

Boot to macOS Recovery by holding down Command R

Steps for 10.11 -10.14 SIP Supported Systems

1. Boot into Recovery

2. Launch Terminal

(only if you're on 10.11+ with SIP)

# chroot /Volumes/[affected install]

# mv var vv

# ln -s private/var var

# chflags -h restricted /var

# chflags -h hidden /var

# xattr -sw com.apple.rootless "" /var



Steps for 10.9-10.10 NON SIP Systems

10.10 or older:

only do symblink and the hidden chflag and

remove launch agents from

/Users/[affected user]/Library/LaunchAgents/

OR

/Library/LaunchAgents/

- com.google.keystone.agent.plist

- com.google.keystone.xpcservice.plist

# exit

(recommended) # csrutil enable (10.11+)

8. Reboot

UPDATE: 9/28 – 2nd Fix is from Google Support

Google is now recommending that you reinstall macOS if you had this issue.

https://support.google.com/chrome/thread/15235262

NOTE: The information below is from Google. Link Below.

chroot /Volumes/Macintosh\ HD #(Macintosh HD is the default) rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle mv var var_back #(Var may not exist, but this is fine) ln -sh private/var /var chflags -h restricted /var chflags -h hidden /var xattr -sw com.apple.rootless "" /var

NOTE: Commenter stockmind has noted that the GoogleSoftwareUpdate.Bundle could reside in the USERS folder. In that case you need to run this string instead. Change USERNAME to your Username. Thanks for the tip!

rm -rf /Users/USERNAME/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle

Official Fix from Google support.google.com/chrome/thread/15235262

news.ycombinator.com/item?id=21066472 & 9to5google.com/2019/09/24/avid-mac-crashing-google/

8. What about Hackintosh users? A fix for you!

I had a few Hackintosh users reach out for a fix. Good News! u/Fargo_Newb on Reddit has a fix for you. – reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/

9. Should I disable Chrome Auto Updates ?

You shouldn’t need to disable Chrome Auto updates at this time. From the Chromium bug report – bugs.chromium.org/p/chromium/issues/detail?id=1007358 Google has disabled Keystone updates for now until the issue is found and resolved.

If you would like to disable Google Chrome Keystone automatic updates, the article below will show you. applehelpwriter.com/2014/07/13/how-to-remove-googles-secret-update-software-from-your-mac/

10. Jamf Pro Extension Attribute

This will help you find machines that are in a state where the /var sys link is broken. Thanks @neilmartin83 & @rmanly

#!/bin/bash



if [[ -h /var ]]; then

echo "<result>symlink</result>"

elif [[ -d /var ]]; then

echo "<result>directory</result>"

fi



exit 0

symlink = System OK!

11. Links

Below are a few of of the reports. It first started out as “AVID Hollywood Mac Pro boot problem“.

Jeff Rosica AVID CEO statement early on 9/24.

Chromium Bug Report 9/25 – https://bugs.chromium.org/p/chromium/issues/detail?id=1007358

Google Official Response 9/25 & 9/27 – support.google.com/chrome/thread/15235262

Official AVID Statement 9/25 – avid.com/92419issue

Rich Trouton step by step fix guide 9/25 –derflounder.wordpress.com/2019/09/25/google-keystone-update-breaks-macs-ability-to-boot-if-system-integrity-protection-is-disabled/

Scott Simmons 9/24 – provideocoalition.com/avid-editors-editors-in-general-be-on-alert-about-this-potential-rebooting-issue/

SlashDot Report 9/26 – https://it.slashdot.org/story/19/09/25/2152236/mysterious-mac-pro-shutdowns-likely-caused-by-chrome-update

Ars Technica Report 9/26 – arstechnica.com/information-technology/2019/09/no-it-wasnt-a-virus-it-was-chrome-that-stopped-macs-from-booting/

Daring FireBall by John Gruber 9/26 – daringfireball.net/linked/2019/09/25/mr-macintosh-chrome-updater

Forbes Report 9/26 – forbes.com/sites/kateoflahertyuk/2019/09/26/google-confirms-buggy-chrome-update-is-breaking-apple-macs/#5aad7747391c

Hacker News Discussion Thread 9/25- news.ycombinator.com/item?id=21064663

Gizmodo.com Report 9/25 – gizmodo.com/whoops-google-says-mysterious-wave-of-unbootable-macs-1838430057

Hackintosh Fix 9/25 – reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/

Updated Report from 9to5google.com 9/25 – 9to5google.com/2019/09/24/avid-mac-crashing-google/

Variety Updated Report 9/25 – variety.com/2019/digital/news/mac-pro-avid-shutdowns-chrome-update-keystone-1203348549/

Variety Report 9/24- variety.com/2019/digital/news/avid-mac-pro-corrupted-hollywood-1203347033/

MacRumors Report 9/24 – macrumors.com/2019/09/24/hollywood-mac-pros-hit-by-avid-issue/

Bleeping Computer Report 9/24 –bleepingcomputer.com/news/software/avid-users-are-suddenly-finding-that-their-macs-won-t-boot/

Some of the first reports on the AVID FaceBook Group 9/24 – https://www.facebook.com/photo.php?fbid=10156759822188022&set=p.10156759822188022&type=3&theater

Disable Google Chrome Keystone Automatic Updates – applehelpwriter.com/2014/07/13/how-to-remove-googles-secret-update-software-from-your-mac/

12. Credit! – Further investigation by #MacAdmins & Apple

It needs to be stated that many MacAdmins and Apple Engineers worked together and after investigating with users who had the issue. Great work guys!!!