Introduction

We, at Beame.io LTD, have been using G Suite for a few years now. At the moment of the story, we had 2FA enabled for all accounts except for one, which was stuck (because of our process) at setup phase. This account had no administrative privileges.

Pending meeting

On 2018-03-22 there is a scheduled remote meeting with big and important client for which we need to prepare. We estimate that the outcomes of this meeting will affect our partnership with the client. Most of the documents we need to deal in order to prepare to the meeting with are in Google Documents.

Involved people

In the events below we will mention employees A, B, C, and D. Employees A, C, and D have (had actually) admin accounts.

TL;DR

Google deleted all our G Suite data. All important data is recovered by now.

2018-03-18 – We got email notification about removal of G Suite. The notification went unnoticed because the domain in the subject and in the body of the email is a domain we do not use. The old domain (mentioned in the email) was primary G Suite domain. 2018-03-22 – Our G Suite is deleted because we did not respond to the email. 2018-03-27 – Support finally says that the data is unrecoverable. After more than an hour an email from another person in support gives us some hope. 2018-03-28 – We are requested to prove domain ownership. The data and accounts are either being restored or partially restored. 2018-03-28 – Restore is in progress. Most accounts are available. 2018-04-01 – The last (and the most important account) was recovered.

It took 5 days to tell us that we are completely f*cked. All our documents and emails are gone. The next day our stuff is being recovered.

Failure to respond to an email gets your G Suite deleted within days.

After recovering the data, Google failed to tell us that they have finished. We were contacted later.

Last update of this blog

2018-04-12 12:52 UTC+3

Timeline of events

2018-03-22 09:56 UTC+2

Message from A to the team – email account doesn’t work. B reports that he observes the same problem. It quickly becomes clear that we all can not access our accounts.

10:21 UTC+2

We have disproved our first theory – we accidentally did not pay for the service. Our Money guy says we paid.

Somewhen

C & D fail to find any obvious way to contact G Suite support except for Twitter.

Around 11:15 (twitter does not show exact times)

Conversation using PM with @gsuite .

D:

looks like our gsuite domain was deleted and we can’t log in but we can’t phone google support because it requires PIN … which you get when logged in

@gsuite:

Hi there, what is the domain name on the G Suite account? Please send a full screenshot of the error message you receive when you access http://admin.google.com from an incognito window https://support.google.com/chrome/answer/95464?co=GENIE.Platform%3DDesktop&hl=en …. -MO

D: pastes screenshots

The conversation continues in parallel to other things below.

11:42 UTC+2

Failing to get through phone support D opens new G Suite domain to get PIN to get some support.

Somewhen

D contacts G Suite support on the phone using the PIN from new G Suite domain.

11:49 UTC+2

Following phone conversation with the support, D gets email. The email does not contain the promised link to a form which D should fill.

11:53 UTC+2

D gets email with a link to account recovery form. D misses the email because he is talking to C, answers the previous email and tries to get to G Suite support via Twitter. D sees this email after a while, when a case using the form is already opened.

11:56 UTC+2

D answers in email (replying to email at 11:49):

I’ve seen that page before. It’s not clear which form you were referring to. I don’t understand how to proceed. Please advise.

12:39 UTC+2

After C and D both contact G Suite support via Twitter account, get link to the account recovery form and fill the form (at https://support.google.com/a/contact/recovery_form ), a case is opened.

Case text is:

Subject: [urgent] (CENSORED-DOMAIN) – Whole G Suite domain does not function We started getting “account deleted”

errors (started less than 24 hours ago) when trying to log in. Looks like

the whole G Suite domain was deleted. We suspect malicious activity. We did

not even consider deleting this domain. I am one of the administrators of

the domain – (CENSORED-EMAIL). Recovering the access and the data is critical

for our organization.”

The email says

Thanks for opening this G Suite support ticket. Your ticket number is (CENSORED-TICKET-NUMBER), and we’ll be in touch with you soon.

13:17 UTC+2

D checks what our clients see when they send us emails. Here is the reply they could get if sent something to one of our mail accounts:

The response was:

The email account that you tried to reach does not exist. Please try double-checking the recipient’s email address for typos or unnecessary spaces. Learn more at https://support.google.com/mail/?p=NoSuchUser (CENSORED-SOME-ID) – gsmtp

13:30 UTC+2

Email from G Suite support regarding the case:

Hello (CENSORED-NAME), Thank you for contacting G Suite Account Recovery team. I understand that you are having issue with your G Suite account associated to the domain CENSORED-DOMAIN and getting error that the account has been deleted. I would like to know if your G Suite account is associated to another G Suite account before? Please let me know by replying to this email. Thank you. Sincerely, (CENSORED-NAME)

G Suite Account Recovery team

13:34 UTC+2

Our reply to the email above:

We are not sure. If yes, it was (CENSORED-OLD-DOMAIN) . Since this is very urgent issue that is critical to our business, can we have a phone conversation with you?

I think it will be more productive. Regards,

(CENSORED-NAME)

15:46 UTC+2

D does following PM to @gsuite:

Hi. Sorry to bother you but it has been 2 hours since my last email to the support. The issue is very urgent to us. Can you check maybe what’s going on with case (CENSORED-TICKET-NUMBER)

16:08 UTC+2

@gsuite to D:

Hi there, the service level agreement for G Suite technical support is 24 hours. The team will reply within 24 hours from your response. I hope this helps. -MO

16:20 UTC+2

D to @gsuite:

Can we buy premium support to accelerate this? The impact of this issue on our business is huge.

This is unanswered at the time of publication.

22:50 UTC+2

Additional email to support:

It is disappointing that I was still not contacted, despite defining this issue as urgent. It’s evening, and I’m going to sleep soon. Since the issue is urgent, please call me ( CENSORED-PHONE-NUMBER ) because I will not see your emails till the morning; I’m at UTC+2.

22:59 UTC+2

Email from support (despite requesting phone communication):

Hello CENSORED-NAME, Good day! I hope this message finds you well. My name is CENSORED-NAME and I am one of the supervisors for G Suite Account Recovery. I understand that you are getting an error that the domain has been deleted when you try to sign in with your email account for CENSORED-DOMAIN. I’ll be taking over now of this case. Google sends notifications to the administrators, especially to the primary administrator, whenever there’s an update or changes that might affect their G Suite account. You mentioned that you are one of the administrators of the account which said was deleted. Have you coordinated with the other administrators and check if they receive any notifications from Google? Also, when the previous agent asked you if there was any other domain affiliated with the account, you said that it might be CENSORED-OLD-DOMAIN. Do you still own that domain? I would also like to ask if you remember signing up for G Suite, what was the domain you used? If you have questions, feel free to reply to this email. Sincerely, CENSORED-NAME

G Suite Account Recovery Team

23:06 UTC+2

Answered the email:

Thanks. It’s good when my emails are responded to. … They did not delete the account and they did not get any notifications. There were no plans to delete the G Suite domain. … We do not own CENSORED-OLD-DOMAIN anymore. Please note that we are not sure whether these were linked. … I’m not sure I understand the question. When I joined the company, G Suite was already in use. Our emails etc were on “CENSORED-DOMAIN” domain, which we still own.

2018-03-23 01:09 UTC+2

Email to support:

New information. C ( CENSORED-EMAILS ) did receive notification about pending removal of CENSORED-OLD-DOMAIN G Suite. He did not think it was related. Apparently it is. Please recover our “CENSORED-DOMAIN” domain as soon as you can. Going to sleep now, please use phone to contact me if you have any questions. Regards,

D

08:15 UTC+3

Email to support:

Hello. Any updates? I’m asking because several hours passed, I haven’t heard from you and our business is continuing to suffer greatly from this issue. Regards,

D

14:56 UTC+3

No reply yet. Publishing in hope to apply some pressure.

19:06 UTC+3

Email from support:

Hello D, Good day! Thank you for your response. I apologize if I let you felt that your case is being not attended. Don’t worry, I have your case under my watch. To set proper expectation regarding my availability, I am in the office every Mondays to Fridays, 5:00AM to 2:00PM PST. I do understand how critical this situation is and how it is affecting your business. I’ll do my best to address the issue you are experiencing. You have mentioned that your co-admin has received a message from Google regarding pending deletion of the G Suite account associated with CENSORED-OLD-DOMAIN. It was addressed to CENSORED-EMAIL-OF-C, which I think is an admin since it has received the notification. Based on your answers, if the email notification from Google was received and resulted to the deletion of the G Suite account where CENSORED-DOMAIN is associated with, most probably the primary domain configured in your G Suite account was CENSORED-OLD-DOMAIN, which its domain ownership had been contested by the new owner for them to be able to sign up for G Suite. In short, CENSORED-DOMAIN was also deleted because it was configured as secondary domain of the G Suite account registered for CENSORED-OLD-DOMAIN. For your reference, you may read this Help Center article about the importance of disassociating expired domain name with your G Suite account:

What if my domain expires? – https://support.google.com/a/answer/6359803

Before you change your primary domain (When do I need to change my primary domain) – https://support.google.com/a/answer/6301932 I regret to inform you that this can no longer be recovered as it underwent through proper process. Google already sent a notification to the primary administrator of the account. What I can advise is to sign up for a new G Suite account registered with the domain that you really own, CENSORED-DOMAIN. If you have questions, feel free to reply to this email. Sincerely, CENSORED-NAME

G Suite Account Recovery Team

19:31 UTC+3

D replies:

The damage to our company due to losing the documents is huge. We estimate it at millions of dollars and CENSORED. Please check how to recover our documents. Escalate if needed. It’s completely unreasonable situation.

Around 20:00 UTC+3

C finds a copy of notification email. It is dated 2018-03-18. C did not even open it till now because the title says CENSORED-OLD-DOMAIN. Well, nothing in the email says anything about CENSORED-DOMAIN.

23:23 UTC+3

Publishing the update, no more emails till now.

2018-03-24 12:10 UTC +3

PM to @gsuite:

Here is how this support nightmare looks from our side: https://www.reddit.com/r/devops/comments/86kq2n/g_suite_support_24_hours_without_our_account_and/

2018-03-24 14:22 UTC+3

Still no reply to email and no reply from @gsuite

18:01 UTC+3

Got this reply. I was thinking that we might not hear from Google till Monday because this guy told us (in one of previous emails) he is working Monday till Friday and today is Saturday.

Hello CENSORED-NAME, Thank you for your response. I understand that you’re getting confused why the CENSORED-DOMAIN got deleted when in fact, you only received a notification regarding the pending deletion of the G Suite account for CENSORED-OLD-DOMAIN. I’ll be addressing the confusion to clear things up. G Suite knows the nature of business when it comes to domain name registrations. When a domain name registration expires, it may be sold again to a new user who wanted to sign up the domain with G Suite. Proof of domain ownership is necessary when we use the service. A G Suite account is recognized for its primary domain. If the primary domain used for G Suite got expired, the administrator needs to switch it to a different domain that they still own. It is because if your primary domain is not updated, there is a chance that someone may buy the domain name and sign it up for G Suite. When that happens, error will occur because the domain they try to sign up (or add as secondary domain or domain alias) is existing in the system. How I understand the situation, this is the same thing happened to CENSORED-OLD-DOMAIN, as you have mentioned that you no longer own the domain name. The new owner must have tried to sign it up for G Suite, but they are getting the error that it is already in use. So they have to contact G Suite to contest the domain ownership for them to be able to use it with G Suite. G Suite sends an email notification to the administrator of the existing G Suite account in the system, that a user has proven their domain ownership and wanted to use the domain name for G Suite. In your case, CENSORED-OLD-DOMAIN was the primary domain of the G Suite account, and if it is the primary domain is being contested, the whole G Suite account will be deleted, including the secondary domain in it, which was in your case CENSORED-DOMAIN. There are some questions as well that I want to ask:

1) if CENSORED-OLD-DOMAIN has long gone expired, why was the primary domain of the G Suite account was not updated and changed it to an active one?

2) There were 3 days for the admin to reply on the said email regarding the pending deletion, was there an attempt of replying to it or was just simply disregarded because it was treated irrelevant?

3) Is there really a different G Suite account for CENSORED-MISSPELED-DOMAIN? If so, then why it got deleted too when CENSORED-OLD-DOMAIN ownership got contested? It would not be deleted if it has a separate G Suite account where it is set as primary domain, right? Please be advised that the information I have provided are based on my deductive reasoning of the answers you have provided to me and my technical expertise regarding this matter. I would also like to let you know that this case has already been escalated and I am the best person to speak with regarding this technical matter. Please also note that in accordance to our privacy policy, I am not allowed to disclose any relevant information from our end since you are contacting us through an unauthenticated channel. If you have questions, feel free to respond to this email. Sincerely, CENSORED-NAME

G Suite Account Recovery Team

My thoughts: If you understand why we are “getting confused”, maybe the system should be fixed?

21:09 UTC+3

Email to support:

[…] 1) if CENSORED-OLD-DOMAIN has long gone expired, why was the primary domain of the G Suite account was not updated and changed it to an active one? CENSORED-OLD-DOMAIN-WITHOUT-DOT-COM is previous name of our business. We are slowly phasing out usage of CENSORED-OLD-DOMAIN in all places. We are busy startup so theses things take time. 2) There were 3 days for the admin to reply on the said email regarding the pending deletion, was there an attempt of replying to it or was just simply disregarded because it was treated irrelevant? The significance of the said email was not understood. The subject had CENSORED-OLD-DOMAIN in it so it was treated like low priority issue to take a look in the future. 3) Is there really a different G Suite account for CENSORED-MISSPELED-DOMAIN? If so, then why it got deleted too when CENSORED-OLD-DOMAIN ownership got contested? It would not be deleted if it has a separate G Suite account where it is set as primary domain, right? We think that there was not a separate account, CENSORED-OLD-DOMAIN-WITHOUT-DOT-COM is previous name of our business. Please be advised that the information I have provided are based on my deductive reasoning of the answers you have provided to me and my technical expertise regarding this matter. I would also like to let you know that this case has already been escalated and I am the best person to speak with regarding this technical matter. Please also note that in accordance to our privacy policy, I am not allowed to disclose any relevant information from our end since you are contacting us through an unauthenticated channel. We own the CENSORED-DOMAIN domain, which we want to recover. How you want us to authenticate? We can prove ownership of CENSORED-DOMAIN by setting a DNS record for example.

21:33 UTC+3

Email to support:

Most importantly, we would like to make sure that a technical team is working on recovering our data: emails and Drive. Every minute without our data is critical to our clients and workers, any help with that would be great. We consider that situation where 3 days without replying to an email causes removal of all our accounts and data is totally unreasonable. Especially so when CENSORED-DOMAIN domain was in active use which could be seen from Google side and it was still deleted.

23:44 UTC+3

Got email to D’s email on the affected CENSORED-DOMAIN domain. D was able to get the email because we use different mail provider for the affected domain for now.

From: PlatformNotifications-noreply@google.com

Subject: Your Google Developers Console project is scheduled for deletion Project Shutdown Announcement Your Google Cloud Platform project CENSORED was shut down on Saturday, March 24, 2018 8:44:47 PM UTC. If you take no action, after Monday, April 23, 2018 8:44:47 PM UTC, you will be unable to recover this project. If this was unintentional, visit this URL before Monday, April 23, 2018 8:44:47 PM UTC to cancel the project shutdown: https://console.developers.google.com/project?pendingDeletion=true&organizationId=CENSORED If you have any questions, please visit the Developers Console Help at this URL, or contact support: https://developers.google.com/console/help/new/ Thanks,

The Google Developers Console Team © 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043 You have received this mandatory email service announcement to update you about important changes to your Google Developers Console product or account.

D has no idea about this and suspects it might be a byproduct of Google working on recovery. C suspects Google are deleting something additional related to our domain.

The link works only for logged in users. Since none of us can log in, we can not use the link to take a look.

2018-03-27 13:34 UTC+3

Email to support:

Any news? It’s almost 3 days since we’ve heard from you … regarding urgent case.

2018-03-27 19:16 UTC+3

Hello CENSORED-NAME, Good day! Thank you for your response. Sorry if my reply came too late as I just came back to the office. I have read your answers from my previous questions and also your request for the G Suite account to be recovered. I’d really want to help out regarding this matter as I know how this is really impacting to your business. However, the best thing that we can do regarding this situation is to tell you the truth that it is no longer recoverable and provide education so that it will not happen again in the future. Here are my responses to your answers: 1) “CENSORED-OLD-DOMAIN-WITHOUT-DOT-COM is previous name of our business. We are slowly phasing out usage of

CENSORED-OLD-DOMAIN in all places. We are busy startup so theses things take time.” – Thank you for letting me know about how CENSORED-OLD-DOMAIN-WITHOUT-DOT-COM is transitioning to a new business name. The opportunity I find here is that since the business is undergoing transition, the G Suite account should have adapted the changes in the organization too by changing the primary domain. To know more how to change your primary domain, you may check this Help Center article: https://support.google.com/a/answer/7009324 2) “The significance of the said email was not understood. The subject had

CENSORED-OLD-DOMAIN in it so it was treated like low priority issue to take a look in

the future.” – I really feel sorry if that email was not prioritized because of its subject containing a domain which you no longer own. However, it is always a best practice as an admin to keep the communication lines open and check notifications from G Suite as it may contain important, sensitive and private message that concerns your G Suite account. 3) “We think that there was not a separate account, CENSORED-OLD-DOMAIN-WITHOUT-DOT-COM is previous name of our

business.” – Thank you for letting us know that you are aware that both domains are configured in one G Suite account. Based from your responses, I think the opportunity here is to understand the importance of these terminologies in G Suite:

1) Primary Domain – The domain name used in which the G Suite account is registered. The very foundation of a G Suite account.

2) Secondary Domain – A type of additional domain that shares from the storage of the primary domain’s G Suite account. Users may be provisioned under this domain and use it to log in.

3) Domain Alias – A type of additional domain that automatically adds email aliases to all users under the primary domain. Email addresses under domain alias can not be configured as login username. It is really important to update the primary domain if the domain name enrolled in it has expired its registration. For your reference, you may visit this Help Center article regarding the importance of updating the primary domain if in case it expires: https://support.google.com/a/answer/6359803 The reason it is no longer recoverable is because the domain CENSORED-OLD-DOMAIN was set as primary domain of your G Suite account that was legitimately contested by the new owner for them to be able to add their new domain with G Suite. In which, it was also admitted by your end that CENSORED-OLD-DOMAIN is no longer owned by your organization. Also, I have also checked that the new owner has already enrolled the domain name to G Suite. Thank you for your understanding. Sincerely, CENSORED-NAME

G Suite Account Recovery Team

2018-03-27 20:54 UTC+3

Email from another person in support. Gives some hope.

Hello CENSORED-NAME, Thank you for contacting G Suite Support. I understand the account CENSORED-DOMAIN. has been deleted. My name is CENSORED-NAME and I am in charge of your case. I am aware that retrieving your domain is very important to you. I have contacted a specialist team regarding this issue. As soon as I hear from them, I will update this case and provide you with further notifications regarding the issue at that point. I would like to be honest with you and provide the right expectations, I can not promise to recover your domain but I will exhaust all the resources in my hands to do it. The case will remain open. If you have any question please respond to this message and I will be very happy to follow up with you. I look forward to contacting you at the earliest convenience. I’ll be available Monday through Friday from 4:00 pm to 1:00 am IDT. Have a nice day! Sincerely, CENSORED-NAME

G Suite Support

2018-03-28 02:33 UTC+3

Email from third support person:

Hello CENSORED-NAME, This is a friendly follow up regarding the case you currently have on Google G Suite Support. I hope this message finds you well. We thank you for patiently waiting for a resolution to your case. I just wanted to let you know that due to the nature of the issue we can’t guarantee that the data will be restored since it got all permanently deleted but, our engineering team is currently working to verify if we have an option to restore it. In the meantime, We would greatly appreciate if you don’t create a brand new account until we get a final resolution to your case. Please don’t hesitate to reply back to this email if you have any additional questions. HAve yourself a wonderful day. Best regards, CENSORED-NAME-OF-THIRD-SUPPORT-PERSON

Google G Suite Support.

2018-03-28 06:14 UTC+3

Email from support:

Hello CENSORED-NAME, This is a friendly follow up regarding the case you currently have on Google G Suite Support. I hope this message finds you well. We thank you for patiently waiting for a resolution to your case. I just wanted to let you know that due to the nature of the issue we can’t guarantee that the data will be restored since it got all permanently deleted but, our engineering team is currently investigating. In the meantime, We would greatly appreciate if you don’t create a brand new account but verify domain ownership by adding the following CNAME: Label/Host: CENSORED-NUMBER Destination/Target: Google.com Time to live (TTL): 3600 Please don’t hesitate to reply back to this email if you have any additional questions. Have yourself a wonderful day. Best regards, CENSORED-NAME-OF-THIRD-SUPPORT-PERSON

Google G Suite Support.

2018-03-28 11:12 UTC+3

Email to support:

Hello. CNAME added as requested. $ dig +short CENSORED-NUMBER.CENSORED-DOMAIN

google.com.

CENSORED-IP Regards,

CENSORED-NAME

2018-03-28 16:18 UTC+3

Email from support:

Hello CENSORED-NAME, Thank you for your reply. I apologize for contacting you until now but I am starting my shift. Your case is a priority for us and if I can not contact you, someone else will. I see that CENSORED-NAME-OF-SUPPORT-PERSON-3 asked you to create a CNAME record. We are still working on your case, I’ll keep you updated. The case will remain open. If you have any question or additional comments in regards to your case, please feel free to reply to this email. Sincerely, CENSORED-NAME-OF-SUPPORT-PERSON-2

G Suite Support

2018-03-28 23:04 UTC+3

Automated email from Google:

30 days remaining to set up billing for G Suite Basic

Hello, Our records indicate that you have no payment information on file for G Suite Basic for CENSORED-DOMAIN. You have until April 27, 2018 to set up your billing information, after which you will be suspended. Please set up billing in order to continue uninterrupted service. Here’s what you need to do to continue your subscription to G Suite: Click the Set up billing button below.

Select payment plan.

Review your purchase and accept the terms of service.

Enter your payment information.

If you would like to pay by check or manual bank transfer every month, please contact G Suite support.

D logs in with his email at CENSORED-DOMAIN. Users appear to be there. D calls C. C fails to log in. In admin, C is a “deleted user”. The documents, at least most of them appear to be there, we are not sure, it’s late, we’ll look into this tomorrow. We also hope to get some official notification about recovery finish.

2018-03-29 06:55 UTC+3

Email from support:

Hello CENSORED-NAME, I want to thank you for your ongoing patience while we have working to resolve your issue. My name is CENSORED-SUPPORT-PERSON-4, and I am the Technical Solutions Engineer (TSE) who will be taking ownership of your escalated case from CENSORED-SUPPORT-PERSON-2. Over the past ~30 hours, I have been engaging with our Product Engineers to identify the safest (and quickest) way to recover the “@CENSORED-DOMAIN” users that previously existed on the G Suite account associated with “CENSORED-OLD-DOMAIN”. In most circumstances, it is easy for us to restore a deleted G Suite account. However, in your case—because you did not maintain ownership of the primary domain—another customer has set-up a new G Suite account with it. Unfortunately, our systems cannot accommodate two G Suite accounts with the same primary domain, and we cannot evict the one that was recently created by the current owner of “CENSORED-OLD-DOMAIN”. If you were that individual, I am sure that you would not appreciate us deleting (or asking you to delete) the G Suite account that you just created. Since “CENSORED-DOMAIN” cannot be restored independently of “CENSORED-OLD-DOMAIN”, we have created a new G Suite account (on your behalf), with “CENSORED-DOMAIN” set as the primary domain. In addition, we have begun the process of moving your deleted users to this new G Suite account—this is outside our normal processes, which is why it has taken longer than desired. At this time, we have been able to restore the following users, and move them to the new G Suite account: CENSORED-USERS-LIST At this this time, the aforementioned users should be able to sign (via https://accounts.google.com/) and access their data. Additionally, as a Super Admin, you should also be able to access the Admin console (via https://admin.google.com/), where you can set-up your billing information, and manage these user accounts. Please understand that were are doing all of this on a best-effort basis, and we cannot guarantee that user data has been (or can be) recovered. Currently, due to technical constraints, we are having difficulty moving “CENSORED@CENSORED-DOMAIN” and “CENSORED@CENSORED-DOMAIN” to the new G Suite account—however, we are continuing to work on this, and will let you know once we have determined whether or not these users can be recovered. I also want to note that we cannot restore any Google Groups—however, I believe that you only had “CENSORED@CENSORED-DOMAIN” and “CENSORED@CENSORED-DOMAIN” on the former G Suite account. If these are necessary, you should be able to re-create them via the “Groups” page within your Admin console. I will reach out to you around this time tomorrow with the latest information on our progress. From your cell number (+972 CENSORED), I understand that you are located in Israel. Since I am located in Mountain View, California, there is little overlap in our working hours—however, if you would like to speak with someone on phone, please let me know, and I will try to co-ordinate something with my European colleagues. Sincerely, CENSORED-SUPPORT-PERSON-4

2018-03-29 14:53 UTC+3

Email to support:

We appreciate the technical efforts being put into solving the problem. Thanks! Me and some other users were able to log in and see our emails and documents. I was able to log into admin. That’s great. C was not able to log in but I understand that you just haven’t finished with the account. C is listed under “deleted users” in admin. I will gladly speak to you tomorrow, if my 7AM are OK for you. I do prefer to speak to the person who did the recovery and not more-time-zone-connvenient colleagues. CENSORED. Regards,

CENSORED-NAME

2018-03-29 18:00 UTC+3

Call from technical person that works on our case.

They are still working on recovering the last account. It was slowed down by the fact that C opened consumer Google account with same email address (he needed to authenticate to an external service). The technical team got the case just about two days ago. Google will be improving the process Better notification that will mention the relevant domains. Better escalation. Apparently first line support (“agents”) were not sure what to do with our case because it was unusual so it took a while to escalate. Support will be instructed to use the mentioned communication channel There should be news in a few hours or maybe tomorrow morning (UTC+3 time)

2018-04-01 10:16 UTC+3

No further communications from Google. Admin shows that the last account was recovered. C checked the documents and they seem to be OK.

2018-04-06 18:42 UTC+3

We started to check possible backup solutions. We are also considering which other systems are critical to us and hence need to be backed up.

2018-04-09 21:55 UTC+3

Email from support:

Hello CENSORED-NAME, My name is CENSORED-SUPPORT-PERSON-5, and I am one of the support managers in Google Cloud Support.

NAME-OF-SUPPORT-PERSON-4 will be out of office this week, so I wanted to follow up on the status of the issue. Based on the case and issue history tracked by Engineering, we have addressed last issues associated with CENSORED-EMAIL and CENSORED-EMAIL on April 4th and thus all the issues should have been addresses. Please do let us know if that is not the case, and I will work with another engineer to look into it. Thank you for your patience with this case. Regards,

CENSORED-SUPPORT-PERSON-5

Manager

Google Cloud Support

2018-04-10 15:59 UTC+3

Email to support:

Hello NAME-OF-SUPPORT-PERSON-5, We have noticed that the data was recovered. It was strange that we had to notice that by ourselves because we were not contacted after the last accounts were recovered. Other than this additional item for your post-mortem (we are really curious what are the resulting action items), everything seems to be OK, thanks. Regards,

CENSORED-NAME

2018-04-10 18:04 UTC+3

Email from support:

Hello CENSORED-NAME, Thank you for your message. I’m really happy to hear that you have been able to access the data for CENSORED-EMAIL and CENSORED-EMAIL. During our last phone conversation, I had mentioned that our engineering team was encountering significant difficulty recovering these users. However, on April 4th (a little before midnight IDT), they confirmed that the necessary changes had been made to undelete them. I did not contact you at the time, as it can take 24-48 hours for all user data to be fully restored—especially for a user like CENSORED-EMAIL, who has used over 9 GB of data storage. A notice about this “propagation delay” used to be mentioned in the Help Center at: https://support.google.com/a/answer/1397578, however it was recently removed for unknown reasons. I have confirmed with our technical writers that this notice has been reincluded in the current article draft, which is currently undergoing review. I am glad to see that NAME-OF-SUPPORT-PERSON-5 contacted to you yesterday, since I was out of the office. As CENSORED mentioned, I will be out for most of this week—however, I did want to take some time to inform you of the progress being made on the “action items” (from the post-mortem) that we discussed over the phone. With regards to your concerns about how your case was handled by our tier-one team, I have relayed your feedback to each of their managers, and have asked that all of them receive additional training to ensure that they do not avoid making outbound calls and/or escalating cases when specifically asked by customers. In addition, we are continuing to work on updating our email templates to ensure that all domains are listed—instead of just the one with contested ownership. We are currently working with our Product Counsel team on several proposed changes to our procedures/workflows, and this has been included amongst them. Finally, I am happy to report that our engineering team did address the root cause of the technical issues that were preventing your two users from being restored. Should any of our customers ever end up in a similar situation, we should be able to initiate the restoration process a lot faster! Because of all the trouble that you experienced, I have gone ahead an extended your “G Suite Business” trial to May 31, 2018 (the farthest back possible)—I hope that this will give you enough time to backup/export all of your user data, and to re-evaluate the product. If you encounter any further issues during this trial period, please do not hesitate to respond back to this email and let NAME-OF-SUPPORT-PERSON-5 or me know! Sincerely, NAME-OF-SUPPORT-PERSON-4

Google Cloud Support

2018-04-12 12:52 UTC+3

Email to support:

Thank you!

TODO: Schedule post-mortem