r/talesfromtechsupport

This happened many moons ago. It is Friday around noon and people only worked until 1 or 2pm during those days. I was having a quiet day, chilling in my office all alone and getting mentally ready for the weekend. For some reason the company decided to have me as the sole admin for 70k square meters (about 750k square feet) and over 300 users at my location.

The first few months were rough but after 2-3 months, I had it figured out. Adjusted the local GPOs, implemented some scripts for the most reoccurring issues and general overall improvements. So despite the amount of users and area I had to cover, I had actually weeks where I didn't get a single support call. This was one of those days...well, until it wasn't.

Player 1: Yours truly ($Me)

Player 2: Sales lead ($SL)

My phone rings and the built-up dust on it starts to fall onto the desk. I see the caller ID and just went with my usual banter.

$Me: Welcome to the mental asylum in $location. Do you want to make use of this week's special of checking in 2 coworkers for the cost of 1?

$SL: Very funny you doofus. Look, I think I might have an issue here. One of our customers sent me a link, but nothing happens when I click it. What can be done?

Usually, I just connect remotely and have a look, but I was bored to death in my office and it felt like my walls were closing up on me. So I decided to rather walk down 2 floors, walk across our main road and climb up 1 floor to the sales team in a different building.

I arrive at the sales department in their full glory and $SL is already awaiting me.

$SL: Thank you for coming so quickly. Do you see the email?

*points at her screen with the email*

$SL: Now, when I go ahead and click the link and put in my credentials, nothing happens.

*$SL goes ahead, clicks the link and is being presented with a microsoft login. $SL goes ahead and enters email and password, but the page just reloads*

Usually, I would have stopped $SL, but I knew $SL had already done this, so there was no point. So I just quietly looked, screaming in pain inside.

$Me: Hmmm, may I sit and have a look?

$SL: Sure go ahead!

I sit down and check the email. Very generic, bla bla bla "please review" more bla, and a random link. URL is not part of our or the sender's domain. How lovely, $SL just trusted the customer's email. We were doing email campaigns back then, which included an external company sending phishing mails to our employees and notifying them if they clicked the links or even entered their credentials. $SL should have known better, but oh well. Just a password reset needed, nothing too bad.

$Me: It looks like your customer's email got hacked and they sent out this email to try to get more credentials from their contact list. Here are the parts where you could have noticed that something was fishy. But not too bad. Not much time has passed and it is just our password for emails.

Back then we had a password for logins, another one for M365 stuff, one for SAP, one for SAP concur and one for SAP Ariba. Don't ask why, we just did way before I had joined.

$SL: Oh ok. But I also tried my other passwords.

*cold sweat*

$Me: Um...what? What do you mean exactly?

$SL: You know, the passwords for SAP stuff. I even tried the affiliated usernames instead of my email.

*If I leave work now and drive to the next airport, I might be at the beach before dinner*

$Me: Why exactly did you do that?

$SL: You know, I just thought it might work

*Absolute genius! Maybe try your Credit card number & expiration date and CV number next?!*

$Me: Oh boy...ok, so we will have to reset all of those now. Sadly, I have to push this up the ladder now and inform our HQ and especially our CIO.

$SL: Oh no! Well, I guess I understand.

*some moments pass in silence*

$SL: But what about the rest of my team?

$Me: What about them?

$SL: Well, since I thought it might be a problem on my laptop only, I forwarded the email to them and had them try their logins too. Do they need to reset their passwords as well?

*There is no way someone can be this dumb. Please tell me there is a hidden camera somewhere and I am on live TV?!*

$Me: Are you joking?

*Insert The Office meme: *softly* Don't*

$SL: No, why?

*Insert The Office meme: Nooooooooooooooooooo*

$Me: Alrighty! You get a new password, you get a new password, and you get a new password!

Making light of the situation was my way of hiding my urge to slap people.

I reset the passwords I was able to reset and then called our internal support line for SAP related support. Explained the situation and I think "No, I am not joking" was used several times. Then I spoke on the voicemail of our CIO as he wasn't picking up.

Still to this day I get something like PTSD twitches when I see $SL's number appear on my phone. I was moved to one of our locations in the US as my wife who is a US citizen got homesick, so I had asked for a transfer and it was granted by our CIO. But $SL still sometimes calls to ask me how I am doing in the US. Nice person, just suffers from being oblivious and gullible.

One of our glorious colleagues has become a term for epic and/or weird failure. Let's call him Smithers. Whenever someone runs into a strange issue, then he got Smithered. Likewise, if Bob from the quality department did something extremely stupid, then Bob Smithered himself.

On a suspiciously quiet day my IT-colleague and I are doing our usual stuff which includes taking care of our RDS farm (2008 R2), ERP system (prior to 2000 - there are newer versions but management decided not to upgrade) and other hellish circumstances. Did I mention this happened in the post-Covid era, so just recently? No? Well, now I did.

Suddenly the one and only Smithers calls us from his home office. His laptop appears to be online on Teamviewer, his emails are working but his Teams is offline. He says he only encounters this randomly at home but never at work or while he is traveling. We remote in via Teamviever and quite frankly just google-searched the situation and it told us to go into Control Panel -> Internet Options -> Connections -> LAN-Settings and there to tick the box that says "Automatically detect settings". To be honest, from what I know that shouldn't fix the issue but it did. I have no idea how his home-office is configured and at that point I was waaaaay too afraid to ask.

Few days later he calls again. Same issue. Same fix, but why was that checkbox not ticked? He claims he didn't touch it and to be honest, that is way past his skill level to dive that deep into any settings. So I quickly wrote a batch script and put it on his desktop. As center as I possible could and named it "Teams No Internet Connection Fix". I show him the icon and tell him to double-click that the next time it happens. "Uh-oh, I will do that. Appreciate you buddy!"....yeah, suuuuure buddy....

Again a few days later...here comes our hero again. Same issue. I ask him whether he clicked the icon on his desktop. He replies "What icon?"....*Facepalm* "The one I showed you last time in the middle of your desktop?"....silence...."What icon?"

I just give up and remote into his laptop again and click the icon for him.

This happened a few more times until I gave up on him. I created an scheduled task that runs every 5 minutes and executes the batch script. Now when he calls, we just tell him everyone is affected by this worldwide Teams-outage and he should check again in about 5 minutes. I feel like that guy from Idiocracy who had to tell the cabinet of the president that he can hear the plants asking for water instead of Gatorade, as the cabinet members couldn't comprehend the actual situation.

Oh, and obviously the script was moved to a new folder directly under C:\ to prevent the genius from deleting the file on his desktop.

I am a fiber optic engineer. And this datacenter will be the death of me. Recognizing the big words will bore the crap out of most people, I'll give the abridged version.

This particular datacenter does not train or equip it's employees properly, and as a result, anytime they have a slightly complex problem, they make it worse by trying to fix it.

Then they call me for help, despite the fact I don't even work for them. I am their OSP vendor, meaning I fix problems outside and between buildings. Inside their own building is supposed to be their own problem.

I receive an email. This infamous datacenter tells me they're experiencing an "ORL" issue. That's optical return loss. It means the connection is too shiny and too much light is returning the way it came, backwards, instead of going forwards through the fiber optic cable to the destination.

I tell them it means their connectors are dirty and to clean them with proper cleaning supplies. Fun fact: they do not have proper cleaning supplies.

Days later they follow up, telling me the issue is now a 10dB degrade. That gives me pause. That could actually be an issue between the buildings I would be responsible for. A degrade means the light going from one building to the other is too dim when it arrives, some of it has been lost. 10dB is not a small amount, it means 90% of the light is missing and only 10% is getting to the destination.

I show up. I start my troubleshooting by asking a lot of questions. The answers I get confused me. The equipment readings I get confused me. Finally I realize what's happened.

On a previous trip I told them they had bad patch cords, they would fall out of their plugs if you so much as breathed at them.

Following my advice, they replaced them with proper cables. So far so good.

But one of the circuits didn't come back up fully. They properly diagnosed the new cable was dirty and needed cleaning.

Mistake number 1- I told them the cleaning supplies they had on hand were terrible, and likely to make things worse. Standard cleaning solution is isopropyl alcohol. I was told that was a hazard and they needed to use an alternative cleaner. This alternative cleaner is a mixture of Propyl Acetate, an industrial solvent that is itself flammable and emits hazardous vapours that are also flammable. Ethanol, which is literally fuel. And- Isopropyl alcohol!

This cleaning agent leaves behind some oily residue that- causes ORL issues because it's shiny. So they cleaned this fiber over and over and failed every time, and concluded the problem must be elsewhere- completely ignoring that I had already told them their cleaning supplies sucked and were incredibly inadequate.

Now if their cleaning supplies weren't awful, that might be a reasonable conclusion. So they performed a loopback test.

The connection normally goes equipment, patch cord, rack, cable leaving building, rack in next building, patch cord, equipment. A loopback test is purposefully looping the patch cord back to the transmitting equipment by connecting it to itself at the rack.

When you do this, you add an attenuator. This is a special plug that adds loss. The system is set up assuming you'll lose so much light going from building to building, so that short range connection needs to be 'dimmed' a bit. That's what the attenuator is for.

Mistake number 2: they connected the attenuator, and forgot where they put it.

The 10dB degrade I was sent to repair was caused by a 10dB attenuator they installed, and couldn't remove, because they forgot where they put it. And on top of that- they still needed me to clean their damn panel for them.

The remote support team knew all of this, and through tactical lies of omission, made me think I might actually need to fix something that was my responsibility. Instead they used my competence to fix shit their internal team doesn't have the training or equipment to deal with: which are basic essential functions of any datacenter team.

These lazy bastards tricked me into troubleshooting diagnosing and repairing their own shit because they can't be bothered to train and equip their own employees to do their jobs.

I put my foot down and told them this is our last courtesy dispatch. For future calls involving this datacenter, we are working strictly to the terms of our contract.