Advertisement
It was a classic stand-off. Vendor number one’s system wasn’t talking to vendor number two’s. What to do? Of course! Blame the customer’s network!
I worked for a TV station that was part of a group run by a common owner. One of the stations in the group used a system known as production automation, which allowed a single operator to control all of the equipment in the control room during newscasts. That would include the video switcher, audio console, camera robotics, video playback, lighting, and graphics generators. The computer system in the newsroom takes the scripts written by reporters and producers, generates a sequence called a rundown, and transmits and updates it in real-time to the automation system.
Do you have a memorable experience solving an engineering problem at work or in your spare time? Tell us your Tale
Advertisement
While performing a major update to one of the systems, communication stopped. Head scratching ensued for a while, and then the two vendors decided the problem must be something in the network that was blocking the IP packets. The station’s engineers pointed out that nothing had been changed in their network, and in any case, there was no internal routing or filtering going on. Not good enough, say the vendors. Prove to us it’s not your fault before we continue. Their advice was to install a copy of Wireshark, analyze the packets, and show us that the path between the systems is clear.
Advertisement
That’s reasonable as far as it goes, but Wireshark is a mighty powerful tool, and it is not for the faint of heart. At the local TV station level, the IT staff generally does not have the expertise needed to fire it up quickly and interpret its results. The station group’s central IT networking folks do, but getting them involved would have taken a good deal of time, and if they had to travel to the site, expense.
I was just a bystander to this. My own station was one of those with the same systems, so I was included in all of the emails flying back and forth. As it happens, not long before this incident, I had written a small one-trick pony Windows utility. All it did was send IP packets from one computer to another via a specific port. As seen in Figure 1, if the path is clear, the receiving computer replies, and the arrows move. Simple as that.
Figure 1 A demonstration of the Windows utility written by the author, sending IP packets from one computer to another via a specific port.
I sent the program to the station’s IT director, and in less than half an hour, he installed it on both systems, checked all of the ports the vendors specified, and found them all clear. With no more finger-pointing at the customer, the vendors had to get to work to find the actual cause of the problem, which turned out not to be network-related.
A few notes about the program. The image shown is just a demonstration, with both ends running on the same machine. In real life, one copy would be on each of two machines on the network, across the room, or across the world. Also, to be honest, I probably spent more time getting the ballistics of the arrow movement looking good than on the rest of the program.
Robert Yankowitz retired as Chief Engineer at a television station in Boston, Massachusetts, where he had worked for 23 years. Prior to that, he worked for 15 years at a station in Providence, Rhode Island.
Related Content
Yep, that was and I’m sure still is standard procedure when two different vendor’s systems don’t work together – blame the customer, the customer is doing something wrong or there’s something wrong with the customer’s network or equipment. In my over 50 years, I have heard that blame many, many, many times.
Very clever tool/solution Robert!
Can you share the application?
There have been a number of times I could have used it for troubleshooting!
Dave
Great story and nice utility. I have used PingPlotter for doing the same thing. It works great to verify network connectivity. There have been more than a few times where taking the time to learn Wireshark and the ins and outs of networking really paid off. Finding things like a vendor’s application not responding correctly to the correctly sent vocoder type using Wireshark really helped resolve a stubborn problem. It’s always the other guy and often the third party needs to point out who really is at fault. Good job and thanks for the telling about your experience.
oh man, that has been my life. Some suppliers say that since you have a service contract and you came to training 5 years ago, you should be able to figure this out yourself even though we are on 6 updates since then and the knowledge is stale. They then tell you its your fault, the network or whatever. “not us” – then its up to you to prove it. I will say that one company installed wire shark in a way that they were able to pull the logs then sent out an update for all users that made the system way more stable.
I have something with another company going on where equipment is randomly rebooting, locking up and only thing that fixes it is power reset where its off for 3-5 min then power it back up the problem goes away. The supplier tells us to “hook up the special expensive cable and use putty to pull the logs” – but wait there isn’t anything there! “oh, when you reset power the log data is erased.!!!!” could have told me that… but you can’t log in when its all latched up how do I pull the logs if its locked up? is it still my fault? I am just thankful this outfit doesn’t design flight and data recorders for the aerospace industry! It might make a good article unto itself.
This is like Brian Dipert’s light bulb teardown. I am left on the edge of my seat seeking the fix to the network problem, but the article abruptly ends!
What was the real problem & its solution?
Sorry Brian, but it’s not that exciting. It was just some checkbox or other on one of the systems that wasn’t checked when it was supposed to, or vice versa. Not uncommon when doing a major upgrade to the software of a big system. (I once had a new automation system with redundant servers that hiccupped every once in a while when it was brand new. The vendor sent an engineer all the way from the UK, who spent three days before he found they were both set as primary in their .ini files. Changing “Primary:” from “Y” to “N” on one of them fixed the problem.)