The issues between DevOps and General Enterprise Technology

In the Facebook outage, it reminded people that you can’t trust a company which thinks they have only a few million users, when they don’t accept they work for a trillion dollar enterprise. This meaning that Facebook’s servers and services are more consumer-class than enterprise class or worse the braintrust is very weak.

It’s important to note, that even though the Internet Protocol is in itself a software stack (think of this as an “extension” or “driver”), but software engineering, web apps, etc., is in itself a different skillset. People who have used Microsoft’s Windows Server solutions really do not know much about IP networking. For many years, the Server editions came with a DHCP server, how many of the Microsoft certified admins know more about DHCP other than it gives IP address at the local level to get out onto “the Internet? I have suspected about VOIP deployments in the past, where NT admins didn’t understand “DHCP options” and alike because you know it’s more important to manage an Active Directory.

Look at Microsoft’s own VOIP systems, it fell shorter beyond Cisco’s Unified Call Manager, and obviously the Avaya, Nortel, Mitel or Shortels of the world. It’s sad when a Cisco can do better. This has a lot to do with Microsoft’s DNA of everything being software and talking to Microsoft’s own blueprint. Anything that routes outside a data center of an in house, on prem Microsoft solution is something Microsoft doesn’t get, and their software shows it. If it has to hit a Cisco, or needs to interact with a Cisco IOS, well good luck to that.

The Session Initiation Protocol part of Voice over IP was yet another rip-off from the traditional telephony, and was created by application people, since SIP was based off the Web standards or HTTP technically speaking if it’s a device talking to another machine. In a lot of ways SIP was designed almost like cell phones because a telephone number is basically a URL, and when you hear the “dial tone” it’s a fake noise to assure the user to replicate it’s a phone. Because the people who developed SIP didn’t understand enterprise voice systems, its basically like a landline with all the 19 potential features you could add on to your home hardwired or broadband phone service, because the people who likely created it looked at their POTS phone and assumed the same.

What a bunch of assholes to make an ass out of themselves.

Understanding software and an imaginary world is the worst thing to have in DevOps, of which is the new IT department fusing move-fast-and-break things punky coders, and wife beating sysadmins who hate change, but preach it to their “end users” or “lusers”. It’s kinda ironic that either type of man typically lacks software of another sorts, people. Understanding people. The IT world needs to be reformed to really not be the evil world to their fellow employees, and they need to stop jacking off to the C-suite, to help them save money by cutting jobs to their own people. This kinda goes full circle of the way money and influence is killing society with Facebook and their technical approach. If you are building a social network, that isn’t based on empathy, you are certainly going to cause rift amongst the people who are using your service.

On Facebook’s Outage…

I am not going to be the asshole tech pundit who trashes others for not knowing things like Border Gateway Protocol, or what have you.

I mean seriously, who at the Facebook’s technical staff even know what BGP stands for? Or any server administration period.

I am talking about the Facebook outage that occurred on Monday morning – the morning after the whistle blower appeared on 60 Minutes. There was little clarity whether it was a Distributed Denial of Service attacks (or DDOS – yes I capitalize the “O” because I find mixed capitals in initials to be tacky) or just a simple routing issue. It turns out, according to their blog post in response to the outage

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.” From Facebook’s engineering blog

Where is the PR to help gel out vague languages such as a “configuration change”… that even though I am not technical, I would certainly test things before applying (but that would violate the Move Fast and Break Things ethos.)

I’ve had a theory that coders were very narrowminded groups of people who don’t know much about networking, much like how networking guys took a literal backwards view on telephony.

Or that coders have basic views of networking, that is no different than a gamer or a YouTube influencer.

This is sad if a large trillion dollar enterprise is acting like a bro startup at a scrappy office building. It’s been confirmed that Facebook’s corporate hierarchy is “flat” according to the whistle blower speaking to Congress this week. It’s not a surprise and coders see the world as flat (maybe they are flat-earthers!)

Facebook’s own computing (of which they do, and not use Google or Microsoft or Amazon), is not just their own farm, but they had built their own bare computers, without standard rack hardware, just sitting out in the open, using open source software along with their wacky hardware designs. Facebook’s software however, should be concerning if the underlying code is basic Linux code, and if Linux servers are acting as routers, they typically are not intended to be built to handle billions of users.

This is really, really bad, if Facebook’s routing is as shallow as a home gamer with consumer grade equipment or consumer grade networking settings. This consumer mindset should alarm enterprises of any size because IP networking and routing is more than just from going in and out.

If you want to be the next Facebook, it’s likely logical you should consider Software Defined Networking, just make sure it’s built for scale and built for serious environments. Facebook’s very casual and reckless approach for managing their systems should also be a wake up call for aspiring web disrupters.

Unsolicited Advice: How Can Facebook Prevent the Latest Screwup

In the last 5 days, Facebook has been under pressure, the whistleblower who broke the stories to The Wall Street Journal of specifically Instagram monitoring and adjusting knowingly content with the intent to harm teenage girls came forward to 60 Minutes Sunday night. While it made media play the following morning, around lunch hour, Facebook, Instagram and Whatsapp all experienced outages  at least for the rest of the day till roughly 6:00 pm Eastern Time when some of the services returned

On Tuesday, it was reported that Facebook had made changes to their core servers that resulted in the outages. In the midst of the outages, rumors flew that Facebook employees literally had to go into the data center physically to perform the reboot. Another cause was a screwup to the Border Gateway Protocol routing tables. This is one of several IP routing protocols to route Internet traffic between machines and users.

Both have no excuse. I have some solutions:

1. Facebook violated COPPA, both on the letter and spirit of the law

For those who say the rules and regs have to be rewritten for a modern era of technology is frankly bullshitting you. The Children’s Online Privacy and Protection Act or COPPA was designed to protect all children under the magic age of 13 as they put into the regs from being tracked, monitored specifically personally identifiable information or PII. And even if the users were over 13, this doesn’t give Facebook the ethical or moral standards to treat a 14 year old user the same as a 24 year old because the world is not flat or bland, where everyone should be treated 100.0000.0000.000% the same.

If anything, Facebook should be fined if there is data that was harvested for the persons under the age of 13. The Instagram for kids app that Facebook was contemplating should be reviewed by a legal firms to ensure it’s all compliant to the existing COPPA regulations, and not universities.

2. Whistleblowers need to be protected (legally)

One of the provisions of the controversial Dodd-Frank law, was to address a decades old issue of how can a company prevent another Enron. Part of this was providing bounty support for anyone who was open and willing to report nefarious actions by corporate executives. While the Trump administration got rid most of the law (because it had Obama-fingerprints on it, other than promoting an indiscriminate free-market agenda) this whistleblower protection still exists because the law was the basis for the whistle blower to come forward since Facebook is a publicly held company.

There should be no legal shame for these people to come forward. It’s typical for a publicly held company to be responsible financially for wrongdoings, but Faceebook is with a zillion dollars by science of public perception, and the idea they can’t seem to have enough money to finance (or “fund”) a group of people to make Facebook’s platforms safe for everyone?

3. Facebook needs to not only employ security and safety, but FFS no one that is running the server farm has no knowledge on IP?

This clearly shows how people (even nerds and techies) assume the Internet is the Web or a platform, not a underlying protocol to connect machines to users. The Monday outage seemed to show how Facebook appears to be running on plastic clad routers or hubs. Worse, I suspect since Facebook’s own servers are built in house, that I wouldn’t be surprised to see if their networking was built in house too (read Software Defined Networking.) I wouldn’t be surprised to see if they shoplifted some open source code for routing and switching that in reality is designed for light traffic and lacks important critical features that would be required for a site like Facebook, and apps like Facebook, Instagram and Whatsapp.

4. After this incident coders can’t be taken seriously anymore

The guys who brag they know Python, are real jerks. I am going out of my way to call them “zealots”. They are completely intolerant to anything that doesn’t interest them. They don’t care about  QoS, or IP routing, all they care about is their own type of networking that the rest of us don’t see it as. Technology professionals need to look at themselves in the mirror, to see if their one skillset is worth having a job in the first place. Just like in telephony, the phone guys didn’t get IP, and the networking admins didn’t understand the underlying telephony features, the same applies to networking admins and server admins. Look at how Microsoft sucks at VOIP because Windows guys don’t get Cisco’s IOS, and how lousy the constantly rebadged Unified Communications System showed that?

Brag about your skills in Python, but Python won’t save you from screwing up a routing table.

And if you disagree with me, then you lack empathy, you lack diversity, and you must be a little creepy perv to spy on underage girls’ PII.

Techie No-Nos – On DNS…

For security purposes and ensuring you’re not clogging your own low voltage pipes, your “DNS address” should be local. If you use Google or your ISP’s DNS address any connection to your local devices can only be accessed by IP addresses because you have no way to have a DNS point to a name to a host with an numeric IP address.

Local IP addresses are kinda like the FRS radio bands, ones that ISPs are expected to not route for the purposes of having a local area networks routable.

Anything from:

  • 10.0.0.0 to 10.0.0.255
  • 172.16.0.0. 172.24.255.255
  • And 192.168.1.0 192.168.254.255

The local DNS address should point out to the firewall or wireless router. Many smaller end devices have the ability to basic DNS if you have a few devices where you want to connect them by domain-name.

With all these cyber attacks, it’s best to separate what’s exposed to the overall Internet, and what should be local. Computing devices should be connected with a local IP and DNS address and appliances that help route local devices to the overall Internet should be the ones with the most exposure like having the Google DNS address.

Just do the right thing, and keep your devices protected with a better structured local network

#

Entercom Happens – in an Audacious Matter

Entercom, the second largest radio group (or third depending on the whole Cumulus/Townsquare/threesome is… I honesty lost track) has bascially been a poorly run company. While Boston bad boys like Howie Carr and Kirk Minihane (where the latter is nothing but a prick, a pretty-boy, and is an Arbitron addict, and only cares about ratings) piss on Entercom by name and personally attacking, harassing and the upper level management at the company, Entercom traditionally has dual personality disorder, one wants to make profit, the other cares about the content, or should I say “cares” in air-quotes.

If you got fooled on the week of April Fools, it’s not a joke. The company rebranded into Aducity (rhymes with odyssey – like John Sculley’s 1987 memoir!) I am not making this up; there’s even a YouTube clip of the Top of the Hour airchecks indicating the station is owned by a company that rhymes with a title of a memoir; and WCBS radio had the audacity (no pun intended) to interview David Fields, the long time CEO of Entercom to talk about horrible rebrand.

Continue reading

Merrimack’s 2021 “Election Day” – is this the Death Sentence to Municipal Democracy?

Merrimack, New Hampshire is a unique town because they are the only municipality that has “elections” outside of First Tuesday of November and the Second Tuesday of March for Town Meeting or officially the “Deliberative Session” type of election or better known by “SB2”, the second article proposed in the New Hampshire Senate season* in the mid 1990s (’95 if I am not mistaken.)

*that is not a misspelling, I like to call the budget period a “season” and most of the issues debated last about 3 months, even if the General Court hangs into late May for budget matters on the odd years.

Continue reading

On Outside Broadcast Trailers… IT’S SUPER BOWL SUNDAY!

Today is a big day here in America, where the most popular final playoff game for the NFL ends today. Ironically one of the winning conferences will be playing in their home turf – a tradition that rarely happens (if at all.) Will I be rooting for the Bucs? Maybe not. I think it’s time for Tom to retire. I also do not have much faith with Rob Gronkowski; who like to play when he feels like it. I can say this cuz I watched many of the Tampa Bay games this season. (And I really don’t care for Fox’s coverage of the NFC games… leave it at that.)

But anyways… CBS is touting a lot of fancy hardware for this year’s Super Bowl. This year, they’re playing on the cinema route, making use of 4K and 8K… quoting TV Technology

“In total, CBS says that it will have more than 120 cameras placed throughout the stadium, including 12 4K and 8K cameras to capture close-up shots during the game. The 4K cameras will be controlled robotically from the stadium concourse levels, while two Sony 8K cameras will be fixed on robotic gimbals from the lower field.”

Citing COVID-19 as the reason where “alternative production facilities” will be the method in producing the event; which means there will be many sterilized spaces  on West 57th St. in Manhattan at the CBS Broadcast Center… which tells me a lot of things. I have 3 points to make:

Continue reading

The Lack of Joy with Email

I used to love email. I seriously thought it was more professional, more methodological and more formal and more meaningful than text/SMS or chat/IM, etc.

Other groups of people who either do not respect the medium or have given email a bad name.

I thought owning a BlackBerry was cool, but one of the big flaws was I couldn’t have full IMAP email, then I went onto the iPhone in December 2011 and realized how cool it was to have an entire inbox on the palm of my hand like I could on my desktop. (POP3 is perfect for small email servers or the ol days of dialup, where once it downloads onto the client, it “pops” away off the server, is the non technical way to describe it.

But with everything, joy disappears. The alleged honeymoon period lasted a decade plus.

It wasn’t necessarily the tech, but the people on the other end. Apparently I showed too much love to email that people took advantage of it.

In the last year it got worse. People in professional circles would send me longwinded emails with up to twenty sentences per paragraph. The biggest pet-peeve was the excessive, liberal use of High Priority, and was used completely indiscriminately. It was more of “Look At Me, Smell my ugly Pits! I am so great you must drop everything and you’ll get back to me!” Worse was professionals would accidentally send out emails; with the failed attempt to recall emails –  assuming I am an Exchange shop (of which I am not), and that the email would magically disappear. Do these people realize no one is using Outlook in the masses as much?  In fact I haven’t used Outlook in years! I still liked the overall Outlook interface; but do not support the O365 approach.

I also had clashed with the millennials, and intra-generation fights.  One former professional was so overwhelmed of a 2 page email, despite it being properly written with appropriate sentences per graphs and it being outlined. Younger people today treat email like SMS or chats; I find that alarming because this leads to a culture of inappropriate communications and if you mix Slack-talk in email, it can really bite you if a discovery is required. I think chat-like emails are more apparent then than back in the 90s or early 00s by boomers and Gen Xers.

The biggest pet-peeve was the excessive, liberal use of High Priority, and was used completely indiscriminately. It was more of “Look At Me, Smell my ugly Pits! I am so great you must drop everything and you’ll get back to me!”

Oh the multi-thread replies… that was the worse. If developments occurred over the weekend, I had “professionals”; would literally react to every reply. If say there was an instance where I had  threads, both were responsive replies, one was a correction to the other, the person would still react to all three and personalize each one. This individual was under high stress and was known in circles for being a chain-smoker. The job didn’t need to be high-stress to begin with.

The joy is no longer. There is no sense of happiness as much as I used to. This isn’t because my responsibilities changed; it’s the other groups of people who either do not respect the medium or have given it a bad name.

Introducing… The Spokesman Podcast

If anyone who knows the history of telephony, the brand name of a loudspeaker tided to an ole Ma Bell phone was called the Spokesman. And unlike a Polycom, the quality may had been suspicious. It was designed to listen in to “morning calls” from Wall Street firms. In fact groups of people would huddle to what was nicknamed “the squawk box”…

This ol device was what inspired CNBC to introduce a “pre game show” in 1995.

25 years later, your’s truly wants to record about 5 minutes of hard-news plus no-so real time quotes and numbers and snatch that degraded brand that is like chewing gum for day traders into ear candy for people who live to crunch numbers… and no I am not talking about the ol Macintosh kids game. I love using that as an idiom.

If I learned anything during COVID19, there is not enough analysis; likewise there is not enough information. Back when Squawk aired on CNBC, it was in the early days of the web; and not everyone had access to the overall Internet. Today it’s so efffing hard to find detailed market information that The Wall Street Journal used to put in the C-section. Our Presidant doesn’t care about the more long term indicies that are measured like a pool; all he cares about is the market of public opinion – The Dow.

Once a week, on my YouTube, my SoundCloud, etc, you’ll hear roughly 5 minutes of reading stories of the previous night’s earnings reports; before the opening bell market reports; and little on politics. Tech stories may dominate the lead.

The audience is for people who follow markets, understand business; understand the over-zealous “profit” culture of todays corporate governance, and trying to put a news story one reads on the web into something that’s broadcast-able; and covering beats that the traditional cable business channels left behind ages ago.

I don’t want to be your daily teacher. I want help you be informed on weekly and daily basis.

Our principal sponsor is Techie Crafts

*