Tag Archives: ransomware


Ransomware: Lessons Learned from One Food Company’s Experience

By Food Safety Tech Staff
No Comments

In fall 2021, G&J Pepsi-Cola Bottlers Inc, came face-to-face with a potential ransomware attack and was able to avert it. We spoke with G&J’s enterprise infrastructure director, Eric McKinney, and cybersecurity engineer, Rory Crabbe, to learn more about how they detected and responded to the attack, the steps they have taken to strengthen their cybersecurity, and what advice they have for other food companies in the wake of the near catastrophe.

What happened to G&J back in 2021, and when did you realize something was wrong?

McKinney: Around Labor Day of 2021, we received a really weird call. The callers were acting as if they were friends looking out for our best interest, and they alerted us to the fact that there may be compromises to our system. They showed us a spreadsheet of usernames in our active directory to verify that they were in our systems, and they said we could pay them to prevent an attack. We did not engage with them further—and we think they may have been part of it—but we believed that something was happening.

Eric McKinney
Eric McKinney

We went through all of our servers—we don’t have a large footprint, because we are a cloud first organization—but we did detect some software that should not have been installed on a couple of our servers. We removed that immediately, but we were unable to find the beacons that they leave behind that act as triggers to start encrypting your files.

We made the decisions that if anything happened, we were not going to negotiate, we were not going to try to get our systems back, we were going to shut everything down and roll back. I put myself on call and sure enough I got a call two days later at 3:00 a.m. from one of our people. He was logging in remotely to a server and he said, “Something don’t look right.” I go to his screen and I immediately see the locked files and realize this is really happening.

The thing that saved us ultimately is we use native platform backups. We use Microsoft Azure. So we immediately shut everything down and started rolling back our systems as far back as we could go. Those backup files were not compromised because we don’t leverage backups that tie to a file system within a server. The only way you can touch them is if you have our Cloud credentials, which are all multi-factored.

How did this affect operations?

McKinney: The net impact was our critical systems were down for about seven to eight hours, and we were recovering PCs for almost a week—there were 100 to 150 PCs that were impacted as it continued to move laterally through our organization, and we had to get them all flushed out. We had to roll the system back two weeks, so we lost two weeks of data. That impacted the accounting team the most.

We did experience an event—it was not an almost event. But we never lost a single case of sales and we never paid a single dollar. We took everyone’s computers and blew them away, handed them right back to them and said you’re starting fresh. Fortunately, this only affected employees’ files. They could still get their emails and the things that were in OneDrive.

The things that really worked in our favor were our Cloud-first strategy and getting away from a legacy client architecture. We were still able to communicate. We could send emails, we could set up Teams and we had all the tools to coordinate and get out of this and recover as quickly as we did. The second thing was having those native platform-based backups.

How did this change your digital and cybersecurity strategies?

McKinney: We were doing weekly backups, now we back up every day. And these are full system backups, which means that if you hit restore, the whole system lights back up not just the data but also your operating system that it runs on.

Crabbe: We also reached out to a lot of companies, including Arctic Wolf, who we ultimately began working with to help us figure out what we didn’t know. We worked with them to go through our environment and come up with ideas on how to improve. We are a big Microsoft shop, and we started utilizing a lot of the native tools that we already had such as Defender for Endpoint and the security portal. This addressed a lot of the low hanging fruit, such as automatic updates and not allowing outside vendors to contact us without going through a vetting process.

Rory Crabbe
Rory Crabbe

Arctic Wolf went through our system and sent us a list of recommendations, and a lot of what we did involved utilizing the native tools that we already had, shoring up our defenses, making sure the backups work and creating a disaster recovery plan.

McKinney:  We quickly went from being a business of convenience, where we said, “let’s allow USB drives,” to changing all of our technical policies by turning on all of our attack surface reduction rules. We blocked all logins from outside the U.S. and brought in new team members dedicated to cybersecurity.

I have some self-confidence issues due to this attack because your failures are put on display, and there is a feeling that if you were doing a better job this would have been prevented. But we were a very small team and we were responsible for cybersecurity, ERP (enterprise resource planning) initiatives, development initiatives, support and infrastructure initiatives and data initiatives. When you’re wearing all of these hats things do get missed, and in the end it ended up being one application update. One application patch was exposed, which set all of this off. in terms of where we’ve gotten better, we signed up with an MSP (managed service provider) to monitor our environment 24 hours a day seven days a week. In addition, these companies assist your team by keeping them up to date with the latest techniques and providing proactive communication on things that we should be doing to secure and protect our environment.

We’ve taken a lot of steps over the past two years and we still have a long way to go. We will never stop or become complacent.

There is a concern among some people that the Cloud is less secure, and it’s better to control your own servers. Is that a misconception?

Crabbe: When it’s on premise it is your responsibility. If something happens to your infrastructure, you’ve got to be on call and wake up to deal with that. So not only is the Cloud a reduction in personnel work; it’s also peace of mind. Microsoft has its own team of engineers, and they have physical security in place as well. The Azure building is protected by armed guards to protect the data from physical hackers. It’s a lot easier to apply security policies to something that’s in the Cloud because Microsoft can give you options for all kinds of things that you didn’t even know you needed. This makes it easier to visualize where you are and where you need to go.

McKinney: These are also publicly traded companies that have to follow all of the controls that come with being publicly traded. They’re going to do a better job than the one or two individuals that you have at your company who cannot work 24/7 365 days a year.

I appreciate you guys talking openly about this, because one of the issues that comes up in food defense and cybersecurity is people aren’t necessarily sharing information that could help others recognize vulnerabilities. Is it difficult to share this information?

McKinney: We didn’t want to talk about it for a long time. It’s hard to put your failures—or at least what is perceived as a failure—out there. But when you look around, you realize this can happen to anyone. It happened to MGM with all their resources. And one issue that isn’t discussed very often is, behind the business implications is an incredibly stressed out IT team that really is traumatized by an event like this.

In talking with others who have been through this, it’s often the most stressful thing that’s ever happened in their lives. It certainly is the most stressed out I’ve ever been. You’re thinking, I just cost my company millions of dollars. I shut down my business. We may not be able to get product to our people. So many things flash through your mind, and you really don’t want to talk about it or advertise it. Luckily for us, we had the right systems but most importantly we had really great executive support and great team members to help us recover.

When it comes to access management, companies have to balance convenience for their employees with the need for stringent security. Were employees understanding of the changes you had to make, and how did you communicate these changes in processes?

Crabbe: There was a lot of frustration with people saying this worked before, why can’t we do it now? One of the benefits of being a family-owned company is that we are a fairly small group, so we were able to deal with it on almost a case-by-case basis. We have an internal system that people can submit their issues or requests through, and we review them. For example, if somebody needs to move a device to a USB stick to take to an external vendor, we can look at that and say what alternatives do we have? Can we use OneDrive or another native tool to share that information? Does it have to be a USB stick? Or, if someone is going on vacation in Mexico, they can submit a ticket and we can allow them remote access from a specific country for a specific amount of time so they can log-in. We can tell them yes or no on a case-by-case basis and explain why we made the decision.

McKinney: This event also made us ask questions like, do we even need USB sticks? There are so many other tools we can use. A lot of the changes involved looking at more modern ways to collaborate. And a lot of that revolves around retraining and catching your workforce up with the new tools that we have available.

Based on your experience, what advice would you offer other companies?

McKinney: The IT spend in the food and beverage industry is typically small compared to industries like insurance or banking or health care. You need to capture all the signals from all your systems—emails being sent, open, received, etc.—and you must monitor those. Then you need the right algorithms and the right people to make sense of that data. If you are not able to maintain a large enough in-house team, investigate an MSP. They can ingest all the signals, funnel them and turn all that data into actionable items. Also, store your backups off site and limit access. Don’t store them with your production data.

Crabbe: Shore up your defenses using your native tools and create a disaster recovery plan. Those would be my two biggest recommendations for any company going forward. Dig deep and utilize what you’ve got. There’s probably a lot more available to you than you realize you have, and don’t be afraid to reach out to third-party vendors for help.



As Cyber Threats Evolve, Can Food Companies Keep Up?

By Maria Fontanazza
No Comments

The recent cyberattack that shut down meat supplier JBS should be a wakeup call to the food industry. These attacks are on the rise across industries, and food operations both large and small need to be prepared. In a Q&A with Food Safety Tech, Brent Johnson, partner at Holland & Hart, breaks down key areas of vulnerability and how companies in the food industry can take proactive steps to protect their operations and ultimately, the consumer.

Food Safety Tech: Given the recent cyberattack on JBS, how vulnerable are U.S. food companies, in general, to this type of attack? How prepared are companies right now?

Brent Johnson, Holland & Hart
Brent Johnson, partner, Holland & Hart

Brent Johnson: Food companies are in the same boat as other manufacturers. Cyber threats are constantly evolving and hackers are developing increasingly sophisticated delivery systems for ransomware. Food companies are obviously focused on making and delivering safe and compliant products and getting paid for them. Cybersecurity is important, but it’s difficult for manufacturers to devote the resources necessary to make their systems bulletproof when it’s an ancillary part of their overall operations and a cost driver. Unfortunately, hackers only have one job.

We tend to think of big tech and financial services companies as the prime targets for ransomware attacks because of the critical nature of their technology and data, but food companies are really no different. Plus, unlike tech companies and the financial services industry, food companies haven’t, as a general matter, developed the robust defenses necessary to thwart attacks, so they’re easier targets.

Food Safety Tech: What is the overall impact of a cyberattack on a food company, from both a business as well as a consumer safety perspective?

Johnson: It may come as a bit of a surprise to those who don’t work in the food industry, but food production (from slaughterhouses to finished products) is highly automated and data driven. That’s one of the lessons of the JBS ransomware attack. The attack shut down meat processing facilities across the United States and elsewhere. I work in Utah and the JBS Beef Plant in Hyrum was temporarily shut down. JBS cancelled two shifts at its meatpacking operation in Greeley, Colorado where my firm has a large presence as well, because of the ransomware attack. So, the impact on a food company’s business from a successful ransomware attack is dramatic.

On the consumer safety side, a ransomware attack that impacts automated safety systems would cause significant problems for a food manufacturer. Software controls much of the food industry’s safety systems—from sanitation (equipment washdowns and predictive maintenance) to traceability (possible pathogen contamination and recalls) to ingredient monitoring (including allergen detection). Every part of a food company’s production system is traced, tracked, and verified electronically. A ransomware attack on a food maker would very likely compromise the company’s ability to produce safe products.

Food Safety Tech: What proactive steps should food companies be taking to protect themselves against a cyberattack?

Johnson: I wish there was an easy and foolproof system for food companies to implement to protect against cyber attacks, but there isn’t. The threats are always changing. The Biden Administration’s recent memorandum to corporate executives and business leaders on strengthening cyber defenses is a good starting point, however. The White House’s Deputy National Security Adviser for Cyber and Emerging Tech, Anne Neuberger, reiterated the following “Five Best Practices” from President Biden’s executive order. These practices are multifactor authentication, endpoint detection and response, aggressive monitoring for malicious activities on the company’s networks and blocking them, data encryption, and the creation of a skilled cyber security team with the ability to train employees, detect threats and patch system vulnerabilities.

Food Safety Tech: Are there specific companies within the food industry that are especially susceptible?

Johnson: Not really. Hackers are opportunistic and look for the paths of least resistance. That said, as can be seen from the recent Colonial Pipeline and JBS ransomware attacks, hackers have transitioned from the early days of going after individuals and small businesses to whale hunting. The money is better.

It’s important to observe that the recent attacks have been directed at industries that present national infrastructure concerns (oil, the food supply). There’s no evidence of any involvement by a foreign government in these attacks, but it’s a fair question as to whether the hackers, themselves, expect that the federal government will step in at some point to assist the victims of cyber attacks financially due to their critical importance.

Food Safety Tech: Where do you see the issue of cybersecurity and cyberattacks related to the food industry headed in the future?

Johnson: Other than the certainty that the attacks will increase in both intensity and sophistication, I have no prediction. It’s not a time for complacency.