Advanced Persistent Threat Activity Exploiting Managed Service Providers

Network Systems

The National Cybersecurity and Communications Integration Center (NCCIC) is aware of ongoing APT actor activity attempting to infiltrate the networks of global managed service providers (MSPs). Since May 2016, APT actors have used various tactics, techniques, and procedures (TTPs) for the purposes of cyber espionage and intellectual property theft. APT actors have targeted victims in several U.S. critical infrastructure sectors, including Information Technology (IT), Energy, Healthcare and Public Health, Communications, and Critical Manufacturing.

This Technical Alert (TA) provides information and guidance to assist MSP customer network and system administrators with the detection of malicious activity on their networks and systems and the mitigation of associated risks. This TA includes an overview of TTPs used by APT actors in MSP network environments, recommended mitigation techniques, and information on reporting incidents.

MSPs provide remote management of customer IT and end-user systems. The number of organizations using MSPs has grown significantly over recent years because MSPs allow their customers to scale and support their network environments at a lower cost than financing these resources internally. MSPs generally have direct and unfettered access to their customers’ networks, and may store customer data on their own internal infrastructure. By servicing a large number of customers, MSPs can achieve significant economies of scale. However, a compromise in one part of an MSP’s network can spread globally, affecting other customers and introducing risk.

Using an MSP significantly increases an organization’s virtual enterprise infrastructure footprint and its number of privileged accounts, creating a larger attack surface for cyber criminals and nation-state actors. By using compromised legitimate MSP credentials (e.g., administration, domain, user), APT actors can move bidirectionally between an MSP and its customers’ shared networks. Bidirectional movement between networks allows APT actors to easily obfuscate detection measures and maintain a presence on victims’ networks.

Note: NCCIC previously released information related to this activity in Alert TA17-117A: Intrusions Affecting Multiple Victims Across Multiple Sectors published on April 27, 2017, which includes indicators of compromise, signatures, suggested detection methods, and recommended mitigation techniques.

Technical Details

APT

APT actors use a range of “living off the land” techniques to maintain anonymity while conducting their attacks. These techniques include using legitimate credentials and trusted off-the-shelf applications and pre-installed system tools present in MSP customer networks.

Pre-installed system tools, such as command line scripts, are very common and used by system administrators for legitimate processes. Command line scripts are used to discover accounts and remote systems.

PowerSploit is a repository of Microsoft PowerShell and Visual Basic scripts and uses system commands such as netsh. PowerSploit, originally developed as a legitimate penetration testing tool, is widely misused by APT actors. These scripts often cannot be blocked because they are legitimate tools, so APT actors can use them and remain undetected on victim networks. Although network defenders can generate log files, APT actors’ use of legitimate scripts makes it difficult to identify system anomalies and other malicious activity.

When APT actors use system tools and common cloud services, it can also be difficult for network defenders to detect data exfiltration. APT actors have been observed using Robocopy—a Microsoft command line tool—to transfer exfiltrated and archived data from MSP client networks back through MSP network environments. Additionally, APT actors have been observed using legitimate PuTTY Secure Copy Client functions, allowing them to transfer stolen data securely and directly to third-party systems.

A successful network intrusion can have severe impacts to the affected organization, particularly if the compromise becomes public. Possible impacts include

• Temporary or permanent loss of sensitive or proprietary information,
• Disruption to regular operations,
• Financial losses to restore systems and files, and
• Potential harm to the organization’s reputation.

Detection

Organizations should configure system logs to detect incidents and to identify the type and scope of malicious activity. Properly configured logs enable rapid containment and appropriate response.

Response

An organization’s ability to rapidly respond to and recover from an incident begins with the development of an incident response capability. An organization’s response capability should focus on being prepared to handle the most common attack vectors (e.g., spearphishing, malicious web content, credential theft). In general, organizations should prepare by

• Establishing and periodically updating an incident response plan.
• Establishing written guidelines that prioritize incidents based on mission impact, so that an appropriate response can be initiated.
• Developing procedures and out-of-band lines of communication to handle incident reporting for internal and external relationships.
• Exercising incident response measures for various intrusion scenarios regularly, as part of a training regime.
• Committing to an effort that secures the endpoint and network infrastructure: prevention is less costly and more effective than reacting after an incident.

Mitigation

Manage Supply Chain Risk

MSP clients that do not conduct the majority of their own network defense should work with their MSP to determine what they can expect in terms of security. MSP clients should understand the supply chain risk associated with their MSP. Organizations should manage risk equally across their security, legal, and procurement groups. MSP clients should also refer to cloud security guidance from the National Institute of Standards and Technology to learn about MSP terms of service, architecture, security controls, and risks associated with cloud computing and data protection.[1] [2] [3]

Architecture

Restricting access to networks and systems is critical to containing an APT actor’s movement. Provided below are key items that organizations should implement and periodically audit to ensure their network environment’s physical and logical architecture limits an APT actor’s visibility and access.

Virtual Private Network Connection Recommendations

• Use a dedicated Virtual Private Network (VPN) for MSP connection. The organization’s local network should connect to the MSP via a dedicated VPN. The VPN should use certificate-based authentication and be hosted on its own device.
• Terminate VPN within a demilitarized zone (DMZ). The VPN should terminate within a DMZ that is isolated from the internal network. Physical systems used within the DMZ should not be used on or for the internal network.
• Restrict VPN traffic to and from MSP. Access to and from the VPN should be confined to only those networks and protocols needed for service. All other internal networks and protocols should be blocked. At a minimum, all failed attempts should be logged.
• Update VPN authentication certificates annually. Update the certificates used to establish the VPN connection no less than annually. Consider rotating VPN authentication certificates every six months.
• Ensure VPN connections are logged, centrally managed, and reviewed. All VPN connection attempts should be logged in a central location. Investigate connections using dedicated certificates to confirm they are legitimate.

Network Architecture Recommendations

• Ensure internet-facing networks reside on separate physical systems. All internet-accessible network zones (e.g., perimeter network, DMZ) should reside on their own physical systems, including the security devices used to protect the network environment.
• Separate internal networks by function, location, and risk profile. Internal networks should be segmented by function, location, and/or enterprise workgroup. All communication between networks should use Access Control Lists and security groups to implement restrictions.
• Use firewalls to protect server(s) and designated high-risk networks. Firewalls should reside at the perimeter of high-risk networks, including those hosting servers. Access to these networks should be properly restricted. Organizations should enable logging, using a centrally managed logging system.
• Configure and enable private Virtual Local Area Networks (VLANs). Enable private VLANs and group them according to system function or user workgroup.
• Implement host firewalls. In addition to the physical firewalls in place at network boundaries, hosts should also be equipped and configured with host-level firewalls to restrict communications from other workstations (this decreases workstation-to-workstation communication).

Network Service Restriction Recommendations

• Only permit authorized network services outbound from the internal network. Restrict outbound network traffic to only well-known web browsing services (e.g., Transmission Control Protocol [TCP]/80, TCP/443). In addition, monitor outbound traffic to ensure the ports associated with encrypted traffic are not sending unencrypted traffic.
• Ensure internal and external Domain Name System (DNS) queries are performed by dedicated servers. All systems should leverage dedicated internal DNS servers for their queries. Ensure that DNS queries for external hosts using User Datagram Protocol (UDP)/53 are permitted for only these hosts and are filtered through a DNS reputation service, and that outbound UDP/53 network traffic by all other systems is denied. Ensure that TCP/53 is not permitted by any system within the network environment. All attempts to use TCP/53 and UDP/53 should be centrally logged and investigated.
• Restrict access to unauthorized public file shares. Access to public file shares that are not used by the organization—such as Dropbox, Google Drive, and OneDrive—should be denied. Attempts to access public file share sites should be centrally logged and investigated. Recommended additional action: monitor all egress traffic for possible exfiltration of data.
• Disable or block all network services that are not required at network boundary. Only those services needed to operate should be enabled and/or authorized at network boundaries. These services are typically limited to TCP/137, TCP/139, and TCP/445. Additional services may be needed, depending on the network environment, these should be tightly controlled to only send and receive from certain allow listed Internet Protocol addresses, if possible.

Authentication, Authorization, and Accounting

Compromised account credentials continue to be the number one way threat actors are able to penetrate a network environment. The accounts organizations create for MSPs increase the risk of credential compromise, as MSP accounts typically require elevated access. It is important organizations’ adhere to best practices for password and permission management, as this can severely limit a threat actor’s ability to access and move laterally across a network. Provided below are key items organizations should implement and routinely audit to ensure these risks are mitigated.

Account Configuration Recommendations

• Ensure MSP accounts are not assigned to administrator groups. MSP accounts should not be assigned to the Enterprise Administrator (EA) or Domain Administrator (DA) groups.
• Restrict MSP accounts to only the systems they manage. Place systems in security groups and only grant MSP account access as required. Administrator access to these systems should be avoided when possible.
• Ensure MSP account passwords adhere to organizational policies. Organizational password policies should be applied to MSP accounts. These policies include complexity, life, lockout, and logging.
• Use service accounts for MSP agents and services. If an MSP requires the installation of an agent or other local service, create service accounts for this purpose. Disable interactive logon for these accounts.
• Restrict MSP accounts by time and/or date. Set expiration dates reflecting the end of the contract on accounts used by MSPs when those accounts are created or renewed. Additionally, if MSP services are only required during business hours, time restrictions should also be enabled and set accordingly. Consider keeping MSP accounts disabled until they are needed and disabling them once the work is completed.
• Use a network architecture that includes account tiering. By using an account tiering structure, higher privileged accounts will never have access or be found on lower privileged layers of the network. This keeps EA and DA level accounts on the higher, more protected tiers of the network. Ensure that EA and DA accounts are removed from local administrator groups on workstations.

Logging Configuration Recommendations

• Enable logging on all network systems and devices and send logs to a central location. All network systems and devices should have their logging features enabled. Logs should be stored both locally and centrally to ensure they are preserved in the event of a network failure. Logs should also be backed up regularly and stored in a safe location.
• Ensure central log servers reside in an enclave separate from other servers and workstations. Log servers should be isolated from the internet and network environment to further protect them from compromise. The firewall at the internal network boundary should only permit necessary services (e.g., UDP/514).
• Configure local logs to store no less than seven days of log data. The default threshold for local logging is typically three days or a certain file size (e.g., 5 MB). Configure local logs to store no less than seven days of log data. Seven days of logs will cover the additional time in which problems may not be identified, such as holidays. In the event that only size thresholds are available, NCCIC recommends that this parameter be set to a large value (e.g., 512MB to1024MB) to ensure that events requiring a high amount of log data, such as brute force attacks, can be adequately captured.
• Configure central logs to store no less than one year of log data. Central log servers should store no less than a year’s worth of data prior to being rolled off. Consider increasing this capacity to two years, if possible.
• Install and properly configure a Security Information and Event Management (SIEM) appliance. Install a SIEM appliance within the log server enclave. Configure the SIEM appliance to alert on anomalous activity identified by specific events and on significant derivations from baselined activity.
• Enable PowerShell logging. Organizations that use Microsoft PowerShell should ensure it is upgraded the latest version (minimum version 5) to use the added security of advanced logging and to ensure these logs are being captured and analyzed. PowerShell’s features include advanced logging, interaction with application allow listing (if using Microsoft’s AppLocker), constrained language mode, and advanced malicious detection with Antimalware Scan Interface. These features will help protect an organization’s network by limiting what scripts can be run, logging all executed commands, and scanning all scripts for known malicious behaviors.
• Establish and implement a log review process. Logs that go unanalyzed are useless. It is critical to network defense that organizations establish a regular cycle for reviewing logs and developing analytics to identify patterns.

Operational Controls

Building a sound architecture supported by strong technical controls is only the first part to protecting a network environment. It is just as critical that organizations continuously monitor their systems, update configurations to reflect changes in their network environment, and maintain relationships with MSPs. Listed below are key operational controls organizations should incorporate for protection from threats.

Operational Control Recommendations

• Create a baseline for system and network behavior. System, network, and account behavior should be baselined to make it easier to track anomalies within the collected logs. Without this baseline, network administrators will not be able to identify the “normal” behaviors for systems, network traffic, and accounts.
• Review network device configurations every six months. No less than every six months, review the active configurations of network devices for unauthorized settings (consider reviewing more frequently). Baseline configurations and their checksums should be stored in a secure location and be used to validate files.
• Review network environment Group Policy Objects (GPOs) every six months. No less than every six months, review GPOs for unauthorized settings (consider reviewing more frequently). Baseline configurations and their checksums should be stored in a secure location and be used to validate files.
• Continuously monitor and investigate SIEM appliance alerts. The SIEM appliance should be continuously monitored for alerts. All events should be investigated and documented for future reference.
• Periodically review SIEM alert thresholds. Review SIEM appliance alert thresholds no less than every three months. Thresholds should be updated to reflect changes, such as new systems, activity variations, and new or old services being used within the network environment.
• Review privileged account groups weekly. Review privileged account groups—such as DAs and EAs—no less than weekly to identify any unauthorized modifications. Consider implementing automated monitoring for these groups.
• Disable or remove inactive accounts. Periodically monitor accounts for activity and disable or remove accounts that have not been active within a certain period, not to exceed 30 days. Consider including account management into the employee onboarding and offboarding processes.
• Regularly update software and operating systems. Ensuring that operating systems and software is up-to-date is critical for taking advantage of a vendor’s latest security offerings. These offerings can include mitigating known vulnerabilities and offering new protections (e.g., credential protections, increased logging, forcing signed software).

It is important to note that—while the recommendations provided in this TA aim at preventing the initial attack vectors and the spread of any malicious activity—there is no single solution to protecting and defending a network. NCCIC recommends network defenders use a defense-in-depth strategy to increase the odds of successfully identifying an intrusion, stopping malware, and disrupting threat actor activity. The goal is to make it as difficult as possible for an attacker to be successful and to force them to use methods that are easier to detect with higher operational costs.

Report Unauthorized Network Access

Contact DHS or your local FBI office immediately. To report an intrusion and request resources for incident response or technical assistance, contact NCCIC at (NCCICCustomerService@hq.dhs.gov or 888-282-0870), FBI through a local field office, or the FBI’s Cyber Division (CyWatch@fbi.gov or 855-292-3937).

October, 3 2018: Initial version