Last updated August 12, 2019

Meet Msticpy, A Python Defender Tool For Security Investigations & Hunting

Share

Published on June 25, 2019

by Ambika Choudhury

Microsoft Threat Intelligence Python Security Tools (msticpy), a package of tools which is meant to be used for security investigations and hunting, was released earlier this month. In this package, most of the tools were inspired by Jupyter notebooks which are repackaged into Python modules. Msticpy is OS-independent and requires Python packages such as pandas, bokeh, matplotlib, seaborn, urllib3, ipywidgets, numpy, iPython and scikit_learn, among others.

There are two main purposes of this package:

Reduce the clutter of code in notebooks making them easier to use and read.
Provide building blocks for future notebooks to make authoring them simpler and quicker.

Msticpy is organised into three main sub-packages, they are mentioned below

sectools

This is the security tools sub-package which is a Python security tool. It is used to provide help with data analysis or any other investigation. These are all focused on data transformation, data analysis or data enrichment. This subpackage contains several data processing modules and classes which are helpful for working on security investigations. Some of the modules are mentioned below

base64unpack: This is a Base64 and archive (gz, zip, tar) extractor which is designed to help decode obscured attack command lines along with HTTP request strings. The module basically identifies any base64 encoded strings and decode them.
iocextract: This module uses a set of built-in regular expressions to look for an Indicator of Compromise (IoC) patterns. The input can be a single string or a pandas dataframe with one or more columns specified as input.
vtlookup: This module is a Wrapper class around Virus Total API. Input can be a single IoC observable or a pandas DataFrame containing multiple observables.
geoip: This module is the geographic location lookup for IP addresses which is implemented as a generic class with support for different data providers. The model has two classes for different services, GeoLiteLookup and IPStackLookup.
eventcluster: This module is designed to be used to summarise large numbers of events into clusters of different patterns. The module contains functions to generate clusterable features from string data.
auditdextract: This module is used to load and decode Linux audit logs.

nbtools

This is the Jupyter-specific UI tools such as widgets and data display which are mostly presentation-layer tools concentrating on how to view or interact with the data. The modules are nbwidgets: This module groups common functionality such as list pickers, time boundary settings, saving and retrieving environment variables into a single line callable command.

nbdisplay: This module functions in implementing the common display of things like alerts.

data

This subpackage is used for data interfaces and query library for log and alert APIs including Azure Sentinel/Log Analytics, Microsoft Graph Security API and Microsoft Defender Advanced Threat Protection (MDATP).

Installation

This package requires Python version 3.6 or more.

To install type:

pip install msticpy

Or for the latest dev build, type:

pip install git+https://github.com/microsoft/msticpy

Outlook

Msticpy is an open source package and was initially developed to support Jupyter Notebook authoring for Azure Sentinel. This package is still in an early preview mode, so there are most likely to be bugs and possible API changes. Also, it is not yet fully optimised for performance.