Maryam: Modular Open Source Framework based on OSINT

Open-source intelligence(OSINT) is a method of using open source tools to collect information and analyze them for a specific purpose. OSINT can be very helpful for hackers to use to garner data about particular organizations. Today, using open-sources like bing, google, yahoo, etc.., for data gathering is one of the important steps for reconnaissance and this is a common task. It should be a tool to automate this routine. One of the best tools in this field is ​The OWASP Maryam.

OWASP Maryam is a modular/optional open-source framework based on OSINT and data gathering. Maryam is written in the Python programming language and has been designed to provide a powerful environment to harvest data from open-sources and search engines and collect data quickly and thoroughly.

If you want

• Extracts Emails, Docs, Subdomains, Social networks from search engines
• Extracts Links, CSS and JS files, CDN links, Emails, Keywords from Web Source
• Find and Brute force DNS, TLD, and important directories and files
• Crawl Web Pages and search your RegExp
• Identify WebApps, WAF, Interesting and important files

Use Maryam

Getting Started

Prerequisites

Maryam requires Python 3.8+ and for package installation also uses python package manager PiPl(pip).

Requires

• requests

Install

The repository can be loaded using the following command:

git clone https://github.com/saeeddhqan/maryam.git
cd maryam

The next step is to install the requirements:

pip3 install -r requirements

The installation is finished and you can run with:

./maryam

Update

If it already exists, these commands remove the old version completely and replace it with the new version. From the Maryam install directory:

cd ..
rm -rf maryam
git clone https://github.com/saeeddhqan/maryam.git
cd maryam
./maryam`

And these commands, update the remote URL of the current repository. From the Maryam install directory:

git remote set-url origin https://github.com/saeeddhqan/maryam.git
git reset --hard HEAD~1
git pull

Modules

Currently, Maryam has three types of modules:

• Footprint
• OSINT
• Search

Footprint

In the section of Footprint, we have modules to Crawl, Identify, Gather, and analyze.

What can be done with footprint modules?

• Identify Web Applications, frameworks, plugins, languages, and server operating systems.
• Search in web pages and find emails, usernames, errors, meta tags, and anything else that interest.
• Search your regular expression on web content and get the result. It’s very helpful to find interesting things on web content and it’s not limited to one page.
• Fast brute force attack to identify subdomains, files, directories, and TLDs with thread supporting.
• Find web entry points like forms, inputs, URLs with params, get requests, and post requests to fuzzing attack.
• Detect Web Application Firewalls(WAF) with 50 payloads.

OSINT

OSINT modules are used to find emails, documentations, DNS names, and social networks. Almost all of the OSINT modules use search engines except crawler.

What can be done with OSINT modules?

• Extracts Emails(with 10 sources to search).
• Documentations(pdf, csv, txt, xlsx, ..) with 10 sources to search.
• DNS names with 24 sources to search(All of the sources are free).
• Social networks.
• Extracts links(in scope, out scope), comments, CSS and JS files, CDN links, emails, docs, and media files from web pages.

Search

Search modules have been created to reduce the time it takes to search for free resources. Users can search on social networks, certificates, images, news, etc.

What can be done with search modules?

• Search in best search engines without API keys.
• Search in common social networks and find peoples, hashtags, and status.
• Search to find images, news, websites, etc.

Quick Guide

Help options

• help or ?
• or help <command-name> for help your command

Show modules

• show modules

Use modules

• <module-name> --help

Show framework options

• show options

Set framework options

• set <option-name> <value>
• Example set VERBOSITY 2

To add a variable

• var <$name> <value>
• Example var $hunter_key XXXXXXXXXXXXXXXX
• To use it, write set HUNTER_KEY $hunter_key
• To show all of variables write var list command
• To delete a variable write var delete <var-name>

To get report from modules output

• Set ‘output’ option with True: set output True
• Or use ‘–output switch’: wapps -d domain.com --output
• And next, use report command: report <format> <file-name-for-output> <module-name>
• Rxample report <format> pdf_docs osint/docs_search localhost

Show history commands

• history all

Modules search

• search <string>

Run shell command

• shell <command> or ! <command> or <command>

Reload all modules

• reload

Config the connections

• See the options show options
• And set the options set TIMEOUT 2.5

To use random user-agent

• set RAND_AGENT true

Examples

• To searching a RegExp to the web pages

./maryam
crawl_pages -d example.com -r https?://[A-z0-9\._\/\-?=]+

OR without entering to the framework

./maryam -e crawl_pages -d domain.tld -r "https?://[A-z0-9\._\/\-?=]+" --limit 2

• Gain pdf files from search engine

docs_search -q amazon -f pdf -e google,bing,metacrawler --thread 3

• DNS brute force

./maryam

dnsbrute -d example.com --thread 10 --wordlist mylist.txt

• Extract links and save data and get JSON report

./maryam -e crawler -d <DOMAIN> --output
./maryam -e report json output_file_name osint/crawler

email_search tool

crawl_pages tool

onion_search tool

osint/docs_search

dns_search tool