Open-source intelligence(OSINT) is a method of using open source tools to collect information and analyze them for a specific purpose. OSINT can be very helpful for hackers to use to garner data about particular organizations. Today, using open-sources like bing, google, yahoo, etc.., for data gathering is one of the important steps for reconnaissance and this is a common task. It should be a tool to automate this routine. One of the best tools in this field is The OWASP Maryam.
OWASP Maryam is a modular/optional open-source framework based on OSINT and data gathering. Maryam is written in the Python programming language and has been designed to provide a powerful environment to harvest data from open-sources and search engines and collect data quickly and thoroughly.
If you want
- Extracts Emails, Docs, Subdomains, Social networks from search engines
- Extracts Links, CSS and JS files, CDN links, Emails, Keywords from Web Source
- Find and Brute force DNS, TLD, and important directories and files
- Crawl Web Pages and search your RegExp
- Identify WebApps, WAF, Interesting and important files
Maryam requires Python 3.8+ and for package installation also uses python package manager PiPl(pip).
The repository can be loaded using the following command:
git clone https://github.com/saeeddhqan/maryam.git cd maryam
The next step is to install the requirements:
pip3 install -r requirements
The installation is finished and you can run with:
If it already exists, these commands remove the old version completely and replace it with the new version. From the Maryam install directory:
cd .. rm -rf maryam git clone https://github.com/saeeddhqan/maryam.git cd maryam ./maryam`
And these commands, update the remote URL of the current repository. From the Maryam install directory:
git remote set-url origin https://github.com/saeeddhqan/maryam.git git reset --hard HEAD~1 git pull
Currently, Maryam has three types of modules:
In the section of Footprint, we have modules to Crawl, Identify, Gather, and analyze.
What can be done with footprint modules?
- Identify Web Applications, frameworks, plugins, languages, and server operating systems.
- Search in web pages and find emails, usernames, errors, meta tags, and anything else that interest.
- Search your regular expression on web content and get the result. It’s very helpful to find interesting things on web content and it’s not limited to one page.
- Fast brute force attack to identify subdomains, files, directories, and TLDs with thread supporting.
- Find web entry points like forms, inputs, URLs with params, get requests, and post requests to fuzzing attack.
- Detect Web Application Firewalls(WAF) with 50 payloads.
OSINT modules are used to find emails, documentations, DNS names, and social networks. Almost all of the OSINT modules use search engines except crawler.
What can be done with OSINT modules?
- Extracts Emails(with 10 sources to search).
- Documentations(pdf, csv, txt, xlsx, ..) with 10 sources to search.
- DNS names with 24 sources to search(All of the sources are free).
- Social networks.
- Extracts links(in scope, out scope), comments, CSS and JS files, CDN links, emails, docs, and media files from web pages.
Search modules have been created to reduce the time it takes to search for free resources. Users can search on social networks, certificates, images, news, etc.
What can be done with search modules?
- Search in best search engines without API keys.
- Search in common social networks and find peoples, hashtags, and status.
- Search to find images, news, websites, etc.
help <command-name>for help your command
set <option-name> <value>
set VERBOSITY 2
var <$name> <value>
var $hunter_key XXXXXXXXXXXXXXXX
- To use it, write
set HUNTER_KEY $hunter_key
- To show all of variables write
- To delete a variable write
var delete <var-name>
To get report from modules output
- Set ‘output’ option with True:
set output True
- Or use ‘–output switch’:
wapps -d domain.com --output
- And next, use
report <format> <file-name-for-output> <module-name>
report <format> pdf_docs osint/docs_search localhost
- See the options
- And set the options
set TIMEOUT 2.5
set RAND_AGENT true
- To searching a RegExp to the web pages
./maryam crawl_pages -d example.com -r https?://[A-z0-9\._\/\-?=]+
OR without entering to the framework
./maryam -e crawl_pages -d domain.tld -r "https?://[A-z0-9\._\/\-?=]+" --limit 2
- Gain pdf files from search engine
docs_search -q amazon -f pdf -e google,bing,metacrawler --thread 3
- DNS brute force
./maryam dnsbrute -d example.com --thread 10 --wordlist mylist.txt
- Extract links and save data and get JSON report
./maryam -e crawler -d <DOMAIN> --output ./maryam -e report json output_file_name osint/crawler
Maryam (this link opens in a new window) by saeeddhqan (this link opens in a new window)
Maryam: Open-source Intelligence(OSINT) Framework