Bug#1055284: RFP: harpoon -- CLI tool for open source and threat intelligence

[Antoine Beaupre]

Package: wnpp
Severity: wishlist

* Package name : harpoon
Version : no releases published https://github.com/Te-k/harpoon/issues/194
Upstream Contact: https://github.com/Te-k
* URL : https://github.com/Te-k/harpoon
* License : GPL-3
Programming Lang: Python
Description : CLI tool for open source and threat intelligence

harpoon doesn't have a long description upstream, but it's a recon
tool with various backends. i personnally want to use it to answer
questions like "who is this set of IPs doing nasty things to my
server?" looking for answers like reverse DNS, geolocation, ASN,
shodan information and so on.

harpoon can do a lot more though, by tapping into various data sources
like virustotal, haveibeenpwned, and many more.

i do not believe there is an equivalent in Debian. maybe metasploit
could be crafted to do *some* of this, but there's so many modules in
metasploit nowadays that it's hard to tell. there *is* a shodan
module, for example and there's a tool to do reverse DNS lookups, but
no ASN lookup I could find.

harpoon is designed with recon in mind specifically. there's even a
separate toolset called harpoontools that give commandline tools to a
basic set of tools:

https://github.com/Te-k/harpoontools

Just that basically answers most of my needs!

Would be maintained under the python team, I suppose.

Code is not the cleanest (and due for a refactor) but has unit tests
and written by a friend of a friend.

[Antoine Beaupré]

Control: forwarded -1 https://github.com/Te-k/harpoon/issues/195

Oh, and another thing I forgot: the harpoon wiki names a few
alternatives to this.

https://github.com/Te-k/harpoon/wiki#other-tools

Of those, the following might be interesting:

https://github.com/kpcyrd/sn0int
https://github.com/aancw/Belati

Out of band, someone also mentioned nmap as an alternative already
packaged in Debian, but I figured nmap was a much more active tool than
what I am looking for here.

Finally, there's also this tool I found that does a *lot* of what I
want:

https://github.com/nitefood/asn

... but it's kind of a messy shell script, which is why I am hoping to
package harpoon instead.

[Antoine Beaupré]

Another thing I forgot, the author wrote a good guide on their blog in:

https://randhome.io/blog/2018/02/23/harpoon-an-osint-/-threat-intelligence-tool/

--
The steel horse fills a gap in modern life, it is an answer not only to
its needs, but also to its aspirations. It's quite certainly here to
stay.
- Le Vélocipède Illustré, 1869

[Antoine Beaupré]

I started working on a Docker container for this, based on a Debian
container. To simplify the install (e.g. not require rebuilding lxml), I
pre-install a bunch of Debian packages for Python dependencies, but a
*lot* are missing.

The following are present:

python3-bs4 \
python3-dateutil \
python3-dnspython \
python3-geoip2 \
python3-ipy \
python3-lxml \
python3-maxminddb \
python3-phonenumbers \
python3-pip \
python3-psycopg2 \
python3-pyasn \
python3-pygments \
python3-tz \
python3-requests \
python3-selenium \
python3-setproctitle \
python3-simplejson \
python3-telethon \
python3-virustotal-api \

But the following are not:

"OTXv2",
"PyGitHub>=1.55",
"archiveis",
"censys==2.2.0",
"consolemd==0.5.1",
"fullcontact.py",
"greynoise>=1.2.0",
"passivetotal>=2.5.9",
"pybinaryedge==0.5",
"pycrtsh==0.3.11",
"pyhashlookup==1.2.1",
"pyhunter",
"pymisp==2.4.159",
"pypdns==1.3",
"pypermacc==0.1.1",
"pysafebrowsing==0.1.2",
"pysecuritytrails==0.1.3",
"pythreatgrid2==0.1.1",
"shodan",
"spyonweb==0.1",
"threatminer==1.0",
"tweepy>=3.8.0",
"zetalytics-api==1.0.1",

The following are available in Debian, but somehow pinned or declared in
a way that makes pip still install them

"beautifulsoup4==4.11.1",
"configparser",
"lxml==4.9.2",
"phonenumbers==8.12.4",
"simplejson==3.17.6"
"telethon==0.19.1.6",

So if we actually want everything in harpoon to actually work, there is
a *lot* of work to package dependencies here.

--
You can't get to the moon by climbing successively taller trees.
- Akin's Laws of Spacecraft Design

[Antoine Beaupré]

I've also looked for more alternatives, thinking "surely someone else
fixed this in a more simple way". I looked mostly at
https://kali.org/tools for now.

Out of those, i have selected the following tools:

https://www.maltego.com/
https://github.com/smicallef/spiderfoot
https://github.com/SparrowOchon/dnsenum2
https://github.com/GoVanguard/legion
https://github.com/owasp-amass/amass

A.
--
The problem is not a lack of highly educated workers, the problem is a
lack of highly educated workers willing to work for the minimum wage or
lower in the U.S. Costs are driving outsourcing, not the quality of
American schools. - Scott Kirwin, IT Professionals Association

[Antoine Beaupré]

More notes on my use case... harpoon serves this poorly, as I explain
upstream here:

https://github.com/Te-k/harpoon/issues/190#issuecomment-1798667942

Basically, harpoon has a good `intel` command to lookup the reputation
of a single IP address on multiple plugins. But that's it: it works only
a on a *single* IP address, not *multiple*.

Also, it doesn't seem like it works very reliably on all backends. For
example, even though the `vt` command works, it doesn't seem to hookup
with the `intel` command.

Effectively, what harpoon fundamentally is is a wrapper around many
backend services. The most interesting I have found are:

* asn and the asncount command in harpoontools: ASN to name mappings
from https://ftp.ripe.net/ripe/asnames/asn.txt,
ftp://archive.routeviews.org/datapath/YYYYMM/ribs/XXXX
http://archive.routeviews.org/bgpdata/%d.%02d/RIBS (from pyasn
package)

* dns: simple reverse/forward DNS checks, not in intel either

* ipinfo.io: provides ASN lookups, VPN/Tor/Proxy checks

* pulsedive.com: tor, blocklists, cryptomining, threat reports

* threatminer.org: unclear

* tor: check tor exit lists, pulls
https://check.torproject.org/torbulkexitlist on each call (!)

* urlhaus.abuse.ch: more malware oriented, https://threatfox.abuse.ch
more interesting but not implemented

* virustotal (vt command): domain, IP reputation, history, API, free to
use but rate limited unless a premium account is requested (note that
there's a separate RFP for the vt-cli commandline, #1034826)

Then there's a bunch more interesting resources that are not implemented
yet but that are still interesting:

* criminalip.io: abuse records, botnet, Tor, VPN, Proxy, Hosting, CDN,
mobile, scanner checks, requires plan to do more
https://github.com/Te-k/harpoon/issues/184

* crowdsec.net: federated collaborative IP reporting, free daily data
source https://github.com/Te-k/harpoon/issues/199

* project honeypot: lists IPs that fell into a honeypot,
https://github.com/Te-k/harpoon/issues/64

* proxycheck.io: simple API, Tor, Proxy, "type" (business, wireless,
residential, etc), VPN check,
https://github.com/Te-k/harpoon/issues/110

More services I found in my search that could be useful to tap for extra
confirmations:

* abuseipdb.com: abuse reports

* dronebl.org: abuse reports of "infected machines", RBL

* check.spamhaus.org: classic spammer database, RBL

Alright, that's what I got so far!

a.

--
The destiny of Earthseed is to take root among the stars.
- Octavia Butler

[Antoine Beaupré]

On 2023-11-03 10:30:49, Antoine Beaupré wrote:
> https://github.com/kpcyrd/sn0int
> https://github.com/aancw/Belati
> https://github.com/nitefood/asn

Another, more minimalistic tool is simply a DNS resolver with support
for looking up ASNs:

https://github.com/natesales/q

Specifically those options are interesting:

-w Resolve ASN/ASName for A and AAAA records
-R, --resolve-ips Resolve PTR records for IP addresses in A and
AAAA records

It's unclear how much batch support this supports and obviously it
doesn't do much beyond that (e.g. no lookup on other databases or
geoip...)

a.

--
If I can't dance, I don't want to be part of your revolution.
- Emma Goldman