invesdwin-webproxy-portscan

[Prasad Katre]

Hi,

I am trying to build TCP SYN based portscanner as simple microservice which will scan the ports(list of ports) for specific Ip address.

Can you please point to the code which you have used to do so using Jpcapng?

Looking for some pointer which will help me to proceed on this line.

Thanks in advance.

Regards,

Prasad.

[subes]

ICMP: https://github.com/invesdwin/invesdwin-webproxy/tree/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan/src/main/java/de/invesdwin/webproxy/portscan/internal/pcap/icmp

(validate host before scanning)

SYN: https://github.com/invesdwin/invesdwin-webproxy/tree/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan/src/main/java/de/invesdwin/webproxy/portscan/internal/pcap/syn

(find open ports)

You could reuse invesdwin-webproxy-portscan (currently there are messaging gateways for this via rabbitmq/AMQP and activemq/JMS):

Here you send requests for ping/syn scans:

https://github.com/invesdwin/invesdwin-webproxy/blob/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan-contract/src/main/java/de/invesdwin/webproxy/portscan/contract/IPortscanService.java

Here you receive response messages for hosts/ports found:

https://github.com/invesdwin/invesdwin-webproxy/blob/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan-contract/src/main/java/de/invesdwin/webproxy/portscan/contract/IPortscanClient.java

Everything is asynchronous and the service implementation handles throttling (configurable) so that the network does not get overloaded and prevents the target from blocking your IP (against intrusion detection).

Best regards,

Edwin

[subes]

Here the actual payloads for the interfaces:

https://github.com/invesdwin/invesdwin-webproxy/blob/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan-contract/src/main/java/META-INF/xsd/webproxy.portscan.xsd

You could also build more communication methods in front of it. I only separated the portscanning from the application layer because jpcapng required admin rights (which a process exposed to the internet via a REST-service should not be).

[subes]

Back to Google-Groups

---------- Forwarded message ---------
From: subes <gsu...@gmail.com>
Date: Tue, Feb 2, 2021 at 2:19 PM
Subject: Re: invesdwin-webproxy-portscan
To: Prasad Katre <prasad...@gmail.com>

I just retested it, on OpenJDK 8 u275 on Ubuntu it works fine for me.

JAVA_HOME=/usr/lib/jvm/default-java/

sudo java -jar invesdwin-webproxy-portscan-dist-1.0.2-SNAPSHOT-standalone.jar
2021-02-02 14:18:15.862 [ |main               ] INFO  d.i.c.b.i.p.u.i.LogbackConfigurationLoader.logLogbackBeingCo - Loading 3 logback configs from classpath [/META-INF/logback/integration.logback.xml, /META-INF/logback/context.logback.xml, /META-INF/logback/integration.jms.logback.xml]
2021-02-02 14:18:15.892 [ |main               ] INFO  d.i.c.b.i.p.u.internal.SystemPropertiesLoader.logPropertiesB - Loading 5 properties files from classpath [/META-INF/de.invesdwin.context.integration.properties, /META-INF/de.invesdwin.context.properties, /META-INF/io.netty.versions.properties, /META-INF/spring.integration.default.properties, /META-INF/de.invesdwin.webproxy.portscan.properties]
2021-02-02 14:18:15.895 [ |main               ] INFO  d.i.c.b.i.p.u.internal.SystemPropertiesLoader.logOverridePro - Loading 0 override properties files from [file:/root/.invesdwin/system.properties(0), classpath*:/META-INF/env/distribution.properties(0)]
2021-02-02 14:18:16.138 [ |main               ] INFO  d.i.c.b.i.p.u.internal.BasePackagesConfigurer.getBasePackage - Loading 1 base package [de.invesdwin]
2021-02-02 14:18:17.550 [ |main               ] WARN  d.i.c.b.i.p.util.DefaultTimeZoneConfigurer.configure         - Changing JVM default TimeZone from [Europe/Berlin] to [UTC] in order to have commonality between systems:
- Use -Dkeep.user.timezone=true to keep the system default. Additionally using -Duser.timezone=<TimeZoneID> allows to change the default of the JVM.
- Hide this warning by using -Duser.timezone=UTC to specify the default to match the convention.
2021-02-02 14:18:17.555 [ |main               ] INFO  d.i.c.b.i.p.util.DefaultTimeZoneConfigurer.info              - Using user.timezone=UTC
2021-02-02 14:18:18.502 [ |main               ] INFO  de.invesdwin.context.beans.init.MergedContext.logContextsBei - Loading 6 spring contexts in PROD config from classpath [/META-INF/ctx.integration.marshaller.xml, /META-INF/ctx.integration.xml, /META-INF/ctx.spring.xml, /META-INF/ctx.webproxy.portscan.service.xml, /META-INF/ctx.integration.jms.cluster.xml, /META-INF/ctx.integration.jms.marshaller.xml]
2021-02-02 14:18:20.332 [ |main               ] INFO  de.invesdwin.context.beans.init.MergedContext.info           - Bootstrap finished after: PT4.808.061.284S

On Tue, Feb 2, 2021 at 1:56 PM subes <gsu...@gmail.com> wrote:

Can you tell me which exact JVM you are using? Are you using Windows or Linux?

It seems there is no: java.lang.ClassNotFoundException: com.sun.tools.attach.spi.AttachProvider

Are you sure you have the JDK installed (execute: java -version) and the JAVA_HOME is setup correctly? Also make sure you have a tools.jar that can be found: https://github.com/invesdwin/invesdwin-instrument/blob/master/invesdwin-instrument/src/main/java/de/invesdwin/instrument/internal/JdkFilesFinder.java

If the automatic loading of the agent does not work, you have to set the " -javaagent:invesdwin-instrument.jar" JVM parameter. replace investing-instrument.jar with the actual jar file location.

More information: https://github.com/invesdwin/invesdwin-instrument

On Tue, Feb 2, 2021 at 12:37 PM Prasad Katre <prasad...@gmail.com> wrote:

Hi Edwin,

I am getting below error while running the standalone jar with open jdk 1.8

[root@localhost target]# java -jar invesdwin-webproxy-portscan-dist-1.0.2-SNAPSHOT-standalone.jar
2021-02-02 17:03:16.846 [ |main               ] INFO  d.i.c.b.i.p.u.i.LogbackConfigurationLoader.logLogbackBeingCo - Loading 3 logback configs from classpath [/META-INF/logback/integration.logback.xml, /META-INF/logback/context.logback.xml, /META-INF/logback/integration.jms.logback.xml]
2021-02-02 17:03:16.925 [ |main               ] INFO  d.i.c.b.i.p.u.internal.SystemPropertiesLoader.logPropertiesB - Loading 5 properties files from classpath [/META-INF/de.invesdwin.context.integration.properties, /META-INF/de.invesdwin.context.properties, /META-INF/io.netty.versions.properties, /META-INF/spring.integration.default.properties, /META-INF/de.invesdwin.webproxy.portscan.properties]
2021-02-02 17:03:16.945 [ |main               ] INFO  d.i.c.b.i.p.u.internal.SystemPropertiesLoader.logOverridePro - Loading 0 override properties files from [file:/root/.invesdwin/system.properties(0), classpath*:/META-INF/env/distribution.properties(0)]
2021-02-02 17:03:17.453 [ |main               ] INFO  d.i.c.b.i.p.u.internal.BasePackagesConfigurer.getBasePackage - Loading 1 base package [de.invesdwin]
Exception in thread "Thread-4" java.lang.RuntimeException: java.lang.RuntimeException: To run on OpenJDK 64-Bit Server VM use -javaagent:invesdwin-instrument.jar
at de.invesdwin.instrument.DynamicInstrumentationLoader$1.run(DynamicInstrumentationLoader.java:90)
Caused by: java.lang.RuntimeException: To run on OpenJDK 64-Bit Server VM use -javaagent:invesdwin-instrument.jar
at de.invesdwin.instrument.internal.DynamicInstrumentationLoadAgentMain.loadAgent(DynamicInstrumentationLoadAgentMain.java:62)
at de.invesdwin.instrument.DynamicInstrumentationLoader.loadAgent(DynamicInstrumentationLoader.java:119)
at de.invesdwin.instrument.DynamicInstrumentationLoader.access$000(DynamicInstrumentationLoader.java:29)
at de.invesdwin.instrument.DynamicInstrumentationLoader$1.run(DynamicInstrumentationLoader.java:87)
Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.spi.AttachProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at de.invesdwin.instrument.internal.DynamicInstrumentationLoadAgentMain.loadAgent(DynamicInstrumentationLoadAgentMain.java:31)
... 3 more

regards,

Prasad.

On Tue, Feb 2, 2021 at 12:11 AM sub es <gsu...@gmail.com> wrote:

1. My projects should be downwards compatible to jdk8. If not, raise a bug, I occasionally test against jdk8.

2. synchronous is a very bad idea here from an architecture perspective.

3. this is only relevant for spring-integration. These objects facilitate the command message pattern. No explicit usage required.

4. just run the standalone jars via java -jar xyz-standalone.jar. If you also deploy webproxy-broker on the same machine, they will communicate via jms. Though you have to setup a mysql database following the docs from invesdwin-context-persistence or configure a h2 inmemory storage. The webproxy-broker will crawl for proxies and validate those using webproxy-portscan. The xyz-dist project create the standalone jars as distributions of the individual modules.

Though I think I had some unit tests in those projects. If you run eclipse as admin you should be able to run tests against jpcapng directly. This should allow for easier debugging.

5. there should be some log outputs that tell about which network interface was selected. I think it was in some static initializer for jpcapng using native apis.

Von: prasad...@gmail.com Gesendet: 1. Februar 2021 19:32 An: gsu...@gmail.com Betreff: Re: invesdwin-webproxy-portscan

Hi  Edwin,

Thanks for your email. This will certainly help me to proceed with the portscan implementation. but there are few differences/limitations on my requirements. JDK on my side is 1.8 not jDK 9 :( . I could the code is jdk 9 based. 

1. There could be chance that Ping is blocked on n/w , so I do not need to check if host is available using ping. So in case the host is not reachable by syn packet will timeout or some exception will come. I will treat it as port closed.

2. I need to make this as Synchronous call as host ip for port scanning is actually coming on kafka message. So I want to make this Portscan call synchronous.

3. I could see PortScanSyncRequest in XSD payload but no api uses it. I think it is not being used. I think this PortScanAsyncRequest is only used.

4. Also can you help me how can I test your code as standalone Java program so that I can debug and see how things are getting invoked.

5. Also how the network interface is detected or loaded ? 

Regards,

Prasad.

On Mon, Feb 1, 2021 at 10:23 PM subes <gsu...@gmail.com> wrote:

ICMP: https://github.com/invesdwin/invesdwin-webproxy/tree/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan/src/main/java/de/invesdwin/webproxy/portscan/internal/pcap/icmp

(validate host before scanning)

SYN: https://github.com/invesdwin/invesdwin-webproxy/tree/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan/src/main/java/de/invesdwin/webproxy/portscan/internal/pcap/syn

(find open ports)

You could reuse invesdwin-webproxy-portscan (currently there are messaging gateways for this via rabbitmq/AMQP and activemq/JMS):

Here you send requests for ping/syn scans:

https://github.com/invesdwin/invesdwin-webproxy/blob/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan-contract/src/main/java/de/invesdwin/webproxy/portscan/contract/IPortscanService.java

Here you receive response messages for hosts/ports found:

https://github.com/invesdwin/invesdwin-webproxy/blob/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan-contract/src/main/java/de/invesdwin/webproxy/portscan/contract/IPortscanClient.java

Everything is asynchronous and the service implementation handles throttling (configurable) so that the network does not get overloaded and prevents the target from blocking your IP (against intrusion detection).

Best regards,

Edwin

Prasad Katre schrieb am Montag, 1. Februar 2021 um 17:47:30 UTC+1:

Hi,

I am trying to build TCP SYN based portscanner as simple microservice which will scan the ports(list of ports) for specific Ip address.

Can you please point to the code which you have used to do so using Jpcapng?

Looking for some pointer which will help me to proceed on this line.

Thanks in advance.

Regards,

Prasad.

--
You received this message because you are subscribed to a topic in the Google Groups "invesdwin-platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/invesdwin-platform/k258KMS6BQU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to invesdwin-platf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/invesdwin-platform/4334748b-ada5-4c5f-a678-0a80b403d145n%40googlegroups.com.

[Prasad Katre]

Hi Edwin,

Java version which I am using is :

 java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)

I got ahead of the earlier issue but now new error related to libpcap file not found. Am I missing something in setup?

sudo java -jar target/invesdwin-webproxy-portscan-dist-1.0.2-SNAPSHOT-standalone.jar

Caused by - org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'portscanServiceActivator': Unsatisfied dependency expressed through field 'service'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'portscanService': Unsatisfied dependency expressed through field 'portScanner'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'portScanner': Unsatisfied dependency expressed through field 'scanner'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'synScanScheduler': Unsatisfied dependency expressed through field 'synSender'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'synSender': Unsatisfied dependency expressed through field 'sender'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'packetSender': Unsatisfied dependency expressed through field 'bootstrap'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'portscanBootstrap' defined in URL [jar:file:/opt/java_portscan/invesdwin-webproxy/invesdwin-webproxy-parent/invesdwin-webproxy-portscan-dist/target/invesdwin-webproxy-portscan-dist-1.0.2-SNAPSHOT-standalone.jar!/de/invesdwin/webproxy/portscan/internal/PortscanBootstrap.class]: Instantiation of bean failed; nested exception is java.lang.UnsatisfiedLinkError: /tmp/investemp_6831759212176352920.tmp: libpcap.so.0.8: cannot open shared object file: No such file or directory

[subes]

Hi Prasad,

can you tell me what operating system you are using?

Maybe there is no precompiled library for your OS available (maybe running on ARM?). These are supported out of the box:

https://github.com/invesdwin/invesdwin-webproxy/tree/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan/src/main/java/jpcap

Maybe you are using Mac-OS? That might also be the reason why the invesdwin-instrument initialization did not work and why the .so libs are not available (did you use the agentlib param or did you find another solution? I might want to mention this in the docs for other people).

I don't have a Mac-OS box available, so I cannot test with that. Maybe you should try a VirtualBox instance with Ubuntu 20.04.

Or try to compile jpcapng yourself for Mac-OS. Dunno if that is easy. FreeBSD should also misses out-of-the-box support alongside other exotic *NIXes.

You could debug this class regarding the native loading mechanism:

https://github.com/invesdwin/invesdwin-webproxy/blob/master/invesdwin-webproxy-parent/invesdwin-webproxy-portscan/src/main/java/de/invesdwin/webproxy/portscan/internal/PortscanBootstrap.java

Here the original repo: https://sourceforge.net/projects/jpcapng/

Here the clone on github: https://github.com/subes/jpcapng

(read the readme)

On older versions of Linux in the past changes of glibc or similar were also causing native loading issues, though with JXGrabKey (different library I wrote). Recompiling on the target machine fixed that.

I happily accept pull requests for solutions.

Best regards,

Edwin

You received this message because you are subscribed to the Google Groups "invesdwin-platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to invesdwin-platf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/invesdwin-platform/bb56667e-ab1c-4a32-9fcd-dba46fec88d1n%40googlegroups.com.

[subes]

Ah, no I get it. You need to install libpcap on the operating system:

Instantiation of bean failed; nested exception is java.lang.UnsatisfiedLinkError: /tmp/investemp_6831759212176352920.tmp: libpcap.so.0.8: cannot open shared object file: No such file or directory

"apt install libpcap0.8"

I will add that to the docs.

[subes]

I think it does not work because it links to a file called libpcap.so.0.8 which I guess you don't have on CentOS. Please try doing a symbolic link so that files exists. Otherwise you might have to try compiling the jpcapng native library for CentOS yourself.

Then replace the library inside invesdwin-webproxy-portscan (which is the only library that does SYN Stealth Scans I know for Java, otherwise you have to build it yourself with libs like jpcapng).

Maybe recompiling the jpcapng library with static linking could solve this issue also for other people on other distros. I am happy to accept pull requests if this helps you. Otherwise you have to replace the lib in your own fork I guess.

Here is what I have: "locate libpcap"

/usr/lib/i386-linux-gnu/libpcap.so.0.8
/usr/lib/i386-linux-gnu/libpcap.so.1.9.1
/usr/lib/x86_64-linux-gnu/libpcap.so.0.8
/usr/lib/x86_64-linux-gnu/libpcap.so.1.9.1
/usr/share/doc/libpcap0.8
/usr/share/doc/libpcap0.8/CREDITS.gz
/usr/share/doc/libpcap0.8/README.Debian
/usr/share/doc/libpcap0.8/README.md
/usr/share/doc/libpcap0.8/changelog.Debian.gz
/usr/share/doc/libpcap0.8/copyright
/var/cache/apt/archives/libpcap0.8_1.9.1-4_amd64.deb
/var/cache/apt/archives/libpcap0.8_1.9.1-4_i386.deb
/var/lib/dpkg/info/libpcap0.8:amd64.list
/var/lib/dpkg/info/libpcap0.8:amd64.md5sums
/var/lib/dpkg/info/libpcap0.8:amd64.shlibs
/var/lib/dpkg/info/libpcap0.8:amd64.symbols
/var/lib/dpkg/info/libpcap0.8:amd64.triggers
/var/lib/dpkg/info/libpcap0.8:i386.list
/var/lib/dpkg/info/libpcap0.8:i386.md5sums
/var/lib/dpkg/info/libpcap0.8:i386.shlibs
/var/lib/dpkg/info/libpcap0.8:i386.symbols
/var/lib/dpkg/info/libpcap0.8:i386.triggers

On Thu, Feb 4, 2021 at 7:43 AM Prasad Katre <prasad...@gmail.com> wrote:

Hi Edwin,

I am using Centos 7 on my virtual box.

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

I have already installed libpcap.

Package 14:libpcap-devel-1.5.3-12.el7.x86_64 already installed and latest version
Package 14:libpcap-1.5.3-12.el7.x86_64 already installed and latest version

Do you have any project/library which is used for TCP syn based port scanner? My main goal is to write the TCY SYN based port scanner in java.

Regards,

Prasad.

[subes]

It might also be the case that libpcap-1.5.3 is incompatible with the compiled against version libpcap-0.8. You could check if your distro also offers the older version. There might be breaking changes that require modification of jpcapng to make it work.

[subes]

Here people with a similar issue, they suggest the symbolic link should solve the issue: https://github.com/mehrdadrad/mylg/issues/74