Prosody will not let me send images or files from one client to another, have been trying for weeks with no success!

[Mitch Tell]

I've been running Prosody for a long time and I remember at one time I could drop an image into a client's compositionb area and hit enter and it would send the image to the other party in the chat.  Or I could sent a file, it would prompt the other person to accept in and then save it in his downloads.  But then that stopped working and for months now, but especially in the last week I have been trying to figure out why.  One thing I did figure out is that all the newer IM clients get really bent out of shape if you don't have valid (not self-signed) certificates for your domains.  But I run this server only for clients on a local network that spans two buildings (house and outbuilding) on the same property.  And I ONLY want to use it for simple instant messaging (chat) and to send and receive images and files.  That's it - no multi user chats, no connections to or from the Internet, just local use only.

The chat works fine but sending images and files is a no go.  Depending on the XMPP client, if I drop an image file into the composition area it will send just the filename, or the path and filename, or it will say it is sending the file but the upload just sits there and the recipient sees nothing, or it will upload the file and send what is supposed to be a URL on the server where it can be obtained (presumably) but if you try to go there you just get a blank white page and nothing happens (which does not surprise me because the system is not running a web server), and anyway that's not what I want to happen, I just want it to send the image to the client.  We have tried about a half dozen different clients under MacOS and Linux but none will actually send or receive the files.  I figure it is probably some simple configuration error but if it is I can't spot it.

I went through the hassle of getting Let's Encrypt certificates, for that I had to create a new dynamic dns domain and make new accounts under that domain, and then because it looked like it was needed for the http_file_share component I got yet another dynamic DNS domain and got certificates for that.  Why some xmpp clients don't understand that if you are on a local network only you really don't need a certificate I just don't get, but once I got those certificates some of the clients I was trying to use magically started working, but only for chat.

I have literally spent so much time trying to make this work it is ridiculous, so I am asking, can anyone tell me what I am doing wrong?  Or is Prosody just the wrong tool to use in this situation?

If I run prosodyctl about I get this:

Prosody 0.12.3

# Prosody directories
Data directory:     /var/lib/prosody
Config directory:   /etc/prosody
Source directory:   /usr/lib/prosody
Plugin directories:
  /var/lib/prosody/custom_plugins - not a directory!
  /usr/local/lib/prosody/modules - not a directory!
  /usr/lib/prosody/modules/


# Operating system
Linux 6.6.31+rpt-rpi-2712

# Lua environment
Lua version:             Lua 5.4

Lua module search paths:
  /usr/lib/prosody/?.lua
  /usr/local/share/lua/5.4/?.lua
  /usr/local/share/lua/5.4/?/init.lua
  /usr/local/lib/lua/5.4/?.lua
  /usr/local/lib/lua/5.4/?/init.lua
  /usr/share/lua/5.4/?.lua
  /usr/share/lua/5.4/?/init.lua
  /var/lib/prosody/custom_plugins/share/lua/5.4/?.lua
  /var/lib/prosody/custom_plugins/share/lua/5.4/?/init.lua

Lua C module search paths:
  /usr/lib/prosody/?.so
  /usr/local/lib/lua/5.4/?.so
  /usr/lib/aarch64-linux-gnu/lua/5.4/?.so
  /usr/lib/lua/5.4/?.so
  /usr/local/lib/lua/5.4/loadall.so

LuaRocks:         Not installed

# Network

Backend: epoll

# Lua module versions
LuaExpat:     1.5.1
LuaFileSystem: 1.8.0
LuaSec:       1.2.0
LuaSocket:     3.0.0
luaunbound:   1.0.0
readline:     3.1

# library versions
libcrypto:     OpenSSL 3.0.13 30 Jan 2024
libexpat:     expat_2.5.0
libunbound:   1.17.1

This is a redacted copy (redacted to remove my actual dynamic dns addresses) of my prosody.cfg.lua file, with many of the commented out lines removed to save space:

---------- Server-wide settings ----------
admins = { "us...@oldaccount.somedns.com", "us...@oldaccount.somedns.com", "us...@newaccount.someotherdns.com", "us...@newaccount.someotherdns.com" }
use_ipv6 = false

plugin_paths = { "/usr/local/lib/prosody/modules" }

-- This is the list of modules Prosody will load on startup.
-- Documentation for bundled modules can be found at: https://prosody.im/doc/modules
modules_enabled = {

-- Generally required
"disco"; -- Service discovery
"roster"; -- Allow users to have a roster. Recommended ;)
"saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
"tls"; -- Add support for secure TLS on c2s/s2s connections

-- Not essential, but recommended
"blocklist"; -- Allow users to block communications with other users
"bookmarks"; -- Synchronise the list of open rooms between clients
"carbons"; -- Keep multiple online clients in sync
"dialback"; -- Support for verifying remote servers using DNS
"limits"; -- Enable bandwidth limiting for XMPP connections
"pep"; -- Allow users to store public and private data in their account
"private"; -- Legacy account storage mechanism (XEP-0049)
"smacks"; -- Stream management and resumption (XEP-0198)
"vcard4"; -- User profiles (stored in PEP)
"vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard

-- Nice to have
"csi_simple"; -- Simple but effective traffic optimizations for mobile devices
"invites"; -- Create and manage invites
"invites_adhoc"; -- Allow admins/users to create invitations via their client
"invites_register"; -- Allows invited users to create accounts
"ping"; -- Replies to XMPP pings with pongs
"register"; -- Allow users to register on this server using a client and change passwords
"time"; -- Let others know the time here on this server
"uptime"; -- Report how long server has been running
"version"; -- Replies to server version requests
--"mam"; -- Store recent messages to allow multi-device synchronization
--"turn_external"; -- Provide external STUN/TURN service for e.g. audio/video calls

-- Admin interfaces
"admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
"admin_shell"; -- Allow secure administration via 'prosodyctl shell'

-- HTTP modules
--"bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
--"http_openmetrics"; -- for exposing metrics to stats collectors
--"websocket"; -- XMPP over WebSockets

-- Other specific functionality
"posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
--"announce"; -- Send announcement to all online users
--"groups"; -- Shared roster support
--"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.
--"mimicking"; -- Prevent address spoofing
--"motd"; -- Send a message to users when they log in
"proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use
--"s2s_bidi"; -- Bi-directional server-to-server (XEP-0288)
--"server_contact_info"; -- Publish contact information for this service
--"tombstones"; -- Prevent registration of deleted accounts
--"watchregistrations"; -- Alert admins of registrations
--"welcome"; -- Welcome users who register accounts
"http_file_share"; -- Let users share files via HTTP
}

-- These modules are auto-loaded, but should you want
-- to disable them then uncomment them here:
modules_disabled = {
-- "offline"; -- Store offline messages
-- "c2s"; -- Handle client connections
-- "s2s"; -- Handle server-to-server connections
}

pidfile = "/run/prosody/prosody.pid";

s2s_secure_auth = true


--s2s_insecure_domains = { "insecure.example" }

--s2s_secure_domains = { "jabber.org" }


-- Rate limits

limits = {
c2s = {
rate = "10kb/s";
};
s2sin = {
rate = "30kb/s";
};
}

-- Authentication

authentication = "internal_hashed"

-- Storage

--storage = "sql" -- Default is "internal" (Debian: "sql" requires one of the
-- lua-dbi-sqlite3, lua-dbi-mysql or lua-dbi-postgresql packages to work)

-- For the "sql" backend, you can uncomment *one* of the below to configure:
--sql = { driver = "SQLite3", database = "prosody.sqlite" } -- Default. 'database' is the filename.
--sql = { driver = "MySQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" }
--sql = { driver = "PostgreSQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" }


-- Archiving configuration

archive_expires_after = "1w" -- Remove archived messages after 1 week

-- Logging configuration
-- Debian:
--  Logs info and higher to /var/log
--  Logs errors to syslog also
log = {
-- Log files (change 'info' to 'debug' for debug logs):
info = "/var/log/prosody/prosody.log";
error = "/var/log/prosody/prosody.err";
-- Syslog:
-- { levels = { "error" }; to = "syslog";  };
}


-- Certificates
-- Location of directory to find certificates in (relative to main config file):
certificates = "certs"

----------- Virtual hosts -----------
------ Additional config files ------
-- For organizational purposes you may prefer to add VirtualHost and
-- Component definitions in their own config files. This line includes
-- all config files in /etc/prosody/conf.d/

VirtualHost "oldaccount.somedns.com"
        ssl = {
                key = "/etc/prosody/certs/oldaccount.somedns.com.key";
                certificate = "/etc/prosody/certs/oldaccount.somedns.com.crt";
        }

VirtualHost "newaccount.someotherdns.com"
        ssl = {
                key = "/etc/prosody/certs/newaccount.someotherdns.com.key";
                certificate = "/etc/prosody/certs/newaccount.someotherdns.com.crt";
        }
        disco_items = {
                { "uploadaccount.someotherdns.com" },
        }

------ Components ------
Component "uploadaccount.someotherdns.com" "http_file_share"

Include "conf.d/*.cfg.lua"

[Christian Weiske]

Hello Mitch,

>The chat works fine but sending images and files is a no go.
>Depending on the XMPP client, if I drop an image file into the
>composition area it will send just the filename, or the path and
>filename, or it will say it is sending the file but the upload just
>sits there and the recipient sees nothing, or it will upload the file
>and send what is supposed to be a URL on the server where it can be
>obtained (presumably) but if you try to go there you just get a blank
>white page and nothing happens (which does not surprise me because the
>system is not running a web server), and anyway that's not what I want
>to happen, I just want it to send the image to the client. We have
>tried about a half dozen different clients under MacOS and Linux but
>none will actually send or receive the files. I figure it is probably
>some simple configuration error but if it is I can't spot it.

>"http_file_share"; -- Let users share files via HTTP

I am using the "http_upload_external" module, which is working fine -
but I have an Apache server running which handles the up- and
downloads. The clients Dino and Conversations work fine with it.

For Apache, you have to enable the "ap_trust_cgilike_cl" setting to get
it working properly:
https://cweiske.de/tagebuch/content-length-header-missing.htm#solution

--
Regards/Mit freundlichen Grüßen
Christian Weiske

-=≡ Geeking around in the name of science since 1982 ≡=-

[Mitch Tell]

Well there are a few problems with that idea for me, first of all I have never installed or configured Apache because I have no reason to want to serve web pages, but there is another unrelated program on the system
that has its own local web interface and it turns out that it runs apache2 so I tried enabling the setting you suggested and that appeared to work.  However Apache is kind of like a big black hole to me, I know it is there but I know absolutely nothing about it other than that it serves web pages for this unrelated program.

But also the "http_upload_external" module does not seem to be installed - if I look in /var/lib/prosody/modules it's not there, whereas mod_http_file_share.lua is there (also mod_http.lua, mod_http_errors.lua,and mod_http_files.lua).  And if you look at the installation instructions for installing mod_http_upload_external it tells you to run sudo prosodyctl install --server=https://modules.prosody.im/rocks/ mod_http_upload_external but if I do that I get this:

# sudo prosodyctl install --server=https://modules.prosody.im/rocks/ mod_http_upload_external
Installing mod_http_upload_external in /var/lib/prosody/custom_plugins
sh: 1: luarocks: not found

So, I tried doing sudo apt install luarocks and it installed but then I got this:

# sudo prosodyctl install --server=https://modules.prosody.im/rocks/ mod_http_upload_external
Installing mod_http_upload_external in /var/lib/prosody/custom_plugins
Installing https://modules.prosody.im/rocks/mod_http_upload_external-33-1.src.rock

Error: Failed finding Lua header files. You may need to install them or configure LUA_INCDIR.

And I tried installing lua-devel (suggested on one site) but apparently there is no such package in the Debian repository.

I then tried installing liblua5.1-dev (suggested on another site) because the installed version of Lua is 5.1.5, and got this:

# sudo apt install liblua5.1-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'liblua5.1-0-dev' instead of 'liblua5.1-dev'
liblua5.1-0-dev is already the newest version (5.1.5-9).
liblua5.1-0-dev set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

So I have no idea why luarocks is complaining about header files.  And this is getting much deeper into the weeds than I would like and in any case I thought mod_http_file_share was supposed to work in this situation.  Doesn't Prosody have a way to do this without using mod_http_upload_external or Apache?

[Matthew Wild]

Hi,

(Prosody dev here)

On Wed, 28 Aug 2024 at 14:42, Mitch Tell <michigan...@gmail.com> wrote:
> Well there are a few problems with that idea for me, first of all I have never installed or configured Apache because I have no reason to want to serve web pages, but there is another unrelated program on the system
> that has its own local web interface and it turns out that it runs apache2 so I tried enabling the setting you suggested and that appeared to work. However Apache is kind of like a big black hole to me, I know it is there but I know absolutely nothing about it other than that it serves web pages for this unrelated program.

I recommend you stick with mod_http_file_share, especially if you
don't already have Apache. Adding more things will only add more
places where things can go wrong or be misconfigured :)

What was missing from your initial post is any error message or
something we can use to figure out what is wrong with your setup.

The one thing I did spot in your config file is that you have
"http_file_share" listed *both* in modules_enabled *and* as a
Component. This will almost certainly confuse clients, and you, and
things may not work as expected. The documentation demonstrates how to
configure the component, and you can just remove "http_file_share"
from the modules_enabled list.

https://prosody.im/doc/modules/mod_http_file_share

If it still doesn't work after fixing this and restarting Prosody,
please tell us what client you tested with and what error message it
gave.

Good luck!

Regards,
Matthew

[Mitch Tell]

Well it just started working about an hour hour ago, I have no idea why other than that the only change I had made this morning was to follow Christian Weiske's instructions:

For Apache, you have to enable the "ap_trust_cgilike_cl" setting to get

it working properly:
https://cweiske.de/tagebuch/content-length-header-missing.htm#solution

Now that SHOULD not have made any difference because I never actually was able to get the mod_http_upload_external module to install, and as far as I know, without that it should not even be trying to interact with Apache.  But anyway file transfers had started working, so I came here to say that, and then I saw Matthew's advice about removing "http_file_share" from the modules_enabled list, so I commented that line out and it continued working after I restarted Prosody.

I am glad it is working now but it is really frustrating when something doesn't work for so long and then starts working again and you really don't know what actually solved the problem.  But I will note that because of an unrelated issue (apt update of some python modules) I had to reboot that server late last night, and didn't really start doing any serious testing again until about an hour ago.  And this time I only tried sending from Gajim to Gajim, or Gajim to Monal.  Adium (running on a machine that's still running MacOS Monterey) absolutely will NOT send or receive files but I guess that is to be expected since it probably hasn't been updated since the days of dial-up modems.

So it may have been the Apache thing (I doubt that, but who knows), rebooting the system (which I had done yesterday about three times but that was before those python modules updated), removing the duplicate reference to http_file_share, just testing with the right combination of clients, or who knows what but I REALLY hope it stays working for a while (I will probably be back if it doesn't!).

Thank you to everyone for your suggestions and assistance!