Does Prometheus support Netapp Trident NFS storage backend?

[tejaswini vadlamudi]

Prometheus documentation recommends to avoid NFS storage backends. But a few users over internet claim support for NFS based storage from Netapp in Prometheus. Just checking for opinions and feedback if this storage backend is compatible enough for regular Prometheus operations.

Thanks,

Teja

[Stuart Clark]

NFS isn't a supported option for Prometheus. Prometheus can do a lot of
I/O and therefore network storage often doesn't work all that well. You
can also have issues if the filesystem isn't fully POSIX compliant.

Just because it isn't supported doesn't mean it won't work. However
there are issues that people find, and if you do find those the
recommendation would always be to switch the filesystem to something
local. So basically you'd be fairly on your own if you did have problems.

--
Stuart Clark

[tejaswini vadlamudi]

Thanks Stuart, now I understand support in Prometheus project.

Though renowned storage vendors claim full compliance and seem to work with Prometheus data, it is better to stay away from it.

Cheers, Teja

[tejaswini vadlamudi]

Hi Stuart,

I forgot to ask the most important question on this topic :-) 

Could you explain the reason for not supporting NFS based storage in Prometheus?

Thanks,

Teja

[Stuart Clark]

On 2022-05-31 15:45, tejaswini vadlamudi wrote:
> Hi Stuart,
>
> I forgot to ask the most important question on this topic :-)
> Could you explain the reason for not supporting NFS based storage in
> Prometheus?
>

At its core Prometheus contains a high performance time-series database.
Network filesystems just don't have the same performance characteristics
or features as a direct local disk. It is similar to not expecting
support for storing a MySQL/Oracle/MSSQL/etc. database on a network
storage system - it might technically work, but you are likely to
encounter problems at some point, and you are unlikely to recenive any
free/paid support in that architecture.

--
Stuart Clark

[tejaswini vadlamudi]

What do you mean by not meeting desired performance characteristics? Is it the write conflicts, file/block size limiations, etc.. eventually over time, or is it with a high ingestion load?

Do we have similar problems if we deploy Prometheus in Agent mode (data streamed to a remote location) but still need WAL storage to handle receiver connection failures?

Thanks, Teja

[Ben Kochie]

Last I remember, most of the issues are around file locking and atomic operations.

Agent mode with WAL-only should be less of an issue, except possibly fsyncs.

Either way, depending on NFS is a bad idea from a reliability perspective, as your monitoring system now depends on the network and storage server to properly alert.

Like Stuart says, would you run your SQL servers over NFS? 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/993b1dc3-120d-4ec7-84d1-e99387e30e15n%40googlegroups.com.

[tejaswini vadlamudi]

I'm not on top of performance degradation issues with the NFS storage type. But by reading more on this topic, I understand the reasoning behind your suggestion.

NFS version 4 improved file locking.

MySQL documentation says "If you choose to use NFS, NFS Version 4 or later is recommended, as is testing your NFS setup thoroughly before deploying into a production environment". 

But Prometheus documentation and also the source code warn users to stay away from it.

https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L990

case "NFS_SUPER_MAGIC":

              level.Warn(logger).Log("fs_type", fsType, "msg", "This filesystem is not supported and may lead to data corruption and data loss. Please carefully read https://prometheus.io/docs/prometheus/latest/storage/ to learn more about supported filesystems.")

What do you recommend if certain users have access to ONLY NFS storage type? Do you suggest sending/streaming Prometheus data to someother long term data store like Victoria Metrics or Cortext? I still wonder about storage type for fsyncs and WAL file handling.
On K8S, if ephemeral storage is used to store Prometheus data,  data is not persistent and highly available.

Thanks!