WorkStream with read-only PETSc parallel vectors

[Arthur Bawin]

Good morning deal.II community,

I am getting segmentation faults when using WorkStream::run and 2+ threads when the worker function involves reinitializing a scratch_data and interpolating the current FE solution with FEValues, for instance:

fe_values[pressure].get_function_values(current_solution, present_pressure_values);

The current solution is a distributed PETSc vector. Overall and following this post (https://groups.google.com/g/dealii/c/Jvt36NOXM4o/m/tytRf3N9f4gJ), does it still hold that multithreaded matrix/rhs assembly with PETSc wrappers is not thread safe? (I think so since PETSc is still stated as not thread safe, but I'm asking in case I missed something)

I am currently testing with Mumps (through PETSc): if I'm not mistaken, then the other possibilities to use it with distributed matrix/vectors and threaded assembly are with deal.II's own distributed vectors + standalone Mumps, or with Trilinos wrappers + Mumps through Amesos/Amesos2, is that correct?

Thank you for your time,

Arthur

[Wolfgang Bangerth]

Arthur:

> I am getting segmentation faults when using WorkStream::run and 2+
> threads when the worker function involves reinitializing a scratch_data
> and interpolating the current FE solution with FEValues, for instance:
>
> fe_values[pressure].get_function_values(current_solution,
> present_pressure_values);

Is the worker function resizing a global vector? These objects are
supposed to be self-contained with what a worker needs, and workers are
not supposed to *change* anything that isn't part of the scratch object.
They may only *read* data that lives outside the scratch object.

So the question here is where `present_pressure_values` lives. Is it
shared between threads.

> The current solution is a distributed PETSc vector. Overall and
> following this post (https://groups.google.com/g/dealii/c/Jvt36NOXM4o/m/
> tytRf3N9f4gJ), does it still hold that multithreaded matrix/rhs assembly
> with PETSc wrappers is not thread safe? (I think so since PETSc is still
> stated as not thread safe, but I'm asking in case I missed something)

I believe you should be able to *read* from multiple threads in
parallel, but not write. Either way, it would be useful to see the
backtrace you get from your segfaults.

> I am currently testing with Mumps (through PETSc): if I'm not mistaken,
> then the other possibilities to use it with distributed matrix/vectors
> and threaded assembly are with deal.II's own distributed vectors +
> standalone Mumps, or with Trilinos wrappers + Mumps through Amesos/
> Amesos2, is that correct?

"...to use it..." -- what is "it" here?

Best
W.

[blais...@gmail.com]

Dear Arthur,

The  get_function_values does not modify current_solution. Consequently, as along as present_pressure_values is either locally declared or a member of the scratch data, you should not get a seg fault.

Do you get any assertion thrown when launching in debug mode?

[Arthur Bawin]

Thank you Wolfgang and Bruno for your answers.

present_pressure_values is part of the scratch_data, so I don't think this is the issue. The attached gdb backtrace shows VecGetSize_Seq on top, called in dealii::PETScWrappers::VectorBase::extract_subvector_to<unsigned int const*, double*> , which would be a read issue on Petsc's end? (Sorry, I should have included the backtrace in the first place).

I also joined the (lengthy) helgrind output for a single rhs assembly with Petsc wrappers on 2 threads and mpirun -n 1, but I'm having a hard time separating the benign messages from the real issue.

It is not particularly urgent, as I will run some full MPI tests in the meantime, but I was curious to know if a hybrid MPI/threads would work with Petsc wrappers and Mumps as solver ("it" refered to Mumps in my first message).

Thank you for your time,

Arthur

[Wolfgang Bangerth]

On 11/6/25 12:08, Arthur Bawin wrote:
> present_pressure_values is part of the scratch_data, so I don't think this is
> the issue. The attached gdb backtrace shows VecGetSize_Seq on top, called
> in dealii::PETScWrappers::VectorBase::extract_subvector_to<unsigned int
> const*, double*> , which would be a read issue on Petsc's end? (Sorry, I
> should have included the backtrace in the first place).

Yes, I think my expectation would also have been that reading the size of a
vector via VecGetSize is thread-safe. Apparently, it is not :-( The function
is implemented here:

https://gitlab.com/petsc/petsc/-/blob/main/src/vec/vec/impls/seq/bvec2.c?ref_type=heads#L122
I can't really say why that would not be thread-safe: The only thing I can
imagine is the call to PetscFunctionBegin.

I do not know how to fix this, but here are a few suggestions:
* When you're in a debugger, take a look where the other threads are. In the
backtrace, one is clearly in a PETSc function. Check where the other thread
is, to make sure that you can narrow down that it's really concurrent access
to PETSc data that are the problem.

* If you have the ability, compile PETSc with debug flags so that you can go
in the debugger and check which line in the PETSc functions actually causes
the problem.

* Talk to the PETSc folks on their mailing list about the issue once you're
sure you know what the problem is.

I acknowledge that none of these options actually solve your problem.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Timo Heister]

Dear Arthur,

See https://github.com/dealii/dealii/issues/18886

Basically, even reading from PETSc ghosted vectors from multiple
threads currently does not work, you can either switch to the current
PETSc master and test if that fixes the issue or wait until we figure
out a way to not lock/unlock the ghost VEC from multiple threads.

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/dealii/c5041521-b593-4d9d-8b4b-a9c4920ef570%40colostate.edu.