Matrix Free Multiple DoFHandlers

[Sean Johnson]

Hello it is me again,

I am trying to something very similar to this discussion: https://groups.google.com/g/dealii/c/Hh4iBsM21FI/m/cF-mzDP4GkwJ

My question would be how could I pass the solution vector of the pre-determined solution into the matrix free operator.

I know this is a bad idea but currently I have the vector as a global variable so I don't have to pass it into the local_apply function or anything really. I do call initialize_dof_vector with the correct index but in the conjugate gradient solve function the preconditioner throws the error included at the bottom of this message.

Also, attached is the just the setup_system() function similar to step-37 to try and keep code to a minimal. Nothing has been altered in the solve() function from step-37 in the code.

Thanks if anyone has the time to get to this beforehand,

Sean Johnson

--------------------------------------------------------
An error occurred in line <238> of file </home/sjohnson/AE_beginnings/dealii/include/deal.II/matrix_free/vector_access_internal.h> in function
    void dealii::internal::check_vector_compatibility(const VectorType&, const dealii::MatrixFree<dim, Number, VectorizedArrayType>&, const dealii::internal::MatrixFreeFunctions::DoFInfo&) [with int dim = 2; Number = double; VectorizedArrayType = dealii::VectorizedArray<double, 2>; VectorType = dealii::LinearAlgebra::distributed::Vector<double>; typename std::enable_if<has_partitioners_are_compatible<VectorType>, VectorType>::type* <anonymous> = 0]
The violated condition was:
    false
Additional information:
    The parallel layout of the given vector is compatible neither with the
    Partitioner of the current FEEvaluation with dof_handler_index=1 nor
    with any Partitioner in MatrixFree. A potential reason is that you did
    not use MatrixFree::initialize_dof_vector() to get a compatible
    vector.

Stacktrace:
-----------
#0  ./step-37_minimal: void dealii::internal::check_vector_compatibility<2, double, dealii::VectorizedArray<double, 2ul>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, (dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>*)0>(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> > const&, dealii::internal::MatrixFreeFunctions::DoFInfo const&)
#1  ./step-37_minimal: void dealii::FEEvaluationBase<2, 1, double, false, dealii::VectorizedArray<double, 2ul> >::read_write_operation<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const, dealii::internal::VectorReader<double, dealii::VectorizedArray<double, 2ul> > >(dealii::internal::VectorReader<double, dealii::VectorizedArray<double, 2ul> > const&, std::array<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const*, 1ul> const&, std::array<std::vector<dealii::ArrayView<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>::value_type const, dealii::MemorySpace::Host>, std::allocator<dealii::ArrayView<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>::value_type const, dealii::MemorySpace::Host> > > const*, 1ul> const&, std::bitset<2ul> const&, bool) const
#2  ./step-37_minimal: void dealii::FEEvaluationBase<2, 1, double, false, dealii::VectorizedArray<double, 2ul> >::read_dof_values<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, unsigned int, std::bitset<2ul> const&)
#3  ./step-37_minimal: Step37::LaplaceOperator<2, 3, double>::local_apply(dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> > const&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, std::pair<unsigned int, unsigned int> const&) const
#4  ./step-37_minimal: dealii::internal::MFWorker<dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> >, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, Step37::LaplaceOperator<2, 3, double>, true>::process_range(void (Step37::LaplaceOperator<2, 3, double>::* const&)(dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> > const&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, std::pair<unsigned int, unsigned int> const&) const, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, unsigned int)
#5  ./step-37_minimal: dealii::internal::MFWorker<dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> >, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, Step37::LaplaceOperator<2, 3, double>, true>::cell(unsigned int)
#6  /home/sjohnson/AE_beginnings/dealii/lib/libdeal_II.g.so.9.4.0: dealii::internal::MatrixFreeFunctions::TaskInfo::loop(dealii::internal::MFWorkerInterface&) const
#7  ./step-37_minimal: void dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> >::cell_loop<Step37::LaplaceOperator<2, 3, double>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >(void (Step37::LaplaceOperator<2, 3, double>::*)(dealii::MatrixFree<2, double, dealii::VectorizedArray<double, 2ul> > const&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, std::pair<unsigned int, unsigned int> const&) const, Step37::LaplaceOperator<2, 3, double> const*, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, bool) const
#8  ./step-37_minimal: Step37::LaplaceOperator<2, 3, double>::apply_add(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#9  ./step-37_minimal: dealii::MatrixFreeOperators::Base<2, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::VectorizedArray<double, 2ul> >::mult_add(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, bool) const
#10  ./step-37_minimal: dealii::MatrixFreeOperators::Base<2, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::VectorizedArray<double, 2ul> >::vmult_add(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#11  ./step-37_minimal: dealii::MatrixFreeOperators::Base<2, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::VectorizedArray<double, 2ul> >::vmult(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#12  ./step-37_minimal: dealii::internal::SolverCG::IterationWorker<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, Step37::LaplaceOperator<2, 3, double>, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >, int>::do_iteration(unsigned int)
#13  ./step-37_minimal: void dealii::SolverCG<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::solve<Step37::LaplaceOperator<2, 3, double>, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> > >(Step37::LaplaceOperator<2, 3, double> const&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> > const&)
#14  ./step-37_minimal: dealii::PreconditionChebyshev<Step37::LaplaceOperator<2, 3, double>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> > >::estimate_eigenvalues(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#15  ./step-37_minimal: dealii::PreconditionChebyshev<Step37::LaplaceOperator<2, 3, double>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> > >::vmult(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#16  ./step-37_minimal: dealii::mg::SmootherRelaxation<dealii::PreconditionChebyshev<Step37::LaplaceOperator<2, 3, double>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::DiagonalMatrix<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> > >, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::apply(unsigned int, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#17  /home/sjohnson/AE_beginnings/dealii/lib/libdeal_II.g.so.9.4.0: dealii::Multigrid<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::level_v_step(unsigned int)
#18  /home/sjohnson/AE_beginnings/dealii/lib/libdeal_II.g.so.9.4.0: dealii::Multigrid<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::level_v_step(unsigned int)
#19  /home/sjohnson/AE_beginnings/dealii/lib/libdeal_II.g.so.9.4.0: dealii::Multigrid<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >::cycle()
#20  ./step-37_minimal: void dealii::internal::PreconditionMGImplementation::vmult<2, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::MGTransferMatrixFree<2, double>, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >(std::vector<dealii::DoFHandler<2, 2> const*, std::allocator<dealii::DoFHandler<2, 2> const*> > const&, dealii::Multigrid<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >&, dealii::MGTransferMatrixFree<2, double> const&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&, bool, dealii::mg::Signals const&, ...)
#21  ./step-37_minimal: void dealii::PreconditionMG<2, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::MGTransferMatrixFree<2, double> >::vmult<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> >(dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>&, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host> const&) const
#22  ./step-37_minimal: dealii::internal::SolverCG::IterationWorker<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, Step37::LaplaceOperator<2, 3, double>, dealii::PreconditionMG<2, dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, dealii::MGTransferMatrixFree<2, double> >, int>::do_iteration(unsigned int)

[Sean Johnson]

Okay the problem is the global vector that I use for the additional DoFHandler does not have a mg_transfer to help the vector have the right amount of indices on all levels of the multigrid.

I don't see a way to construct the preconditioner with multiple transfers though it does allow multiple DoFHandlers. Is there something I am missing or does Deal.II not allow for this yet?

Thanks,

Sean Johnson